KR20220129433A - Method and apparatus for generating and editing images using contrasitive learning and generative adversarial network - Google Patents

Abstract

Disclosed are a method and an apparatus for generating and editing images using contrastive learning and a generative adversarial network. The method for generating and editing images using contrastive learning and a generative adversarial network comprises: a step of extracting common features from an input image input into a discriminator to generate common feature data; a step of projecting the common feature data to the dimension of class embedding of a real image; and a step of operating to minimize a conditional contrastive loss based on the common feature data and class embeddings, or to make a conditional contrastive loss converge. The operating step discriminates the veracity of the input image, generates a fake image, or changes a portion of the input image based on the data-data relationship and the data-class relationship among multiple image embeddings in the same dimension using the conditional contrastive loss. Therefore, the present invention can effectively generate images with excellent performance or edit images using contrastive learning.

Description

METHOD AND APPARATUS FOR GENERATING AND EDITING IMAGES USING CONTRASITIVE LEARNING AND GENERATIVE ADVERSARIAL NETWORK

The present invention relates to conditional image generation technology, and more particularly, to an image generation and editing method and apparatus utilizing contrast learning and adversarial generative neural networks.

Generative adversarial network (GAN) is a method that was actually first proposed by Ian Goodfellow. It learns the patterns of data by learning the generator and discriminator in an adversarial way, and then generates new data through this method. to be.

Adversarial generative neural networks are widely used as techniques for image synthesis and editing in the field of computer vision.

If the adversarial generative neural network is used, it is possible to perform a task such as synthesizing a face image as shown in FIG. 1 or performing a task such as a painting style transformation as shown in FIG. 2 . That is, FIG. 1 shows newly generated faces that do not exist in the training data as faces generated by the adversarial generative neural network. And Fig. 2 shows the pictures transformed by the adversarial generative neural network.

On the other hand, apart from the image generation performance of adversarial generative neural networks, training an adversarial generative neural network requires a huge amount of data, stable structure generators, discriminators, and delicate initial settings. Therefore, many stabilization techniques have been proposed to alleviate the learning instability of adversarial generative neural networks.

As an example, a method of stabilizing image generation in an adversarial generative neural network has been proposed by giving the correct label of the image to be trained, for example, dog, cat, deer, hedgehog, etc. as a hint. However, such an existing conditional image generation task still has problems that need to be improved, such as the overfitting of the discriminator and the need to additionally provide data for consistency normalization.

As such, there is a demand for a conditional image generation model using a new adversarial generative neural network that can improve existing problems.

The present invention was derived to solve the problems of the existing conditional image generation model, and an object of the present invention is to use a new model, an adversarial generating neural network through contrast learning, to create or edit an image more effectively. is to provide

Another object of the present invention is to provide a method and apparatus for generating and editing images based on various types of adversarial generative neural networks by utilizing adversarial generative neural networks through contrast learning.

An image generation and editing method using contrast learning and an adversarial generating neural network according to an aspect of the present invention for solving the above technical problem includes the steps of extracting common features from an input image input to a discriminator to generate common feature data ; projecting the common feature data into the dimension of class embeddings of the real image; and minimizing the conditional contrast loss due to common feature data and class embeddings or operating so that the conditional contrast loss converges, wherein the operating includes: data between multiple image embeddings in the same dimension using the conditional contrast loss Based on the data relationship and the data-class relationship, the authenticity of the input image is determined, a fake image that is determined as a real image is generated, or a part of the input image is changed.

In one embodiment, the operating step. In the embedding space where the data-data relationship and data-class relationship are expressed, another embedding with a high similarity to the vertex embedding or the objective embedding is pulled closer to be positioned, and another embedding with a low similarity is pushed to be positioned further away.

In one embodiment, the image creation and editing method may further comprise the step of deceiving the authenticity of the input image and generating a realistic fake image with low contrast loss.

In an embodiment, the image creation and editing method may further include inputting a fake image as an input image to the discriminator or inputting a fake image and a real image as input images to the discriminator.

In accordance with another aspect of the present invention for solving the above technical problem, there is provided an image generating and editing apparatus using contrast learning and an adversarial generating neural network, a display; one or more cameras; one or more processors; and a memory storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method of any one of the above-described embodiments.

An apparatus for generating and editing an image using contrast learning and an adversarial generating neural network according to another aspect of the present invention for solving the above technical problem is one including instructions for performing any one of the above-described embodiments and a computer-readable storage medium in which the above program is recorded. Here, one or more programs may be executed by one or more processors of a computing device having one or more cameras and displays.

An image generation and editing method utilizing contrast learning and adversarial generative neural networks according to another aspect of the present invention for solving the above technical problem is a data-data relationship between multiple image embeddings in the same dimension using conditional contrast loss and determining the authenticity of the input image based on the data-class relationship; and deceiving the authenticity of the input image and generating a realistic fake image with low contrast loss.

In an embodiment, the conditional contrast loss may correspond to projecting common feature data obtained by extracting common features from input data as a dimension of class embedding of real data.

In one embodiment, the conditional contrast loss pulls another embedding with a high similarity to a vertex embedding or an objective embedding in an embedding space in which data-data relationship and data-class relationship are expressed to be located closer to each other, and another embedding with low similarity is used. It can be processed or learned to push and position further.

An image generating and editing apparatus utilizing contrast learning and adversarial generating neural network according to another aspect of the present invention for solving the above technical problem extracts common features from an input image input to a discriminator to generate common feature data feature extraction model; a projection model that projects common feature data into the dimension of class embeddings of real images; and an optimization model that operates to minimize the conditional contrast loss due to common feature data and class embedding or to converge the conditional contrast loss. The optimization model uses conditional contrast loss to determine the authenticity of an input image based on data-data and data-class relationships between multiple image embeddings in the same dimension, or to generate a fake image that is determined to be real, or to generate a fake image that is Outputs the edited edited image.

In one embodiment, the optimization model is . In the embedding space where the data-data relationship and data-class relationship are expressed, it can be operated or learned to pull another embedding with high similarity to the vertex embedding or objective embedding to be positioned closer and push another embedding with low similarity to be positioned further away. have.

In one embodiment, the image creation and editing apparatus may further include a generator that deceives the authenticity of the input image and generates a realistic fake image with low contrast loss. The fake image and the real image may be input to the discriminator as input images.

An image generation and editing apparatus utilizing contrast learning and adversarial generative neural network according to another aspect of the present invention for solving the above technical problem is a data-data relationship between multiple image embeddings in the same dimension using conditional contrast loss and a discriminator for determining the authenticity of the input image based on a data-class relationship; and a constructor that deceives the authenticity of the input image and generates a photorealistic image with low contrast loss. Discriminators use conditional contrast loss to determine the authenticity of an input image based on data-data relationships and data-class relationships between multiple image embeddings in the same dimension, or create fake images based on input images, or alter parts of an input image. to output

In an embodiment, the image generating and editing apparatus further comprises a generator separating unit or an input conversion unit that separates the generator from the discriminator and connects the input unit to which the sample is inputted to the discriminator, and receives the sample input to the discriminator as an input image. It can be processed through a neural network to generate a new image or to output an edited image obtained by editing an input image.

According to the present invention, a novel Contrastive Generative Adversarial Networks (ContraGAN) for conditional image generation is provided. ContraGAN uses conditional contrast loss (2C loss), that is, contrast learning based on both data-class relationships and data-data relationships, to create images or edit images with good performance and efficiency. Indeed, it has been experimentally confirmed that ContraGAN improves state-of-the-art results by 7.3% and 7.7%, respectively, on Tiny ImageNet and ImageNet datasets.

In addition, according to the contrastive adversarial neural network using the contrast learning of the present invention, it can greatly contribute to alleviating the overfitting problem of the discriminator, and can obtain favorable results in the adversarial generative neural network without data increase for consistency normalization, and a large amount of data Excellent image creation or image editing results can be obtained even when consistency normalization is applied using

1 is a face generated by the existing generative adversarial networks (GAN), and is an exemplary diagram of a result of generating a new face that does not exist in the training data.
FIG. 2 is an exemplary diagram of style-transformation pictures generated through an existing adversarial generative neural network.
3A is a diagram illustrating the main structure of an auxiliary classifier GAN (ACGAN) as an example of a conditional image generation model of a comparative example.
3B is another example of the conditional image generation model of the comparative example, which shows a GAN with projection discriminator (ProjGAN), respectively.
4 is a diagram showing the main structure of a contrastive adversarial neural network according to an embodiment of the present invention.
5 is a schematic diagram for explaining the contrast learning algorithm of the contrastive neural network of FIG. 4 .
6 is a view showing the metric learning results of the contrastive adversarial network according to the present embodiment together with comparative examples.
7 is an exemplary diagram of a training algorithm of the contrastive neural network of FIG. 4 .
8 is a diagram illustrating a result of learning and generating an ImageNet data set using a contrastive adversarial neural network according to the present embodiment.
9 is a graph showing the results of spectral analysis for various performances of the contrastive adversarial network according to the present embodiment in comparison with the corresponding results of the comparative example (ProjGAN).
10 is a block diagram of an apparatus for generating and editing an image using a contrastive adversarial network according to another embodiment of the present invention.
FIG. 11 is a block diagram for explaining a process of selectively learning an image or generating or editing an image using a contrastive neural network in the image generating and editing apparatus of FIG. 10 .
12 is a flowchart of an image creation and editing method using a contrastive adversarial network according to another embodiment of the present invention.
13 is a flowchart for explaining a modified embodiment of the image creation and editing method of FIG. 12 .
14 is a flowchart for explaining a training mode in the image creation and editing method of FIG. 12 .
15 is a flowchart for explaining a process of generating a new image in an image creation and editing method using a contrastive adversarial network according to another embodiment of the present invention.
16 is a flowchart illustrating a process of generating a style-converted image in an image creation and editing method according to another embodiment of the present invention.
17 is a flowchart illustrating a process of generating an avatar based on a face image in an image generating and editing method according to another embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term "and/or" includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that there is no other element in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Before describing specific embodiments of the present invention, a brief description of the invention background of the present invention is as follows.

First, the existing conditional image generation is the operation of generating various images using class label information. Although many conditional generative adversarial networks (GANs) have shown realistic results, these existing methods reduce the pair-based relationship between image embeddings and corresponding label embeddings, i.e. data-to-class relations, to loss of condition. consider

Accordingly, the present embodiment proposes a ContraGAN (ContraGAN) using a conditional contrast loss in consideration of the relationship between multiple image embeddings and the data-class relationship in the same arrangement, that is, data-to-data relations. Here, ContraGAN's discriminator determines the authenticity of a given sample and minimizes the contrasting object for learning the relationship between training images. And the generator works to deceive authenticity and produce realistic images with low contrast loss.

The configuration of the present invention described above will be described in more detail based on the existing conditional generative adversarial networks (GANs).

Conditional image generation is one of methods for stably training an adversarial generated neural network by using information about web crawled images and image categories such as cats and dogs.

The most representative method is a GAN with a projection discriminator proposed by Miyato et al., which is a model that performs adversarial learning by projecting a given image category onto the feature map of the discriminator.

However, the projective discriminator is vulnerable to the overfitting problem of adversarial learning and has a problem in that it is difficult to learn meaningful correlations between images because all images are assumed to be independent of each other and learned.

Accordingly, the present invention proposes a new framework, an adversarial generative neural network through contrast learning (simply a 'contrastive adversarial neural network') to solve the above two problems, namely, the overfitting problem and the difficult problem of correlation learning.

The contrastive neural network significantly improves contrast loss, which is widely used in self-supervised contrast learning, and replaces image augmentation with image category features for efficient learning, and positive addition to eliminate the effect of false positives on learning. technology has been added.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

3A is a diagram illustrating the main structure of an auxiliary classifier GAN (ACGAN) as an example of a conditional image generation model of a comparative example. 3B is another example of the conditional image generation model of the comparative example, which shows a GAN with projection discriminator (ProjGAN), respectively. And FIG. 4 is a diagram showing the main structure of a contrastive adversarial neural network according to an embodiment of the present invention.

A general approach of the existing conditional GAN is to input label information into a generator and a discriminator. For example, ACGAN (auxiliary classifier GAN), which is a kind of conditional GAN, is configured to determine the class of an image by attaching an additional classifier on the convolutional layers of the discriminator. And, another form of conditional GAN, projection GAN (ProjGAN), is configured to optimize adversarial loss by utilizing class embedding and inner product.

That is, referring to FIG. 3A , the discriminator of the ACGAN of the comparative example (hereinafter, the main discriminator) includes two classifiers and an auxiliary classifer. In this specification, two classifiers of the main discriminator will be referred to as a feature extractor (Dφ ₁ ) and a discriminator (Dφ ₂ ) to distinguish them from the auxiliary classifier, and the auxiliary classifier will be briefly referred to as a classifier. . A secondary classifier may also be referred to as an additional classifier.

The feature extractor Dφ ₁ functions to extract common features from data of the input image(x). Data (Dφ ₁ (x)) (hereinafter, common feature data) from which common features are extracted from the input image data by the feature extractor Dφ ₁ is transmitted to the input of the discriminator Dφ ₂ and the input of the classifier, respectively do.

The discriminator Dφ ₂ determines whether the input image (x) is real or fake through the common feature data (Dφ ₁ (x)) _. ) corresponds to the adversarial loss.

The classifier classifies the category of the input image through the common feature data (Dφ ₁ (x)). The output of the classifier corresponds to a classification loss based on the difference from the original or real image (input label (y)).

The main discriminator or discriminator (Dφ ₂ ) of ACGAN is learned in such a way that the loss value becomes false based on the input label (input label(y)) corresponding to the real image.

The data generated by such ACGAN tends to be classified well even when put into other classifiers. In other words, ACGAN is suitable for synthesizing images with good class classification or generating reasonable data as the auxiliary classifier guides the constructor.

ACGAN generator generates fake image or fake data by combining label information such as class and noise, similar to the case of general conditional GAN. The fake data is input to the discriminator along with the data including the input image.

The objective function of the ACGAN may be the same as the objective function of the discriminator of the existing conditional GAN. In other words, the objective function of ACGAN can correspond to determining whether the data is real or fake and classifying the data category.

Such ACGAN is useful for learning push and pull information between image classes, but if the number of classes increases, learning takes a long time and learning is difficult. Accordingly, the present embodiment provides a neural network that performs image synthesis well even in a large number of detailed data sets through contrastive learning that utilizes the relationship between image data and image data together in addition to the relationship between image data and classes.

In addition, as shown in Fig. 3(b), the projection GAN (ProjGAN) of the comparative example includes the common feature data (Dφ ₁ (x)) extracted from the feature extractor (Dφ ₁ ) and the real image (input label (y) ) is configured to improve the learning and performance of ACGAN through the inner product with the class embedding(e).

Here, class embedding refers to the process of finding or vectorizing the pattern of the real image while preserving the necessary information in the real image (input label(y)). Class embedding should include the process of normalizing by dividing the L2 distance after finding the L2 distance because it is difficult to determine the value if the scale of the real image is too large and the value distribution range is wide.

As such, in the projection GAN, a problem of overfitting the discriminator occurs not a little, and there are problems such as additional data must be provided for consistency normalization.

On the other hand, as shown in FIG. 4 , in the contrastive generative adversarial network (ContraGAN) according to the present embodiment, the conditional functions of ACGAN and ProjGAN are the data of the training sample - a pair that considers only the class relationship It effectively combines the strengths of ACGAN and ProjGAN with the fact that they can be interpreted as pair-based losses. In other words, ContraGAN utilizes conditional contrast loss, that is, contrast learning, considering both the advantage of using the information of the input image in ACGAN and the advantage of easy learning in ProjGAN.

In other words, the contrastive adversarial network according to this embodiment has common feature data Dφ ₁ (x) corresponding to extracting common features of the input image (x) from the feature extractors Dφ ₁ , 10, and A conditional contrastive loss (2C loss) is formed using the original feature vector obtained through the class embedding model (class embedding (e), 40) of the real image (input label (y)) and used.

In this case, in forming the conditional contrast loss (2C loss), the dimension of the common feature data Dφ ₁ (x) is fundamentally different from the dimension of the original feature vector. Accordingly, in the present embodiment, the common feature data Dφ ₁ (x) may be projected onto the original feature vector through the projection model (projection (h), 30 ) to be converted into the same dimension.

The conditional contrast loss (2C loss) considers both data-data relationships and data-class relationships. In order to utilize the data-data relationship, a loss function used in self-supervised learning or metric learning can be appropriately utilized.

That is, the ContraGAN of this embodiment is implemented to explicitly control the distance between embedded image features according to labels by adding metric learning or self-supervised learning goals to the discriminator and the generator. Suitable candidates for the metric learning loss may include a contrastive loss, a triplet loss, a quadruplet loss, and an N-pair loss. The processing of triplet loss or quadruplet loss requires more training complexity and may take longer training time, but it can be adopted as needed.

Also, proxy-based losses mitigate mining complexity using trainable class embedding vectors, but these losses do not explicitly take into account data-data relationships, but in this example conditional contrast losses (2C loss) ) to solve these problems. In other words, ContraGAN using conditional contrast loss (2C loss) has the advantage that it is easy to learn even with large data while utilizing information from the input image by considering the data-class relationship of the input image and the relationship between the data-data together. have

FIG. 5 is a schematic diagram for explaining the operation principle of the contrast learning algorithm of the contrastive neural network of FIG. 4 . And FIG. 6 schematically shows the metric learning loss of the contrastive adversarial network of this embodiment and the metric learning loss of the comparative examples.

All learning losses in conditional GANs are usually designed to collect samples if they have the same label and keep them away if they don't. In the case of using the loss function of ProjGAN in the comparative example, the reference and the corresponding class embedding are close to each other when the reference is an actual image, but are pushed away if not.

On the other hand, in the contrastive adversarial network of this embodiment, as shown in Fig. 5, the discriminator minimizes and otherwise maximizes the distance between actual image embeddings of the same class to perform self-updating, and class embeddings are conditional contrastive, 2C) allows discriminators to learn detailed representations of real-world images by forcing them to be related through loss, and allows constructors to learn more detailed representations of real-world images, such as intra-class characteristics and higher-order representations of real-world images. It makes use of the discriminator's knowledge to create a more realistic image.

That is, in FIG. 5 , the type or color of the hatching (red, blue, yellow) represents the class label, and the shape represents the role. That is, the circle shape (C1) is the embedding of the original or reference image to which the loss is applied, and the square shape (R1, R2, R3, R4, R5, R6, R7) is the embedding of the input image. That is, the star shape (S1, S2, S3) represents the embedding of the class label, respectively. And the thickness of the solid line or red line and the dotted line or blue line thickness indicate the strength of pulling force and strength of pushing force, respectively, in the order described.

Another case of the 2C loss (f) of this embodiment, the metric learning loss (a, b, c) of the comparative example, and the conditional GAN (d, e) of another comparative example are schematically shown in FIG. 6 .

6, (a) of the metric learning loss represents a triplet, (b) is P-NCA (proxy-neighborhood component analysis), and (c) represents NT-Xent (normalized temperature-scaled cross entropy), respectively, In the conditional GAN of another comparative example, (d) represents ACGAN and (e) represents ProjGAN, respectively.

Also, in FIG. 6 , a color indicates a class label, and a shape indicates a role. That is, the circle shape represents the embedding of the original or reference image, the square shape represents the embedding of the input image, the diamond shape represents the embedding of the augmented image, and the star shape represents the embedding of the input image. The embedding of the class label and the triangle shape represent one-hot encoding of the class label, respectively. And the thickness of the solid line or red line and the dotted line or blue line thickness indicate the strength of pulling force and strength of pushing force, respectively, in the order described.

As shown in Fig. 6, unlike ACGAN and ProjGAN of the comparative example, the 2C loss used for image generation and synthesis of this example is data-to-class between training samples including input images or real images. Relationships and data-to-data relationships can be considered.

Before introducing the aforementioned 2C loss, the corresponding part of this embodiment can be expressed with NT-Xent loss. That is, assuming that there is a set of randomly sampled mini-batch and corresponding class labels from training images, a deep neural network encoder and a projection layer embedded on a new unit hypersphere can be defined. NT-Xent loss is for unsupervised learning, and augmented images can be considered as positive samples to consider the data-data relationship between the original and augmented images.

However, compared with the 2C loss, NT-Xent hardly collects image embeddings of the same class because there is no supervision in the label information. Moreover, NT-Xent loss requires additional data growth and additional forward propagation and additional backpropagation.

As such, the 2C loss used in this example utilizes weak supervision of label information, and additional data increase and additional forward propagation and additional backpropagation are not required, so the learning time is shorter than the model of the comparative example with NT-Xext loss. This can be at least several times shorter.

In other words, when compared with the comparative examples (see (a) to (e) of Fig. 7), the 2C loss of this embodiment (f) considers the data-data relationship and the data-class relationship, and the total information without data increase By inferring , it is possible to effectively measure and represent the similarity between related objects.

7 is an exemplary diagram of a training algorithm of the contrastive neural network of FIG. 4 .

Referring to Figure 7, as described in lines 4 (4:) and 5 (5:) of Algorithm 1 for training of the ContraGAN of this embodiment, m in the discriminator training step It can be seen that the 2C loss is calculated using m real images and m intermediate images generated in the generator training step. The intermediate image corresponds to a common feature image or common feature data.

As shown in Algorithm 1, the discriminator performs self-updating by minimizing, otherwise maximizing the distance between actual image embeddings of the same class.

Moreover, in this embodiment, by forcing the class embeddings to be related through a 2C loss, the discriminator can learn a detailed representation of the real image. Similarly, constructors can utilize the knowledge of discriminators, such as intra-class characteristics and higher-order representations of real-world images, to create more realistic images.

In the image generation and synthesis method of this embodiment or its framework (i.e. ContraGAN) using the aforementioned conditional contrast loss (2C loss), similar to the general training procedure of conditional GAN, discriminant training discriminator calculating adversarial loss It has a child training phase and a constructor training phase. ContraGAN is configured to further calculate the 2C loss using either real or fake image sets based on the above trainings.

Contrastive neural networks using the conditional contrast loss described above, unlike existing methods, learn commonalities and differences between images, and can generate images by well understanding the detailed characteristics of each category, and the loss function cannot be counted. Since it is determined by a large number of image pairs, it is robust to the overfitting problem.

Equation 1 below expresses the loss function of the contrastive neural network proposed in this embodiment as an equation.

는 이미지,

는 이미지 레이블,

는 레이블 임베딩 모델,

은 이미지 임베딩 모델,

는 하드니스 조절을 위한 하이퍼매개변수를 각각 나타낸다. 또한,

는 레퍼런스 샘플(reference sample, )에 대한 이미지 임베딩 함수를,

는 클래스 임베딩 함수를,

는 클래스 임베딩을,

는 데이터-클래스 관계를,

는 데이터-데이터 관계를 각각 나타낸다.In Equation 1,

is the image,

is the image label,

is the label embedding model,

silver image embedding model,

denotes hyperparameters for hardness control, respectively. In addition,

is the image embedding function for the reference sample,

is the class embedding function,

is the class embedding,

is the data-class relationship,

represents a data-data relationship, respectively.

In this embodiment, the loss function of the contrastive neural network is defined as a deep neural network encoder S(x) and a projection layer embedded in a hypersphere (h) with a radius of 1, and then the deep neural network encoder (S) is used as a unit hypersphere function ( It can be configured to map the data space to the initial sphere using the synthesis result reflected in h()).

Here, the contrastive adversarial network uses a part of the feature extractor (Dφ ₁ ), which is the first discriminant neural network of the discriminator, as the encoder network (S) in front of the fully connected layer, and multi-layer parameterized with Φ. Multi-layer perceptrons can be used as the projection head (h).

The learning algorithm according to this embodiment was tested by building an integrated software library for learning adversarial generative neural networks in order to analyze the algorithm fairly and thoroughly.

The software of this example has a total of 18 adversarial generative neural networks, 4 learning techniques (parallel variance processing, mixed precision learning, batch synchronization, batch statistics accumulation), 4 analysis techniques (image visualization, nearest neighbor analysis, linear interpolation analysis) , frequency analysis), and finally, it can have any configuration selected from the four evaluation measures (inception score, prechet inception distance, precision, and salience rate).

18 adversarial generative neural networks include DCGAN, LSGAN, GGAN, WGAN-WC, WGAN-GP, WGAN-DRA, ACGAN, ProjGAN, SNGAN, SAGAN, BigGAN, BigGAN-Deep, CRGAN, ICRGAN, LOGAN, DiffAugGAN, ADAGAN, ContraGAN including, but not limited to.

Table 1 below shows the experimental results showing that the performance of the contralateral neural network of this example is compared with the models of the comparative example, and that the contralateral neural network is robust to overfitting.

Table 1 shows the results of the CIFAR10 data image generation experiment, indicating that the higher the value, the better the performance of all measures except the prechet inception distance. The parts indicated in bold are the results of the ContraGAN and its variants (CRGAN, ICRGAN, DiffAugGAN) using the contrast learning of this embodiment.

Next, an image generation experiment was performed on the ImageNet dataset, and the results are shown in FIG. 8 below. 8 shows the results generated after learning about the ImageNet dataset according to the method according to the present embodiment.

Referring to FIG. 8 , it can be visually confirmed that the model of this embodiment can naturally generate an image without memorizing the training data. The experimental results showed that ContraGAN outperformed the state-of-the-art model by 7.3% and 7.7% on Tiny ImageNet and ImageNet datasets, respectively.

In addition, we experimentally show that control learning helps mitigate discriminator overfitting. For fair comparison, 12 latest GANs were implemented and tested using the PyTorch library.

As such, in this embodiment, a new conditional contrast loss is used to maximize the lower limit of mutual information between samples of the same class. That is, the framework of the present embodiment, referred to as ContraGAN, is configured to synthesize images using class information and data-data relationships of training samples.

In addition, ContraGAN's discriminator differentiates the authenticity of a given sample and maximizes the mutual information between true image embeddings of the same class. And ContraGAN's constructor is configured to synthesize images to deceive the discriminator and maximize the mutual information of fake images before the same class.

For fair comparison, the present inventors implemented and compared the existing nine newest approaches (comparative examples) in order to test various methods under the same conditions. As a result of the experiment, it was confirmed that the ContraGAN of this example was robust in network architecture selection, and surpassed the state-of-the-art models of the data sets of the comparative examples (CIFAR10, Tiny ImageNet, etc.) by 3.7% and 11.2% without data increase.

9 shows the results of spectral analysis of the contrast-versus-learning neural network according to the present embodiment.

Referring to the first graphs on the left in columns 1 and 2 of FIG. 9 , as a result of testing the training dataset and the validation dataset with the model learned using the training dataset, that is, the discriminator determines the specific feature data of the sample and As a result of comparing classification accuracy when generating sample details by determining certainty or authenticity, in the comparative example ProjGAN, the training accuracy and It can be determined that overfitting has occurred due to a large difference in validation accuracy, and it can be seen that overfitting occurs in the stack section of about 45000 to about 70000 in the ContraGAN of this embodiment.

As shown in the above spectrum analysis results, it can be confirmed that the contraGAN of this example is more robust to overfitting than the ProjGAN of the comparative example. That is, it can be experimentally confirmed that the contraGAN of this embodiment is robust to overfitting, and therefore the adversarial learning collapse occurs later than the baseline model (ProjGAN).

In other words, as can be seen from the results of the baseline model (ProjGAN) of the comparative example (left in the upper column of FIG. 9) and the results of the model (ContraGAN, ours) of this example (the left of the lower column of FIG. 9), It can be seen that the difference in training and validation accuracy of the contrasting neural network of the embodiment increases slowly.

In addition, as can be seen from the middle graphs of the middle and lower columns of FIG. 9 , the increase in frechet inception distance (FID) indicating that the lower the performance is, the higher the frechet inception distance (FID) is later in the case of this example compared to the comparative example. wake up can be confirmed. This is a phenomenon that occurs because the model of this embodiment is robust to overfitting.

Finally, as can be seen from the graphs on the right of the upper column and the right graph of the lower column of FIG. 9 , looking at the results of spectral analysis of the baseline model (ProjGAN) of the comparative example and the model (ContraGAN) of this example, the contrast between the comparative example and the present example It can be seen that the spectra of the Daeshin light network are visually more stable.

As such, the contrastive neural network according to the present embodiment tends to perform image synthesis well in a detailed data set, and in particular, due to the size constraint of an image feature vector, learning collapse does not occur and data augmentation is performed. It goes well with the regularizstion based on it and shows a big performance improvement in terms of FID (see Table 2).

Referring to Table 2, the contrastive adversarial network of this embodiment has an advantage with respect to a large batch size, as can be seen in different implementations (A, C, E, H).

In addition, the effect of the 2C loss of the implementations (A, C, E, H) of the contrastive neural network of this embodiment is significantly better than the existing ones based on the FID score, and in particular, the vanilla networks (A, C) showed a decrease of 21.6% and 11.2%, respectively.

In addition, as a result of comparing before and after application of APS (augmented positive samples), it was confirmed that the embodiment (F) to which the APS was applied took about 12.9% more time (Time) than the embodiment (E) to which the APS was not applied. . It is speculated that this is because each class embedding can be a representative of the class and acts as an anchor to pull the corresponding image. In addition, it is determined that if there is no class embedding, images of a mini-batch may be collected depending on the sampling state, so that learning may become unstable.

In addition, as a result of comparing the consistency regularization (CR) performance, the implementations (A, E, G) and other implementations (C, H, I) showed that the combination of vanilla network, 2C loss, and CR We simply show that we can reduce the FID of either the result of the vanilla network (A, C), and the result of the combination of the vanilla network and the 2C loss (E, H). This synergy is applicable when CR is used with 2C loss and vanilla network, 2C loss and CR wins vanilla network and CR(B, D) by a large margin.

10 is a block diagram of an apparatus for generating and editing an image using a contrastive adversarial network according to another embodiment of the present invention. FIG. 11 is a block diagram for explaining a process of selectively learning an image or generating or editing an image using a contrastive neural network in the image generating and editing apparatus of FIG. 10 . 12 is a flowchart of an image creation and editing method using a contrastive adversarial network according to another embodiment of the present invention. 13 is a flowchart for explaining a modified embodiment of the image creation and editing method of FIG. 12 . And FIG. 14 is a flowchart for explaining a training mode in the image creation and editing method of FIG. 12 .

Referring to FIG. 10 , the image generating and editing apparatus 100 stores instructions for instructing at least one processor 110 and at least one processor 110 to perform a series of steps. It may be included in a computing device that includes or includes a memory 120 .

In addition, the image generating and editing apparatus 100 may include a transceiver 130 that exchanges signals and data with an external device through a wired, wireless, or wired/wireless network, and may include an input interface, an output interface, or an input/output interface. It may further include an interface 140 having a , and a storage device 150 .

Each of the components included in the image generating and editing apparatus 100 may be connected to each other by an internal network line or bus 160 to exchange signals and data.

The processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed.

Each of the memory 120 and the storage device 150 may be formed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 120 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM). At least one of the memory 120 and the storage device 150 may include a computer-readable storage medium in which one or more programs including the above instructions are recorded, or may be included in the storage medium.

The interface 140 may include a display device, a camera, and the like. The device including the display device and the camera may include a mobile phone, a personal digital assistant (PDA), a portable terminal such as a smart pad, a notebook computer, a personal computer, and the like.

In addition, the above-described processor 110 is equipped with instructions stored in the electronically connected memory 120 or programs or software modules implemented by these instructions, and by the instructions, to implement the method of the present invention A series of steps can be performed. That is, the processor 110 may be equipped with a contrastive neural network 170 , and the contrastive neural network 170 may be configured by the above instructions or software modules.

As shown in FIG. 11, these software modules are elements constituting the contrastive adversarial network 170 of the present embodiment, and include a generator 172, a preprocessor 174, an input conversion unit 176, and a discriminator. (178). In addition, it may further include an output interface or output unit 140 .

In this embodiment, the generator 172 used in the training mode of the contralateral neural network is separated from the discriminator 178 through the input conversion unit 176, a sample is input using the discriminator 178, and a sample is received through the discriminator It is configured to create or edit an image based on

The generator 172 and the preprocessor 174 may be selectively connected to the discriminator 178 through the input conversion unit 176 . The description of the generator 172 and the discriminator 178 overlaps with the description of the above-described embodiment and thus will be omitted.

When the generator 172 coupled to the discriminator 178 in the training mode by the input conversion unit 176 is in a separated state, the preprocessor 174 may receive a sample and input it to the discriminator 178 Samples can be converted to a given shape or size.

The discriminator 178 extracts common features from the input image to generate common feature data (S112), projects the common feature data into the dimension of class embedding of the truth image (S114), and uses the common feature data and class embedding An operation S116 may be performed to minimize the conditional contrast loss or to converge to a preset value (see FIG. 12 ).

In other words, the discriminator 178 extracts common features from the input image to generate common feature data (S112), and projects the common feature data as the dimension of class embedding of the true image (S114), and the common feature data and class It is possible to minimize the hostile loss of the discriminator using the conditional contrast loss due to embedding or operate to converge to a preset value ( S118 ) (see FIG. 13 ).

The above-described discriminator 178 may include a feature extraction model, a projection model, and an optimization model. Here, the feature extraction model may correspond to the feature extractor, the projection model may correspond to the projection function, and the optimization model may correspond to a model optimizing the 2C loss (see FIG. 4 ).

The above-described feature extraction model may generate common feature data by extracting common features from an input image input to the discriminator. The projection model can project common feature data into the dimension of class embeddings of the real image. In addition, the optimization model may operate to minimize the conditional contrast loss due to common feature data and class embedding or to converge the conditional contrast loss.

In particular, the optimization model uses conditional contrast loss to determine the authenticity of an input image based on data-data relationships and data-class relationships between multiple image embeddings in the same dimension, to determine whether to create a fake image or change a part of the input image. can train you.

Referring back to FIG. 11 , the generated image generated by the discriminator 178 , the style-converted image, or the avatar may be output through the output unit 140 . The output unit 140 may output an image when it is determined that the 2C loss of the discriminator 178 is minimized or optimized based on a preset reference value. The output unit 140 may output the final generated image of the discriminator 178 according to the signal of the optimization model.

In addition, as shown in FIG. 14 , for the learning of the aforementioned contrastive neural network, in a state in which the generator is connected to the discriminator, the generator can generate a fake image, and the fake image generated by the generator is the real image and the real image. It may be transmitted to the discriminator together (S110). The discriminator extracts common features from the input image to generate common feature data (S112), projects the common feature data into the dimension of class embedding of the true image (S114), and reduces conditional contrast loss due to common feature data and class embedding It can be minimized or learned to converge to a preset value (S116).

15 is a flowchart for explaining a process of generating a new image in an image creation and editing method using a contrastive adversarial network according to another embodiment of the present invention. 16 is a flowchart illustrating a process of generating a style-converted image in an image creation and editing method according to another embodiment of the present invention. And FIG. 17 is a flowchart for explaining a process of generating an avatar based on a face image in an image generating and editing method according to another embodiment of the present invention.

Referring to FIG. 15 , an image generation and editing method utilizing a contrastive adversarial network is first input based on data-data relationship and data-class relationship between multiple image embeddings in the same dimension using conditional contrast loss by a discriminator. It is possible to determine the authenticity of the image (S142).

Next, the generator may deceive the authenticity of the input image and generate a realistic fake image with low contrast loss (S144).

In addition, as shown in Fig. 16, the image creation and editing method utilizing the contrastive neural network is a data-data relationship and data-class relationship between multiple image embeddings in the same dimension using conditional contrast loss by a discriminator. After determining the authenticity of the input image based on (S142), it is determined whether only the background area of the input image is selected (Yes in S143), and accordingly, the authenticity of the determined input image is deceived and a background image with low contrast loss is generated And (S145), by synthesizing the input image and the background image may be configured to output a style-converted image.

In addition, as shown in Fig. 17, the image creation and editing method utilizing the contrastive neural network is a data-data relationship and data-class relationship between multiple image embeddings in the same dimension using conditional contrast loss by a discriminator. After determining the authenticity of the face image based on (S142), it is possible to deceive the authenticity of the determined face image and generate a fake face image with low contrast loss (S146), and output an avatar combining the generated close face image. (S148).

The operation of the image creation and editing method of the present invention according to the above-described embodiments can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. In addition, the computer-readable recording medium may be distributed in networked computer systems to store and execute computer-readable programs or codes in a distributed manner.

In addition, the computer-readable recording medium may include a hardware device specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although some aspects of the invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, wherein a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method may also represent a corresponding block or item or a corresponding device feature. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, the field programmable gate array may work with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

Although the above has been described with reference to the preferred embodiment of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. You will understand that you can.

Claims (15)

An image creation and editing method utilizing contrast learning and adversarial generative neural networks, comprising:
generating common feature data by extracting common features from an input image input to a discriminator;
projecting the common feature data into a dimension of a class embedding of a real image; and
Minimizing the conditional contrast loss due to the common feature data and the class embedding or operating so that the conditional contrast loss converges;
The operating step uses the conditional contrast loss to determine the authenticity of the input image based on the data-data relationship and the data-class relationship between multiple image embeddings in the same dimension, generate a fake image, or select a part of the input image. How to change, create and edit images.
The method according to claim 1,
The above operating steps. In the embedding space in which the data-data relationship and the data-class relationship are expressed, a vertex embedding or another embedding having a relatively high similarity to the target embedding is pulled to be located closer, and another embedding having a relatively low similarity is pushed further away. How to create and edit images.
The method according to claim 1,
The conditional contrast loss is expressed by Equation 1 below,
[Equation 1]

In Equation 1 above,

is the image,

is the image label,

is the label embedding model,

silver image embedding model,

is an image creation and editing method, each representing hyperparameters for hardness adjustment.
The method according to claim 1,
and deceiving the authenticity of the input image and generating a realistic fake image with low contrast loss.
5. The method according to claim 4,
and inputting the fake image and the real image as the input image to the discriminator.
An image generation and editing device utilizing contrast learning and adversarial generative neural networks, comprising:
display;
one or more cameras;
one or more processors; and
a memory storing one or more programs configured to be executed by the one or more processors;
The one or more programs include instructions for performing the method of any one of claims 1 to 5, image creation and editing apparatus.
An image generation and editing device utilizing contrast learning and adversarial generative neural networks, comprising:
A computer-readable storage medium in which one or more programs including instructions for performing the method of any one of claims 1 to 5 are recorded,
wherein the one or more programs are executed by one or more processors of a computing device having one or more cameras and displays.
An image creation and editing method utilizing contrast learning and adversarial generative neural networks,
determining the authenticity of an input image based on data-data relationships and data-class relationships between multiple image embeddings in the same dimension using conditional contrast loss; and
and deceiving the authenticity of the input image and generating a realistic fake image with low contrast loss.
9. The method of claim 8,
and the conditional contrast loss corresponds to projecting common feature data obtained by extracting common features from input data as a dimension of class embedding of real data.
10. The method of claim 9,
The conditional contrast loss pulls another embedding having a relatively high similarity to a vertex embedding or an object embedding in an embedding space in which the data-data relationship and the data-class relationship are expressed to be located closer to each other, and another embedding having a relatively low similarity is pulled. How to create and edit images, which is processed to be pushed further away.
An image generation and editing device using contrast learning and adversarial generative neural networks, comprising:
a feature extraction model for generating common feature data by extracting common features from an input image input to a discriminator;
a projection model for projecting the common feature data into a dimension of class embedding of a real image; and
an optimization model operable to minimize a conditional contrast loss due to the common feature data and the class embedding or to converge the conditional contrast loss;
includes,
The optimization model uses the conditional contrast loss to determine the authenticity of the input image based on the data-data relationship and data-class relationship between multiple image embeddings in the same dimension, generate a fake image, or change a part of the input image to output, image creation and editing devices.
12. The method of claim 11,
The optimization model is In the embedding space in which the data-data relationship and the data-class relationship are expressed, a vertex embedding or another embedding having a high similarity to the target embedding is pulled to be located closer, and another embedding having a low similarity is pushed to be positioned further away, Image creation and editing device.
12. The method of claim 11,
further comprising a constructor that deceives the authenticity of the input image and generates a realistic fake image with low contrast loss;
and the fake image and the real image are input to the discriminator as the input image.
a discriminator that uses conditional contrast loss to determine the authenticity of an input image based on data-data relationships and data-class relationships between multiple image embeddings in the same dimension; and
a generator that deceives the authenticity of the input image and generates a realistic image with low contrast loss;
The discriminator uses the conditional contrast loss to determine the authenticity of the input image based on data-data relationships and data-class relationships between multiple image embeddings in the same dimension, or generate a fake image based on the input image, or An image creation and editing device that changes parts of an image and outputs it.
15. The method of claim 14,
It further comprises a generator separation unit or an input conversion unit that separates the generator from the discriminator and connects an input unit to which a sample is inputted to the discriminator,
An image generating and editing apparatus for generating a new image or outputting an edited image obtained by editing the input image by processing a sample input to the discriminator as the input image through a neural network.

Images