KR102567128B1 - Enhanced adversarial attention networks system and image generation method using the same - Google Patents

Abstract

The present invention relates to an adversarial attention network system capable of reconstructing a converted image by utilizing an attention sharing mechanism between domains in an image conversion process, generating a more consistent and realistic output, and improving image conversion capability, and an image generation method using the same. it's about

Description

Improved adversarial attention network system and image generation method using the same

The present invention relates to an image generation technology using an improved adversarial attention network, and more particularly, in an image conversion process, by utilizing an attention sharing mechanism between domains to reconstruct a converted image, create a more consistent and realistic output, It relates to an adversarial attention network system capable of improving conversion capability and an image generation method using the same.

In general, in order to learn an artificial neural network-based image recognition model, the accuracy of recognition can be improved by learning the model using learning data including normal images and bad images.

However, in reality, it is easy to obtain various training data for normal images, but various training data for defective images or rare images (images that do not occur completely but contain errors different from normal) There are downsides that are very difficult to find.

In order to solve this problem, there is a method in which a person manually creates a bad image based on a normal image, but this method has a disadvantage in that the efficiency is significantly lowered in terms of time and cost.

In order to solve this problem, a problem-solving method of generating a bad image through a GAN model is emerging.

A Generative Adversarial Networks (GAN) is a learning network and may be composed of a generator, which is a model in charge of image generation, and a discriminator, which is a model in charge of classification. GAN is a model in which generators and discriminators compete against each other by improving each other's performance.

In GAN, the process of learning the discriminator first and then the generator is repeated. Training of GANs is done in two stages. First, it learns to classify the real image and the fake image generated by the random vector in the generator, and second, the generator obtains a new random noise vector and generates a fake image.

That is, the generator can be trained to produce an image similar to real data to the extent that the discriminator classifies a fake image created by the generator from a random vector as real.

As this learning process is repeated, the discriminator and generator recognize each other as hostile competitors and both develop. will not be able to That is, in GAN, the generator tries to lower the probability of success in classification, and the discriminator tries to increase the probability of success in classification, forming a structure in which each develops in a competitive way.

There were difficulties in introducing the self-attention mechanism to the field of image conversion using GAN. Although self-attention has shown excellent results in various fields such as NLP and computer vision tasks, there is a limitation in that memory complexity occurs when similarity is calculated through the dot product between each pair of tokens. The complexity of these memories presents a problem that can be prohibitively expensive for certain applications.

In addition, since CNN processes input data using a local receptive field, it can process only information that is spatially close to the current processing area. Therefore, as the distance between the input image and the information increases, it is difficult for the CNN to learn the information. However, attention mechanisms are effective at learning information far from images because they allow the model to selectively focus on the most relevant features and capture long-range dependencies between different parts of the input. Therefore, if the memory complexity of self-attention is solved, when it is introduced into GAN, it is possible to learn about images and distant information, thereby improving performance.

According to an embodiment of the present invention, an adversarial attention network system and method utilizing multiple self-attention blocks mathematically equivalent to self-attention are provided.

An object to be solved by the present invention is to provide an adversarial attention network system and method for sharing dependencies between a source image and a target image using cross-domain attention-sharing mechanisms.

An adversarial attention network system according to an embodiment of the present invention uses an encoder that extracts meaningful features from an input image through a series of convolutions, and a series of learned residual blocks to extract the extracted features from a source domain. A translator that changes to the target domain, uses the translated features to construct a translated image with the same resolution as the image input to the encoder, and uses a series of transposed convolutions to raise the features to a desired dimension. scale, and the last convolution layer includes a generator including a decoder that maps outputs to RGB images of multiple channels and a discriminator that discriminates the result of the generator, and the encoder is a long-term It may include an attention module that computes attention maps for resolving long-range dependancy.

The encoder and decoder may use skip connections between an attention module on the i-th layer of the encoder and an attention module on the i-L-th layer of the decoder in order to share the calculated attention map between them.

The attention module may be disposed after the down-sample convolution of the encoder and before the transposed-convolution of the decoder, respectively.

The attention module may be disposed before convolution of the decoder and after transposed-convolution of the decoder, respectively.

In the initial training stage for dependency learning, the network system may introduce a learnable scale parameter (yi) of the formula below to wait until local dependencies are more dependent and long-term dependencies are more meaningful.

The attention module interprets the key (K) of each channel as a single attention map in order to learn the input image from a semantic point of view, and a series of the above omitting between the key vectors of each of the attention modules. Connections may be located.

Between the series of omitted connections, a refinement network for refining the attention map from the original domain may be further located.

The output of the attention module on the encoder side is concatenated with the input of the attention module on the decoder side, and the connected attention module of the decoder operates based on the concatenated feature map of the previous layer and the output of the corresponding attention module of the encoder. can do.

The network system learns two mapping functions to perform image translation between domain A and domain B, but two generators and two discriminators can be trained simultaneously to enhance cycle consistency between domains. there is.

In an image generation method of an adversarial attention network system including an encoder, a translator, and a decoder according to another embodiment of the present invention, the encoder extracts meaningful features from an input image through a series of convolutions; Changing the extracted features from a source domain to a target domain using a series of learned residual blocks, and constructing a translated image having the same resolution as an image input to the encoder by the decoder using the translated features. wherein the decoder utilizes a series of transposed convolutions to upscale the features to a desired dimension and the decoder maps the output to a multi-channel RGB image in a final convolutional layer; Extracting the features may include calculating, by an attention module of the encoder, attention maps for resolving long-range dependancy.

The encoder and decoder may use skip connections between an attention module on the i-th layer of the encoder and an attention module on the i-L-th layer of the decoder in order to share the calculated attention map between them.

The attention module may be disposed after the down-sample convolution of the encoder and before the transposed-convolution of the decoder, respectively.

The attention module may be disposed before convolution of the decoder and after transposed-convolution of the decoder, respectively.

In the initial training stage for dependency learning, the network system may introduce a learnable scale parameter (yi) of the formula below to wait until local dependencies are more dependent and long-term dependencies are more meaningful.

The attention module interprets the key (K) of each channel as a single attention map in order to learn the input image from a semantic point of view, and a series of the above omitting between the key vectors of each of the attention modules. Connections can be located.

Between the series of omitted connections, a refinement network for refining the attention map from the original domain may be further located.

The output of the attention module on the encoder side is concatenated with the input of the attention module on the decoder side, and the connected attention module of the decoder operates based on the concatenated feature map of the previous layer and the output of the corresponding attention module of the encoder. can do.

The network system learns two mapping functions to perform image translation between domain A and domain B, but two generators and two discriminators can be trained simultaneously to enhance cycle consistency between domains. there is.

According to an embodiment of the present invention, it is possible to overcome the limitation of excessive memory use by utilizing multiple self-attention blocks mathematically equivalent to self-attention.

According to an embodiment of the present invention, a long-range dependency can be solved by sharing dependencies between a source image and a target image using cross-domain attention-sharing mechanisms.

According to an embodiment of the present invention, an attention sharing mechanism is utilized in an image-to-image conversion process to reconstruct the converted image and to generate a more consistent and realistic output, thereby improving image conversion capability.

1 is a diagram showing the EAGAN architecture of Embodiment 1;
2 (A) and (B) are views showing the arrangement of the attention module.
FIG. 3 is a table showing performance comparison between deployment options of the attention module of FIG. 2 .
4 is a diagram showing the attention sharing mechanism of EAGAN according to the first embodiment.
5 is a diagram showing the UEAGAN architecture of Embodiment 2;
6 is a diagram showing sample images for entity conversion between images by EAGAN and UEAGAN.
7 is a diagram showing sample images for scene conversion between EAGAN and UEAGAN images.
8 (A) and (B) are tables comparing performance of scene conversion tasks between architectures for each task.
9 is a diagram for visually comparing scene transformations for each architecture.
10 is a diagram for visually comparing object transformations for each architecture.
11 is a diagram for detailed visual inspection of an object conversion model created by the architecture.
12 is a diagram for precise visual inspection of a scene transformation model generated by the architecture.
13 is a diagram illustrating an attention map of the EGAN architecture.
4 is a flowchart of a method for generating an image of an adversarial attention network system according to a third embodiment.

Hereinafter, several embodiments of the present invention will be described in detail using drawings. However, this is not intended to limit the present invention to any specific embodiment, and it should be understood that all transformations, equivalents and substitutions including the technical idea of the present invention are included in the scope of the present invention. do.

In this specification, singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, when a component is described as “having” or “comprises” a certain sub-configuration, it does not exclude other configurations unless otherwise stated, but further includes other configurations. This means that it may contain

In this specification, the terms "...unit", "...module" and "component" mean a unit that processes at least one function or operation, and includes hardware, software or It may be implemented as a combination of hardware and software.

<Example 1>

Embodiment 1 relates to an adversarial attention network system technology using EAGAN (Eficient Attention Generative Adversarial Network) architecture.

1 is a diagram showing the EAGAN architecture of Embodiment 1;

The adversarial attention network system of Example 1 uses EAGAN (Eficient Attention Generative Adversarial Network) architecture to perform image translation (IMAGE TO IMAGE) between unpaired domains A and B.

The EAGAN architecture of the adversarial attention network system of Embodiment 1 is based on GAN.

A Generative Adversarial Networks (GAN) is a learning network and may be composed of a generator, which is a model in charge of image generation, and a discriminator, which is a model in charge of classification. GAN is a model in which generators and discriminators compete against each other by improving each other's performance.

The EAGAN architecture incorporates Attention Mechanisms into the transformation process in a way that can use long-range dependencies.

Long-range dependencies are a phenomenon that can occur in the analysis of spatial or time series data. It is related to the rate at which the statistical dependence of two points decreases as the time interval or spatial distance between them increases. A dependency is generally considered to have a long-term dependency if it decreases more slowly than an exponential decay (usually a decrease such as power). Long-range dependencies are related to self-similar processes or fields.

EAGAN overcomes the limitation of memory overuse by utilizing multiple self-attention blocks that are mathematically equivalent to self-attention. In addition, unlike CNN, where it is difficult to learn the information as the distance between the input image and the information increases, it effectively learns information far from the image using multiple self-attention. This is because the model can selectively focus on the most relevant features and capture long-range dependencies between different parts of the input. FYI, self-attention is an attention mechanism involving different positions in a single sequence to compute a representation of the sequence.

EAGAN solves the long-range dependence problem by sharing dependencies between a source image and a target image using cross-domain attention-sharing mechanisms.

EAGAN of the adversarial attention network system includes a generator and a discriminator.

The generator is responsible for generating the image, and the discriminator determines the output of the generator.

The adversarial attention network system is based on GAN and can repeat the process of first learning the discriminator and then learning the generator.

In GAN, the process of learning the discriminator first and then the generator is repeated. Training of GANs is done in two stages. First, it learns to classify the real image and the fake image generated by the random vector in the generator, and second, the generator obtains a new random noise vector and generates a fake image.

The generator includes an encoder, a translator and a decoder.

The encoder extracts meaningful features from the input image through a series of convolutions. For example, an image can be transformed into a latent feature space through a series of convolutions that down-sample the dimensions of the input.

The encoder includes an attention module that computes attention maps for resolving long-range dependancy.

The attention module is implemented as an improved attention module (eicient-attention modules) using a generator based on CycleGAN (CycleGan). For reference, CycleGAN is a Generative Adversarial Network (GAN) used for image style transfer between domains. Using CycleGAN, you can train an image of one area to transform into another area, and CycleGAN is performed by unsupervised learning. At this time, CycleGAN does not perform one-to-one mapping of images in both domains.

The attention module solves the long-range dependency problem by sharing dependencies between a source image and a target image using cross-domain attention-sharing mechanisms.

A translator changes extracted features from a source domain to a target domain using a series of learned residual blocks.

The translator uses a series of residual blocks to learn how to change the extracted features from the source domain to the target domain. At this stage, the dimensions of the shape are not changed, and the number of remaining blocks can be used as a hyperparameter to strike a balance between speed and quality of the result.

A decoder constructs a translation image having the same resolution as an image input to an encoder by utilizing features translated by the translator.

The decoder utilizes a series of transposed convolutions to upscale the features to the desired dimension, and a final convolutional layer maps the output to a multi-channel RGB image.

When the EAGAN architecture is used in the adversarial attention network system, the encoder and decoder of the generator omit connection between the attention module on the ith layer of the encoder and the attention module on the i-L layer of the decoder to share the calculated attention map between them ( skip connections). Here, i is the layer order of the encoder and L is the layer number of the decoder. For reference, skip connections add the output of one layer to the input of the next layer by skipping several layers.

The location of the attention block can be set closer to the input and output layers. Because of this, when the generator introduces the attention mechanism, it can learn long-range dependencies from the source domain and then share these dependencies with the target domain to improve the consistency of image transformation.

Here features are set closer to the characteristics of each image domain. Therefore, the encoder and decoder appear as ideal locations for calculating long-range dependencies and can share attention maps through skipped connections between attention modules in layers i and i-L.

On the other hand, the adversarial attention network system introduces the learnable scale parameter (yi) of the formula below in order to depend more on local dependencies and wait until long-term dependencies become more meaningful in the initial training stage for dependency learning.

The scale parameter multiplies the output of the attention and then adds back the input function.

[Equation 1]

yi = γo _i + x _i

Here, o _i is the output value of the i th attention, x _i is the i th input feature, and the weight gamma r is initially set to 0 and then set to a value greater than 0 as long-term dependencies are learned from meaningful features.

2 is a diagram showing the arrangement of attention modules.

As shown in (A) of FIG. 2, the attention module is placed after the down-sample convolution of the encoder and before the transposed-convolution of the decoder, respectively.

As shown in (B) of FIG. 2, the attention module is placed before the decoder's convolution and after the decoder's transposed-convolution, respectively.

FIG. 3 is a table showing performance comparison between deployment options of the attention module of FIG. 2 .

FIG. 3 shows the performance results of an architecture in which the attention module is placed before ((A) in FIG. 2) and after ((B) in FIG. 2) the convolution block. The value is expressed as KID (Kernel Inception Distance) and is adjusted in increments of 100. The lower the value, the better the performance.

As shown in the results of FIG. 3, performance is improved in most tasks when the attention module is placed before the decoder's convolution and after the decoder's transposed-convolution, as shown in FIG. 2 (B). appears to be

On the other hand, the attention module interprets the key (K) of each channel as a single attention map in order to learn the input image from a semantic point of view, and a series of omitted connections between the key vectors of each of the attention modules ( skip connections are located.

The Attention module learns to represent semantic aspects of an image instead of corresponding to attention between two points in an image. Since the input and output images are expected to be highly content-related, we can infer that the attention between features in the initial layer i of the generator can be reused in the corresponding ending layers i-L.

Since the attention modules are computed on real samples, sharing these attention maps can be used to reconstruct the transformed image to produce more consistent and realistic output. A series of elided connections are then placed between each key vector in the mirrored Attention module block to share long-distance dependencies between the two ends of the architecture. The attention map of the key (K) network is shared between a pair of attention blocks.

4 is a diagram showing the attention sharing mechanism of EAGAN according to the first embodiment.

Referring to FIG. 4 , a refinement network for refining the attention map from the original domain may be further located between a series of omitted connections located between key vectors of each attention module.

Considering that the attention module interprets each channel as a single attention map, the refinement network is implemented as a depth-wise convolution using a 3 × 3 kernel. This special type of circuit divides the input channels into a given number of groups and uses a different kernel for each of these groups.

That is, the attention module interprets the main result as an attention map shared between the corresponding i and i-L layers, and the attention map is passed through group convolution so that information is not mixed between different channels.

On the other hand, the adversarial attention network system learns two mapping functions to perform image translation between domain A and domain B, but two generators GA and GB and two discriminators DA and DB are domain The mapping functions are simultaneously trained to enhance inter-cycle consistency.

For reference, cycle consistency loss is a type of loss used in generative adversarial networks that perform unpaired image-to-image transformations.

<Example 2>

Embodiment 2 relates to an adversarial attention network system technology using UEAGAN (U-net Efficient Attention Generative Adversarial Network) architecture.

5 is a diagram showing the UEAGAN architecture of Embodiment 2;

The adversarial attention network system uses the UEAGAN architecture to perform image translation (IMAGE TO IMAGE) between unpaired domains A and B.

The EAGAN architecture of the adversarial attention network system is based on GAN.

Since the description of GAN was described in Example 1, redundant description is omitted.

The UEAGAN architecture integrates Attention Mechanisms into the transformation process in a way that can use long-range dependencies.

The UEAGAN architecture overcomes the limitations of memory overuse by utilizing multiple self-attention blocks that are mathematically equivalent to self-attention. In addition, unlike CNN, where it is difficult to learn the information as the distance between the input image and the information increases, it effectively learns information far from the image using multiple self-attention. This is because the model can selectively focus on the most relevant features and capture long-range dependencies between different parts of the input.

The UEAGAN architecture solves the long-range dependence problem by sharing dependencies between a source image and a target image using cross-domain attention-sharing mechanisms.

The UEAGAN architecture of the adversarial attention network system includes a generator and a discriminator.

Since the generator and the discriminator are the same as in Example 1, duplicate descriptions are omitted.

The output of the attention module on the encoder side is concatenated with the input of the attention module on the decoder side, and the connected attention module of the decoder operates based on the concatenated feature map of the previous layer and the output of the corresponding attention module of the encoder. do.

On the other hand, the adversarial attention network system learns two mapping functions to perform image translation between domain A and domain B, but two generators GA and GB and two discriminators DA and DB are domain The mapping functions are simultaneously trained to enhance inter-cycle consistency.

For reference, cycle consistency loss is a type of loss used in generative adversarial networks that perform unpaired image-to-image transformations.

6 is a diagram showing sample images for entity conversion between images by EAGAN and UEAGAN.

Referring to FIG. 6 , a sample image is a sub-task for converting a specific entity displayed in an image between EAGAN and UEAGAN.

In FIG. 6, a data set is built for transformations from apples to oranges and from horses to zebras. For example, the data set may consist of unpaired image samples of orange and apple images that differ in location, quantity, aspect, and environment. there is. The corresponding data set is used for training and evaluation of the model according to the corresponding label.

7 is a diagram showing sample images for scene conversion between EAGAN and UEAGAN images.

Referring to FIG. 7 , scene transformation is used when a transformation target is distributed over the entire area rather than a specific area (or specific object). To do this, the model needs to learn a domain transform for each element that can be displayed in a given image.

For the scene translation task, ACDC (Adverse Conditions Data set with Correspondences) data set may be utilized. This dataset represents images of driving scenes under various environmental conditions such as night, snow, rain, and fog. For each adverse condition image, the corresponding scene taken under normal conditions can be used. A data set can be used in an unpaired style, even if that data set provides an approximation of paired samples.

Although the ACDC dataset provides different data splits for training, validation, and testing, it is preferable to utilize images labeled “validation” and “test” for the training process in order to utilize most of the data for training.

8 (A) and (B) are tables comparing performance of scene conversion tasks between architectures for each task.

Referring to FIG. 8, for comparison between CycleGAN (CycleGAN), AGGAN (Attention-guide GAN), ASGIT (Attention-Based Spatial Guidance for Image-to-Image Translation) EAGAN, and UEAGAN architectures, different models of each architecture are used. learn The weights are initialized by a random seed and learn for each different method. The lowest KID (Kernel Inception Distance) score obtained by each model during training is selected as the criterion for performance, after which the average of the three KID (Kernel Inception Distance) scores is taken for each different method.

For reference, CycleGAN is a Generative Adversarial Network (GAN) used for image style transfer between domains.

AGGAN (Attention-guided GAN) differs from EAGAN in that it uses the same training process and similar architecture as CycleGan, and branches at two decoders that utilize an intermediate feature map generated by the end of the residual block after the residual block. there is.

ASGIT (Attention-Based Spatial Guidance for Image-to-Image Translation) uses the same generator and classifier network as CycleGAN, but constructs an attention map for each input image based on the second convolutional layer of the classifier.

8(A) and 8(B) show a comparison between different architectures in terms of KID (Kernel Inception Distance) scores for landscape and object translation tasks, respectively.

8(A) is a table showing performance results for scene conversion, and FIG. 9 is a diagram for visually comparing scene conversion for each architecture. As shown in FIG. 8(A) and FIG. 9, the EAGAN proposed in Example 1 shows the best performance except for the night-related task where the cycle GAN model performs best.

8(B) is a table showing performance results for entity transformation, and FIG. 10 is a diagram for visually comparing entity transformations for each architecture. As shown in FIG. 8(B) and FIG. 10, it can be seen that EAGAN and UEAGAN proposed in Example 1 showed excellent performance.

The EAGAN architecture showed the best performance for “apples to oranges” and “horses to zebras” tasks, while the UEAGAN architecture performed well for “oranges to apples” and “horses to zebras” tasks. " It showed equivalent performance to the EAGAN architecture in the task.

FIG. 11 is a diagram for precise visual inspection of an entity transformation model generated by the architecture, and FIG. 12 is a diagram for precise visual inspection of a scene transformation model generated by the architecture.

As shown in FIG. 11 , in the “conversion from horse to zebra” task, a specific area can be enlarged for detailed visual inspection. Although all images show the characteristic black and white stripes of zebras, CycleGAN still retains the eyebrow color of the original image, and AGGAN shows that the stripes are too thin to be realistic. AGGAN also converted the tail part, which should not have a striped pattern.

ASGIT shows some of the essential characteristics of a zebra, but there seems to be a difference in the color of the output.

EAGAN exhibits a relatively consistent stripe pattern, albeit with some artificial transformations in the tail.

UEAGAN exhibits noise at the boundaries of the magnification region and artifactual transitions in the tail.

As shown in FIG. 12, cycleGAN and AGGAN in the "SNOW to SNOW" task show that the torso of the subject (human) tends to blend with the surroundings, while the retention rate of the subject's feet is reduced.

ASGIT appears to have a higher retention rate of the subject's feet than CycleGAN and AGGAN, while having more prominent colors on the subject's torso. However, ASGIT shows that the entire image is noisy compared to other images.

EAGAN shows relatively low noise and high preservation rate of the subject's feet and torso area compared to other images.

The UEAGAN model appears to preserve most of the details, such as the subject's arms and facial regions better.

13 is a diagram illustrating an attention map of the EGAN architecture.

As shown in Figure 13, EAGAN creates an attention map in which the attention module focuses on different regions of the image. In EAGAN, each channel of the key vector in the attention module goes through the Soft-max function, subtracts the minimum value for each Attention Map, divides it by the maximum value, and expands it to the range of 0 and 1. And EAGAN can visualize the attention map as a gray scale image through this. For reference, the brighter the pixel, the higher focus is set for the corresponding input area.

For example, when EAGAN transforms objects in the second row, i.e. apples to oranges, the first attention map will focus on the apple, the second attention map will focus on the bush, and the third attention map will focus on the hand. The fourth attention map focuses on the sky and transforms apples into oranges.

<Example 3>

Embodiment 3 is a technique for generating an image using the adversarial attention network system of Embodiment 1 or Embodiment 2.

First, a method of generating an image using the EAGAN (Eficient Attention Generative Adversarial Network) architecture of the first embodiment in the image generating method will be described.

As shown in FIG. 14, the method of Example 3 includes extracting features (S110), changing features from a source domain to a target domain (S120), constructing a translation image (S130), and converting features to a desired dimension. Upscaling (S140) and mapping an output object to a multi-channel RGB image (S150).

In step S110 of extracting features, an encoder extracts meaningful features from an input image through a series of convolutions.

The encoder extracts meaningful features from the input image through a series of convolutions. For example, an image can be transformed into a latent feature space through a series of convolutions that down-sample the dimensions of the input.

The encoder includes an attention module that computes attention maps for resolving long-range dependancy.

The attention module is implemented as an improved attention module (eicient-attention modules) using a generator based on CycleGAN (CycleGan). For reference, CycleGAN is a Generative Adversarial Network (GAN) used for image style transfer between domains. Using CycleGAN, you can train an image of one area to transform into another area, and CycleGAN is performed by unsupervised learning. At this time, CycleGAN does not perform one-to-one mapping of images in both domains.

The attention module solves the long-range dependency problem by sharing dependencies between a source image and a target image using cross-domain attention-sharing mechanisms.

The attention module is either placed after the encoder's down-sample convolution and before the decoder's transposed-convolution (see FIG. 2(A)), or before the decoder's transposed-convolution (prior ) and after the transposed-convolution of the decoder (see FIG. 2(B)).

The performance results of the architecture in which the attention module is placed before ((A) in FIG. 2) and after ((B) in FIG. 2) the convolution block can be seen in FIG. It appears to improve performance for most tasks when placed before the convolution of , and after the transposed-convolution of the decoder.

On the other hand, the attention module interprets the key (K) of each channel as a single attention map in order to learn the input image from a semantic point of view, and a series of omitted connections between the key vectors of each of the attention modules ( skip connections are located.

The Attention module learns to represent semantic aspects of an image instead of corresponding to attention between two points in an image. Since the input and output images are expected to be highly content-related, we can infer that the attention between features in the initial layer i of the generator can be reused in the corresponding ending layers i-L.

Since the attention modules are computed on real samples, sharing these attention maps can be used to reconstruct the transformed image to produce more consistent and realistic output. A series of elided connections are then placed between each key vector in the mirrored Attention module block to share long-distance dependencies between the two ends of the architecture. The attention map of the key (K) network is shared between a pair of attention blocks.

A refinement network for refining the attention map from the original domain may be further positioned between a series of omitted connections located between key vectors of each of the attention modules.

Considering that the attention module interprets each channel as a single attention map, the refinement network is implemented as a depth-wise convolution using a 3 × 3 kernel. This special type of circuit divides the input channels into a given number of groups and uses a different kernel for each of these groups.

That is, the attention module interprets the main result as an attention map shared between the corresponding i and i-L layers, and the attention map is passed through group convolution so that information is not mixed between different channels.

In the step of changing features from the source domain to the target domain (S120), the translator changes the extracted features from the source domain to the target domain using a series of learned residual blocks.

A translator uses a series of residual blocks to learn how to change the extracted features from the source domain to the target domain. At this stage, the dimensions of the shape are not changed, and the number of remaining blocks can be used as a hyperparameter to strike a balance between speed and quality of the result.

In the step of constructing the translated image (S130), a decoder constructs a translated image having the same resolution as the image input to the encoder by using features translated by the translator.

In step S140 of upscaling the features to a desired dimension, a decoder utilizes a series of transposed convolutions to upscale the features to a desired dimension.

In the step of mapping the output to the multi-channel RGB image (S150), a decoder maps the output to the multi-channel RGB image in the last convolution layer to generate a converted image. The generated images may refer to FIGS. 9 to 12 .

When the EAGAN architecture is used in the adversarial attention network system, the encoder and decoder of the generator omit connection between the attention module on the ith layer of the encoder and the attention module on the i-L layer of the decoder to share the calculated attention map between them ( skip connections). Here, i is the layer order of the encoder and L is the layer number of the decoder. For reference, skip connections add the output of one layer to the input of the next layer by skipping several layers.

The location of the attention block can be set closer to the input and output layers. Because of this, when the generator introduces the attention mechanism, it can learn long-range dependencies from the source domain and then share these dependencies with the target domain to improve the consistency of image transformation.

Here features are set closer to the characteristics of each image domain. Therefore, the encoder and decoder appear as ideal locations for calculating long-range dependencies and can share attention maps through skipped connections between attention modules in layers i and i-L.

On the other hand, the adversarial attention network system introduces the learnable scale parameter (yi) of the formula below in order to depend more on local dependencies and wait until long-term dependencies become more meaningful in the initial training stage for dependency learning.

The scale parameter multiplies the output of the attention and then adds back the input function.

[Equation 2]

yi = γo _i + x _i

Here, oi is the output value of the i-th attention, x _i is the i-th input feature, and the weight gamma r is initially set to 0 and then set to a value greater than 0 as long-term dependencies are learned from meaningful features.

On the other hand, the adversarial attention network system learns two mapping functions to perform image translation between domain A and domain B, but two generators GA and GB and two discriminators DA and DB are domain The mapping functions are simultaneously trained to enhance inter-cycle consistency.

For reference, cycle consistency loss is a type of loss used in generative adversarial networks that perform unpaired image-to-image transformations.

Although the above has been described with reference to several embodiments of the present invention, those skilled in the art can make the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. It will be appreciated that various modifications and variations may be made.

In addition, the invention related to the method among the embodiments described above may be implemented as a program or a computer-readable recording medium in which the program is stored.

That is, the present invention can be implemented in the form of an application, implemented as a software program running on a mobile terminal such as a smartphone or tablet PC running on the basis of Google's Android or Apple's IOS, or a Google Glass, Apple Watch, Samsung It may be implemented as a software program running on a wearable device such as a Galaxy Watch or a smart watch, or may be implemented as a software program running on a laptop PC or desktop PC running based on Microsoft's Windows or Google's Chrome OS.

In addition, partial functions of the above-described device or system may be provided by being included in a computer-readable recording medium by tangibly implementing a program of instructions for implementing them. A computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. Included are hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, USB memory, and the like.

S110: Extracting features
S120: Changing features from the source domain to the target domain
S130: Step of constructing translation image
S140: Upscaling the features to the desired dimension
S150: Step of mapping the output to a multi-channel RGB image

Claims (18)

An encoder that extracts meaningful features from an input image through a series of convolutions;
A translator for changing the extracted features from a source domain to a target domain using a series of learned residual blocks;
A translation image having the same resolution as the image input to the encoder is constructed using the translated features, and the features are upscaled to a desired dimension using a series of transposed convolutions, and the last convolution layer converts outputs into multiple outputs. a generator including a decoder that maps to an RGB image of a channel; and
It includes a discriminator that discriminates the result of the generator,
The encoder includes an attention module that computes attention maps for resolving long-range dependancy,
The encoder and decoder,
In order to share the calculated attention map between them, skip connections are used between the attention module on the i-th layer of the encoder and the attention module on the iL-th layer of the decoder.
Improved adversarial attention network system. delete According to claim 1,
The attention module,
An improved adversarial attention network system disposed after down-sample convolution of the encoder and before transposed-convolution of the decoder, respectively. According to claim 1,
The attention module,
An improved adversarial attention network system disposed before the decoder's convolution and after the transposed-convolution of the decoder, respectively. According to claim 1,
The network system,
In the initial training phase for dependency learning, an improved adversarial attention network system that introduces a learnable scale parameter (yi) in the equation below to rely more on local dependencies and wait until long-term dependencies are more meaningful. According to claim 1,
The attention module,
In order to learn the input image from a semantic point of view, the key (K) of each channel is interpreted as a single attention map,
An improved adversarial attention network system in which a series of the omitted connections are located between key vectors of each of the attention modules. According to claim 6,
The improved adversarial attention network system further includes a refinement network for refining the attention map from the original domain between the series of skipped connections. According to claim 1,
The output of the attention module on the encoder side is concatenated with the input of the attention module on the decoder side,
The improved adversarial attention network system in which the attention module of the connected decoder operates based on the connected feature map of the previous layer and the output of the corresponding attention module of the encoder. According to claim 1,
The network system,
Learn two mapping functions to perform image translation between domain A and domain B,
An advanced adversarial attention network system in which two generators and two discriminators are trained concurrently to enforce cycle consistency across domains. An image generation method of an adversarial attention network system including an encoder, a translator and a decoder,
extracting, by the encoder, significant features from an input image through a series of convolutions;
changing the extracted features from a source domain to a target domain using a series of learned residual blocks by the translator;
constructing, by the decoder, a translated image having the same resolution as an image input to the encoder by utilizing the translated features;
the decoder utilizing a series of transposed convolutions to upscale the features to a desired dimension; and
Mapping, by the decoder, an output from a last convolution layer to a multi-channel RGB image;
The step of extracting the features includes calculating, by an attention module of the encoder, attention maps for resolving long-range dependancy;
The encoder and decoder,
In order to share the calculated attention map between them, skip connections are used between the attention module on the i-th layer of the encoder and the attention module on the iL-th layer of the decoder.
Image generation method of adversarial attention network system. delete According to claim 10,
The attention module,
The image generation method of the adversarial attention network system, which is disposed after the down-sample convolution of the encoder and before the transposed-convolution of the decoder, respectively. According to claim 10,
The attention module,
The image generation method of the adversarial attention network system, which is respectively disposed before the decoder's convolution and after the transposed-convolution of the decoder. According to claim 10,
The network system,
In the initial training stage for dependency learning, an image generation method of an adversarial attention network system that relies more on local dependencies and introduces the learnable scale parameter (yi) of the equation below to wait until long-term dependencies are more meaningful. According to claim 10,
The attention module,
In order to learn the input image from a semantic point of view, the key (K) of each channel is interpreted as a single attention map,
An image generation method of an adversarial attention network system in which a series of the omitted connections are located between key vectors of each of the attention modules. According to claim 15,
The method of generating an image of an adversarial attention network system, wherein a refinement network for refining the attention map from the original domain is further located between the series of omitted connections. According to claim 10,
The output of the attention module on the encoder side is concatenated with the input of the attention module on the decoder side,
The method of generating an image of an adversarial attention network system in which the attention module of the connected decoder operates based on a connected feature map of a previous layer and an output of a corresponding attention module of an encoder. According to claim 10,
The network system,
Learn two mapping functions to perform image translation between domain A and domain B,
An image generation method of an adversarial attention network system in which two generators and two discriminators are trained simultaneously to enhance cycle consistency between domains.