KR102611893B1 - Apparatus And Method For Generating Images Based On Generative Adversarial Networks Capable Of Dual Learning - Google Patents

Abstract

The present invention relates to a generative adversarial network (GAN)-based image generation apparatus and method capable of dual learning, and more specifically, to a first source encoder for extracting a first source feature from a first domain of a first source image, a first a first target encoder extracting first target features from a second domain of a target image, a second source encoder extracting second source features from a second domain of a second source image, 2 A second target encoder that extracts target features, a convolutional block attention module that extracts refine features that indicate where to focus in each source image from each source feature output from each source encoder, and a given query. A generative adversarial capable of double learning that includes a query attention module that generates temporary attention matrices for each refined feature, sorts each temporary attention matrix based on entropy, and then outputs the final attention matrix with the selected rows. This relates to a neural network (GAN)-based image generation device and method.

Description

Apparatus and method for generating images based on generative adversarial networks (GAN) capable of dual learning {Apparatus And Method For Generating Images Based On Generative Adversarial Networks Capable Of Dual Learning}

The present invention relates to a generative adversarial network (GAN)-based image generation device and method capable of dual learning, which enables learning for two domains simultaneously by arranging the generative adversarial network (GAN) symmetrically.

Image-to-Image (I2I) transformation is one of the fields of computer vision and machine learning that aims to generate images from a source domain to a target domain. This has been greatly developed with the emergence of Generative Adversarial Networks (GAN).

Generative adversarial network (GAN) is based on unsupervised learning, and a generator that generates virtual data from random noise can distinguish between authenticity and authenticity so that it can generate more realistic data. It is a model that is learned competitively by adding a discriminator. It is an efficient model that performs well on relatively small datasets. On the other hand, the generative adversarial network (GAN) has the disadvantage of being very unstable in that it competitively learns the generator and discriminator at the same time, and that it is difficult to train with large datasets.

In Image-to-Image (I2I) conversion, a generative adversarial network (GAN) can generate high-quality images through supervised learning using data that pairs input and output values. However, there are technical limitations that make it quite difficult to collect paired data through pixel-to-pixel mapping for supervised learning, and that constructing such data is quite expensive.

In Image-to-Image (I2I) conversion, generative adversarial network (GAN) is also capable of unsupervised learning using prior information, i.e. data without labels, but the adversarial loss is not limited and several There are possible mappings that may produce images of poor quality. CycleGAN, DiscoGAN, and DualGAN, which were developed to solve this problem, make cycle consistency assumptions, which can also limit their ability to perform shape changes and do not always make the relationship between the two domains an ideal bijection. .

DCLGAN and QS-Attn were developed to complement the technical limitations of CycleGAN, DiscoGAN, and DualGAN. DCLGAN does not select a target anchor in contrastive learning, and each anchor feature has a problem in that its receptive field is limited without considering the relationship with other positions. QS-Attn uses one embedding for two separate domains, which may not capture domain gaps, reducing training stability. And when QS-Attn obtains an anchor feature, it calculates the entropy distribution as a metric to measure how important the feature is in reflecting the domain characteristics, then sorts the entropy to select the smallest N points in the image. When we analyzed the entropy metric, we found zero entropy points that exceeded the number of samples, which meant that important feature selection was missing. In other words, in QS-Attn, missing features may be concentrated in parts that correspond to the background rather than the target.

Accordingly, technology that can generate better quality images using GAN capable of unsupervised learning in Image-to-Image (I2I) conversion is urgently needed in this technical field.

Republic of Korea Patent Publication No. 10-2023-0070128 Republic of Korea Patent Publication No. 10-1975186

The present invention is intended to solve the above problems, and symmetrically deploys a generative adversarial network (GAN) to generate better quality images based on unsupervised learning, enabling simultaneous learning in two domains. The purpose is to obtain a generative adversarial network (GAN)-based image generation device and method capable of double learning.

The technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description of the present invention.

In order to achieve the above object, the generative adversarial network (GAN)-based image generation apparatus capable of dual learning of the present invention includes a first source encoder for extracting a first source feature from a first domain of a first source image; a first target encoder that extracts a first target feature from a second domain of the first target image; a second source encoder for extracting second source features from a second domain of the second source image; a second target encoder extracting second target features from the first domain of the second target image; A convolutional block attention module that extracts refine features indicating where to focus in each source image from each source feature output from each source encoder; and a query attention module that generates a temporary attention matrix for each refined feature using a given query, sorts each temporary attention matrix based on entropy, and then outputs a final attention matrix with the selected row. do.

In order to achieve the above object, the generative adversarial network (GAN)-based image generation method capable of dual learning of the present invention is a first source feature in which the first source feature is extracted from the first domain of the first source image by the first source encoder. 1 Source feature extraction step; A first target feature extraction step of extracting a first target feature from a second domain of the first target image by a first target encoder; A second source feature extraction step of extracting second source features from a second domain of the second source image by a second source encoder; A second target feature extraction step of extracting a second target feature from the first domain of the second target image by a second target encoder; A refine feature extraction step in which refine features indicating where to focus in each source image are extracted from each source feature output from each source encoder by a convolutional block attention module; A temporary attention matrix generation step in which a temporary attention matrix for each refined feature is generated by a query attention module using a given query; and a final attention matrix output step in which, by the query attention module, each temporary attention matrix is sorted based on entropy and then a final attention matrix having the selected row is output.

As described above, according to the present invention, a generative adversarial network (GAN) is arranged symmetrically to enable learning for two domains simultaneously, which has the effect of generating better quality images based on unsupervised learning.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the detailed description and claims.

Figure 1 is a configuration diagram of a generative adversarial network (GAN)-based image generation device capable of dual learning according to the present invention.
Figure 2 is a diagram showing a first source encoder that processes a first source image and a convolutional block attention module that processes first source features according to an embodiment of the present invention.
Figure 3 is a diagram showing a second source encoder that processes a second source image and a convolutional block attention module that processes second source features according to an embodiment of the present invention.
Figure 4 is a diagram showing a query attention module that processes the first refine feature and a first target encoder and a learning unit that process the first target image according to an embodiment of the present invention.
Figure 5 is a diagram showing a query attention module that processes second refine features and a second target encoder and learning unit that process a second target image according to an embodiment of the present invention.
Figure 6 is a flowchart of a generative adversarial network (GAN)-based image generation method capable of dual learning of a first source image and a first target image according to an embodiment of the present invention.
Figure 7 is a flowchart of a generative adversarial network (GAN)-based image generation method capable of dual learning of a second source image and a second target image according to an embodiment of the present invention.

The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in the present application, should not be interpreted in an idealized or excessively formal sense. No.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Figure 1 is a configuration diagram of a generative adversarial network (GAN)-based image generation device capable of dual learning according to the present invention.

Figure 2 shows a first source encoder 500 that processes the first source image (I _x ) and a convolutional block attention module 900a that processes the first source feature (F _{x_1} ) according to an embodiment of the present invention. ) is a drawing showing. Figure 3 shows a second source encoder 700 that processes a second source image (I _y ) and a convolutional block attention module (900b) that processes a second source feature (F _{y_2} ) according to an embodiment of the present invention. ) is a drawing showing.

Figure 4 shows a query attention module 1000a that processes the first refined feature (Refined F _{x_1} ) and a first target encoder 600 that processes the first target image (G _y ) according to an embodiment of the present invention. This is a diagram showing the learning unit 1100a. Figure 5 shows a query attention module 1000b that processes the second refined feature (Refined F _{y_2} ) and a second target encoder 800 that processes the second target image (G _x ) and learning according to an embodiment of the present invention. This is a diagram showing part 1100b.

Figure 6 is a flowchart of a generative adversarial network (GAN)-based image generation method capable of dual learning on the first source image (I _x ) and the first target image (G _y ) according to an embodiment of the present invention. Figure 7 is a flowchart of a generative adversarial network (GAN)-based image generation method capable of dual learning of a second source image (I _y ) and a second target image (G _x ) according to another embodiment of the present invention.

Generative adversarial network (GAN)-based image generation device capable of double learning

Referring to Figure 1, the generative adversarial network (GAN)-based image generation device capable of dual learning of the present invention includes a first source encoder 500, a first target encoder 600, a second source encoder 700, and a second It includes a target encoder 800, convolutional block attention modules 900a and 900b, and query attention modules 1000a and 1000b.

First, the first source encoder 500 extracts a first source feature (F _{x_1} ) from the first domain (x) of the first source image (I _x ). The first target encoder 600 extracts the first target feature (F _{y_1} ) from the second domain (y) of the first target image (G _y ).

Next, the second source encoder 700 extracts the second source feature (F _{y_2} ) from the second domain (y) of the second source image (I _y ). The second target encoder 800 extracts a second target feature (F _{x_2} ) from the first domain (x) of the second target image (G _x ).

Next, the convolutional block attention modules (900a, 900b) determine where in each source image (I _x , I _y ) from each source feature (F _{x_1} , F _{y_2} ) output from each source encoder (500, 700). Refined features (Refined F _{x_1} , Refined F _{y_2} ) indicating whether to focus are extracted respectively.

Looking at the embodiment of FIG. 2, the first source feature (F _{x_1} ) output from the first source encoder 500 is input to one convolutional block attention module 900a. Here, the first source feature (F _{x_1} ) may have a size of height (H) * width (W) * channel (C). One convolutional block attention module 900a may include one channel attention module 910a and one spatial attention module 920a.

First, the one channel attention module 910a is used to encode which channel should be focused more by examining the relationship between channels of the first source feature (F _{x_1} ). This is as shown in [Equation 1] below.

[Equation 1]

First, one channel attention module 910a can output only the maximum value for each channel from the first source feature (F _{x_1} ) as a vector through max pooling. And one channel attention module 910a can reduce the dimension of the maximum pooled vectors through a multi-layer perceptron (MLP) with two layers. One channel attention module 910a can convert dimension-reduced vectors into probability values using a sigmoid function. And one channel attention module 910a can encode the first source feature (F _{x_1} ) by focusing on the channel with the largest probability value among the probability values for each channel.

Accordingly, one convolutional block attention module 900a multiplies the first source feature ( _F Source features (Channel_refined F _{x_1} ) can be output.

In addition, one spatial attention module 920a is used to encode which position should be focused more by examining the positional relationship among a plurality of pixels in the H*W*1 size of the channel refined first source feature (Channel_refined F _{x_1} ). am. This is the same as [Equation 2] below.

[Equation 2]

First, a channel-refined first source feature (Channel_refined F _{x_1} ) may be input to one spatial attention module 920a. One spatial attention module 920a can output only the maximum value from the channel refined first source feature (Channel_refined F _{x_1} ) as a vector through max pooling targeting the size of H*W*1. And one spatial attention module 920a can generate a feature map of size H*W*2 by connecting two channels of size H*W*1. One spatial attention module 920a can perform a 7*7 convolution operation because the feature map is 2 channels. One spatial attention module 920a can convert the feature map calculated by 7*7 convolution into a probability value using a sigmoid function. And one spatial attention module 920a can encode the channel-refined first source feature (Channel_refined F _{x_1} ) by focusing on the position with the largest probability value among the probability values.

Accordingly, one convolutional block attention module 900a combines a channel-refined first source feature (Channel_refined F _{x_1} ) and a channel-refined first source feature (Channel_refined By multiplying F _{x_1} ), one refined feature (Refined F _{x_1} ) can be output.

Looking at the embodiment of FIG. 3, the second source feature (F _{y_2} ) output from the second source encoder 700 is input to another convolutional block attention module 900b. Here, the second source feature (F _{y_2} ) may have a size of height (H) * width (W) * channel (C). Another convolutional block attention module 900b may include another channel attention module 910b and another spatial attention module 920b.

First, another channel attention module 910b is used to encode which channel should be focused more by examining the relationship between channels of the second source feature (F _{y_2} ). This is the same as [Equation 3] below.

[Equation 3]

First, another channel attention module 910b can output only the maximum value for each channel from the second source feature (F _{y_2} ) as a vector through max pooling. Next, another channel attention module 910b can reduce the dimension of the maximum pooled vectors through a multi-layer perceptron (MLP) with two layers each. Another channel attention module 910b can convert dimension-reduced vectors into probability values using a sigmoid function. And another channel attention module 910b may encode the second source feature (F _{y_2} ) by focusing on the channel with the largest probability value among the probability values for each channel.

Accordingly, another convolutional block attention module 900b multiplies the second source feature (F _{y_2} ) and the second source feature in which the channel is encoded from another channel attention module 910b to obtain the channel The refined second source feature (Channel_refined F _{y_2} ) can be output.

In addition, another spatial attention module 920b encodes which position to focus more on by looking at the positional relationship among a plurality of pixels in the H*W*1 size of the channel refined second source feature (Channel_refined F _{y_2} ). This is to do it. This is the same as [Equation 4] below.

[Equation 4]

First, a channel-refined second source feature (Channel_refined F _{y_2} ) may be input to another spatial attention module 920b. Another spatial attention module 920b can output only the maximum value from the channel-refined second source feature (Channel_refined F _{y_2} ) as a vector through max pooling targeting the size of H*W*1. there is. And another spatial attention module 920b can generate a feature map of size H*W*2 by connecting two channels of size H*W*1. Another spatial attention module 920b can perform a 7*7 convolution operation because the feature map is 2 channels. Another spatial attention module 920b can convert the feature map calculated by 7*7 convolution into a probability value using a sigmoid function. And another spatial attention module 920b can encode the channel-refined second source feature (Channel_refined F _{y_2} ) by focusing on the position with the largest probability value among the probability values.

Accordingly, another convolutional block attention module 900b uses a channel-refined second source feature (Channel_refined F _{y_2} ) and a second channel-refined channel-refined source feature whose position is encoded from another spatial attention module 920b. Another refined feature (Refined F _{y_2} ) can be output by multiplying the source feature (Channel_refined F _{y_2} ).

Next, the query attention modules 1000a and 1000b generate temporary attention matrices for each refined feature (Refined F _{x_1} , Refined F _{y_2} ) using a given query, and each temporary attention matrix is generated based on entropy. After sorting the matrix, output the final attention matrix with the selected rows.

Preferably, the query attention modules 1000a and 1000b calculate the value of each refined feature (Refined F _{x_1} , Refined F _{y_2} ) and each final attention matrix to output a source attention matrix, respectively. Comprising units 1010a and 1010b and target attention matrix output units 1020a and 1020b that calculate the value of each target feature (F _{y_1} , F _{x_2} ) and each final attention matrix and output the target attention matrix, respectively. It is characterized by

Next, the learning units 1100a and 1100b calculate the similarity of pairs in each source attention matrix and each target attention matrix, and learn to reduce the loss value of each final attention matrix using a loss function. It is characterized by

Generally, three vectors can be used in attention operations: query, key, and value. Query is the token at the current time, and key and value are the target tokens for which attention is sought. In other words, a query is a fixed token, and the key and value are searched from beginning to end to find the token that best matches the query, that is, the token with the highest attention. will be. Query, key, and value have the same size.

4, one query attention module 1000a converts rows and columns using a query with a size of (n × d _k ) to create a key with a size of (d _k × n). It can be multiplied by (key). And one query attention module 1000a can convert the previously multiplied value into a normalized value between 0 and 1 using the softmax function. At this time, adding up all the normalized values with the Softmax function becomes 1.

That is, each pixel of one temporary attention matrix with a size (n×n) output from one query attention module 1000a has a probability of belonging to a specific class in the overall probability, and has a normalized value between 0 and 1. have One temporary attention matrix may be displayed in different colors for each class to which each pixel belongs, with higher values in the same class being displayed in darker colors, and lower values in the same class being displayed in lighter colors. You can. Here, the class may include positive and negative.

Additionally, one query attention module 1000a can sort one temporary attention matrix with a size of (n×n) based on entropy and output one final attention matrix with the selected row. For example, there may be d _k selected rows. That is, one final attention matrix may have a size of (d _k × n).

In addition, the one source attention matrix output unit 1010a outputs a value having a size of (n×d _k ) for one refined feature (Refined_F _{x_1} ) and a final attention matrix having a size of (d _k ×n). By multiplying, one source attention matrix with a size of (n×n) can be output.

In addition, one target attention matrix output unit 1020a outputs a value (Value) having a size of (n×d _k ) and (d) for the first target feature (F _{y_1} ) output from the first target image (G _y ). One final attention matrix with a size of _k × n) can be multiplied to output a target attention matrix with a size of (n × n).

In addition, one learning unit 1110a is a source attention matrix with a size of (n × n) and a target attention matrix with a size of (n × n). The similarity of a pair can be calculated. Additionally, the learning unit 1110a is based on contrastive learning and can learn to reduce the loss value of one final attention matrix using a loss function (x).

Looking at one embodiment of Figure 5, another query attention module 1000b uses a query with a size of (n × d _k ) and a key with a size of (d _k × n) It can be multiplied. And another query attention module 1000b can convert the previously multiplied value into a normalized value between 0 and 1 using the softmax function. At this time, adding up all the normalized values with the Softmax function becomes 1.

That is, each pixel of another temporary attention matrix with a size (n×n) output from another query attention module 1000b has a probability of belonging to a specific class in the overall probability, between 0 and 1. It has normalized values. Another temporary attention matrix may be displayed in different colors for each class to which each pixel belongs, with higher values in the same class being displayed in darker colors, and lower values in the same class being displayed in lighter colors. can be displayed. Here, the class may include positive and negative.

Additionally, another query attention module 1000b can sort another temporary attention matrix with a size of (n×n) based on entropy, and create another final attention matrix with the selected row. Can be printed. For example, there may be d _k selected rows. That is, another final attention matrix may have a size of (d _k × n).

In addition, another source attention matrix output unit 1010b has a value having a size of (n×d _k ) and (d _k ×n) for another refined feature (Refined_F _{y_2} ). Another final attention matrix can be multiplied to output another source attention matrix having a size of (n×n).

In addition, another target attention matrix output unit 1020b outputs a value having a size of (n×d _k ) for the second target feature (F _{x_2} ) output from the second target image (G _x ) and Another final attention matrix with a size of (d _k × n) can be multiplied to output another target attention matrix with a size of (n × n).

In addition, another learning unit 1110b can calculate the similarity of pixel pairs in another source attention matrix and another target attention matrix. there is. Additionally, the learning unit 1110b is based on contrastive learning and can learn to reduce the loss value of another final attention matrix using a loss function (y).

Looking again at Figure 1, the present invention includes one generative adversarial network (GAN) including a first generator 100 and a first discriminator 200, a second generator 300, and a second discriminator 400. One generative adversarial network (GAN) including one can be placed symmetrically.

Here, the first generator 100 may generate a first target image (G _y ) including the second domain (y) from the first source image (I _x ). The first generator 100 may include a first generator encoder 110 for encoding at the front end and a first generator decoder 120 for decoding at the end.

Next, the second generator 300 may generate a second target image (G _x ) including the first domain (x) from the second source image (I _y ). It may include a second generator encoder 310 for encoding at the front end of the second generator 300 and a second generator decoder 320 for decoding at the end.

Next, the first discriminator 200 can determine authenticity between the first source image (I _x ) and the second target image (G _x ). That is, the first discriminator 200 determines whether the second target image (G _x ) generated from the second generator 300 is real or fake. And the learning goal of the second generator 300 is to generate a second target image (G _x ) that the first discriminator 200 cannot distinguish between real and fake.

Next, the second discriminator 400 can determine authenticity between the second source image (I _y ) and the first target image (G _y ). That is, the second discriminator 400 determines whether the first target image (G _y ) generated from the first generator 100 is real or fake. And the first generator 100 has a learning goal of generating a first target image (G _y ) that the second discriminator 400 cannot distinguish between real and fake.

Generative adversarial network (GAN)-based image generation method capable of double learning

First, according to an embodiment of the present invention, the image generation method based on a generative adversarial network (GAN) capable of dual learning of the present invention generates a first image from the first source image (I _x ) by the first generator 100. A first target image generation step (S100) in which a first target image (G _y ) including two domains (y) is generated, and a first image is generated from the second source image (I _y ) by the second generator 300. It may include a second target image generation step (S200) in which a second target image (G _x ) including one domain (x) is generated.

In addition, according to one embodiment of the present invention, the first discriminator 200 determines whether the authenticity of the first source image (I _x ) and the second target image (G _x ) is determined. A determination step (S1400) and a second determination step (S1500) in which authenticity between the second source image (I _y ) and the first target image (G _y ) is determined by the second discriminator 400. More may be included.

That is, the first determination step (S1400) determines whether the second target image (G _x ) generated from the second generator 300 is real or fake. And the learning goal of the present invention is to generate a second target image ( _G Do this.

And in the second determination step (S1500), it is determined whether the first target image (G _y ) generated from the first generator 100 is real or fake. And the learning goal of the present invention is to generate a first target image (G _y ) that cannot distinguish whether the second discriminator 400 is real or fake from the second determination step (S1500). Do this.

(1) The first source image (I _x ) and the first target image (G _y ) learning about

Looking at the embodiment of FIG. 6, with respect to the first source image (I _x ) and the first target image (G _y ), the present invention encodes the first source image (I A first source feature extraction step ( _S300 ) in which the first source feature (F _{x_1} ) is extracted from the first domain ( _x ) of It includes a first target feature extraction step (S400) in which the first target feature (F _{y_1} ) is extracted from the second domain (y) of.

Next, the present invention converts each source image (I _x , _I ) from each source feature ₍ F It further includes a refine feature _extraction step (S700a, S700b) in which refine features (Refined F _{x_1} , Refined F _{y_2} ) indicating where to focus in y) are extracted, respectively.

In the refine feature extraction step (S700a) according to one embodiment, one refined feature (Refined F _{x_1} ) indicating where to focus in the first source image (I _x ) can be extracted from the first source feature (F _{x_1} ). there is. Here, the first source feature (F _{x_1} ) may have a size of height (H) * width (W) * channel (C).

The refine feature extraction step (S700a) according to one embodiment examines the relationship between channels of the first source feature (F _{x_1} ) by the channel attention module 910a in one convolutional block attention module 900a. It is characterized by including a channel attention step (S710a) in which it is encoded whether more focus should be placed on the channel. This is the same as [Equation 1] above.

First, in the channel attention step (S710a) according to one embodiment, only the maximum value for each channel in the first source feature (F _{x_1} ) can be output as a vector through max pooling. Next, in the channel attention step (S710a) according to one embodiment, the dimension of the maximum pooled vectors can be reduced through a multi-layer perceptron (MLP) having two layers. In the channel attention step (S710a) according to one embodiment, vectors with reduced dimensions may be converted into probability values using a sigmoid function. And in the channel attention step (S710a) according to one embodiment, the first source feature (F _{x_1} ) may be encoded by focusing on the channel with the largest probability value among the probability values for each channel.

Accordingly, in the channel attention step (S710a) according to one embodiment, the first source feature ( _F Source features (Channel_refined F _{x_1} ) may be output.

In addition, the refine feature extraction step (S700a) according to one embodiment is performed by the spatial attention module (920a) within one convolutional block attention module (900a). It further includes a spatial attention step (S720a) in which positional relationships among a plurality of pixels are examined in the H*W*1 size of the channel-refined first source feature (Channel_refined F _{x_1} ) to encode which position to focus on more. Do it as This is the same as [Equation 2] above.

First, in the spatial attention step (S720a) according to one embodiment, the channel refined first source feature (Channel_refined F _{x_1} ) may be input. The spatial attention step (S720a) according to one embodiment outputs only the maximum value from the channel-refined first source feature (Channel_refined F _{x_1} ) as a vector through max pooling targeting the size of H*W*1. It can be. And in the spatial attention step (S720a) according to one embodiment, two channels with sizes of H*W*1 are connected to create a feature map of size H*W*2. In the spatial attention step (S720a) according to one embodiment, since the feature map is 2 channels, a 7*7 convolution operation can be performed. In the spatial attention step (S720a) according to one embodiment, a sigmoid function is used so that the 7*7 convolution calculated feature map can be converted into a probability value. And in the spatial attention step (S720a) according to one embodiment, the channel-refined first source feature (Channel_refined F _{x_1} ) may be encoded by focusing on the position with the largest probability value among the probability values.

Accordingly, the spatial attention step (S720a) according to one embodiment includes a channel-refined first source feature (Channel_refined F _{x_1} ) and a channel-refined first source feature (Channel_refined F _{x_1} ) may be multiplied to output one refined feature (Refined F _{x_1} ).

Next, the present invention includes a temporary attention matrix generation step in which a given query is used by the query attention modules 1000a and 1000b to generate temporary attention matrices for each refined feature (Refined F _{x_1} , F _{y_2} ). (S800a, S800b) and the query attention module (1000a, 1000b), each temporary attention matrix is sorted based on entropy, and then the final attention matrix with the selected row is output, respectively (S900a, S900b). ) further includes.

In the temporary attention matrix generation step (S800a) according to one embodiment, a query having a size of (n×d _k ) may be used and multiplied by a key having a size of (d _k ×n). And in the temporary attention matrix generation step (S800a) according to one embodiment, a softmax function may be used to convert the previously multiplied value into a normalized value between 0 and 1. At this time, adding up all the normalized values with the Softmax function becomes 1.

That is, each pixel of one temporary attention matrix with a size (n×n) output from one query attention module 1000a has a probability of belonging to a specific class in the overall probability, and has a normalized value between 0 and 1. have One temporary attention matrix may be displayed in different colors for each class to which each pixel belongs, with higher values in the same class being displayed in darker colors, and lower values in the same class being displayed in lighter colors. You can. Here, the class may include positive and negative.

Additionally, in the final attention matrix generation step (S900a) according to one embodiment, one temporary attention matrix with a size of (n×n) can be sorted based on entropy, and one final attention matrix with the selected row is output. It can be. For example, there may be d _k selected rows. That is, one final attention matrix may have a size of (d _k × n).

Next, the present invention calculates the value of each refined _{feature (} Refined F By the source attention matrix output step (S1000a, 1000b) and the query attention module (1000a, 1000b), the value (Value) of each target feature (F _{y_1} , F _{x_2} ) and each final attention matrix are calculated to create a target attention matrix. It may further include target attention matrix output steps (S1100a, S1100b), respectively.

In the source attention matrix output step (S1000a) according to one embodiment, a value (Value) having a size of (n×d _k ) for one refined feature (Refined_F _{x_1} ) is generated by one source attention matrix output unit (1010a). and the final attention matrix having a size of (d _k × n) may be multiplied to output one source attention matrix having a size of (n × n).

In addition, the target attention matrix output step (S1100a) according to one embodiment is the first target feature (F _{y_1} ) output from the first target image (G _y ) by one target attention matrix output unit (1020a). A value with size (n×d _k ) is multiplied by a final attention matrix with size (d _k ×n), resulting in a target attention matrix with size (n×n). can be printed.

Next, the present invention includes a similarity calculation step (S1200a, 1200b) in which the similarity of pairs within each source attention matrix and each target attention matrix is calculated by the learning units (1100a, 1100b), and the learning unit (1100a, By 1100b), a learning step (S1300a, S1300b) may be further included in which a loss function is used to learn to reduce the similarity of positive pairs and to learn to increase the similarity of negative pairs. there is.

In the similarity calculation step (S1200a) according to one embodiment, the similarity of a pair within one source attention matrix and one target attention matrix may be calculated. And in the learning step (S1300a) according to one embodiment, a loss function can be used to learn to decrease the similarity of positive pairs and to increase the similarity of negative pairs.

(2) second source image (I _y ) and the second target image (G _x ) learning about

Looking at the embodiment of FIG. 7, with respect to the second source image (I _y ) and the second target image (G _x ), the present invention provides a second source image (I The second source feature extraction step ( _S500 ) in which the second source feature (F _{y_2} ) is extracted from the second domain (y) of the second target image (G _x ) by the second target encoder 800. It includes a second target feature extraction step (S600) in which the second target feature (F _{x_2} ) is extracted from the first domain (x).

The refine feature extraction step (S700b) according to another embodiment is a refinement feature (Refined F _{y_2} ) that indicates where to focus in the second source image (I _x ) from the second source feature (F _{y_2} ). This can be extracted. Here, the second source feature (F _{y_2} ) may have a size of height (H) * width (W) * channel (C).

The refine feature extraction step (S700b) according to another embodiment is performed by the channel attention module (910b) in another convolutional block attention module (900b), between channels of the second source feature (F _{y_2} ). It is characterized by including a channel attention step (S710b) in which which channel to focus more on is encoded by examining the relationship. This is the same as [Equation 1] above.

First, in the channel attention step (S710b) according to another embodiment, only the maximum value for each channel in the second source feature (F _{y_2} ) can be output as a vector through max pooling. Next, in the channel attention step (S710b) according to another embodiment, the dimension of the maximum pooled vectors can be reduced through a multi-layer perceptron (MLP) with two layers. there is. In the channel attention step (S710b) according to another embodiment, vectors with reduced dimensions can be converted into probability values using a sigmoid function. And in the channel attention step (S710b) according to another embodiment, the second source feature (F _{y_2} ) may be encoded by focusing on the channel with the largest probability value among the probability values for each channel.

Accordingly, in the channel attention step (S710b) according to another embodiment, the second source feature (F _{y_2} ) and the second source feature in which the channel is encoded from another channel attention module 910b are multiplied to obtain a channel The refined second source feature (Channel_refined F _{y_2} ) may be output.

In addition, the refine feature extraction step (S700b) according to another embodiment is performed by the spatial attention module (920b) within one convolutional block attention module (900b). It further includes a spatial attention step (S720b) in which positional relationships among a plurality of pixels are examined in the H*W*1 size of the channel-refined second source feature (Channel_refined F _{y_2} ) to encode which position to focus more on. Do it as This is the same as [Equation 2] above.

First, the channel-refined first source feature (Channel_refined F _{y_2} ) may be input to the spatial attention step (S720b) according to another embodiment. In the spatial attention step (S720b) according to another embodiment, only the maximum value in the channel-refined second source feature (Channel_refined F _{y_2} ) is vector through max pooling targeting the size of H*W*1. It can be output as . And in the spatial attention step (S720b) according to another embodiment, two channels with sizes of H*W*1 are connected to generate a feature map of size H*W*2. In the spatial attention step (S720b) according to another embodiment, a 7*7 convolution operation can be performed because the feature map is 2 channels. In the spatial attention step (S720b) according to another embodiment, a sigmoid function can be used to convert the 7*7 convolution calculated feature map into a probability value. And in the spatial attention step (S720b) according to another embodiment, the channel-refined second source feature (Channel_refined F _{y_2} ) may be encoded by focusing on the position with the largest probability value among the probability values.

Accordingly, the spatial attention step (S720b) according to another embodiment is a channel-refined second source feature (Channel_refined F _{y_2} ) and a second channel-refined source feature whose position is encoded from another spatial attention module 920b. The source feature (Channel_refined F _{y_2} ) may be multiplied to output another refined feature (Refined F _{y_2} ).

Next, in the temporary attention matrix generation step (S800b) according to another embodiment, a query with a size of (n × d _k ) is used and multiplied by a key with a size of (d _k × n). It can be. And in the temporary attention matrix generation step (S800b) according to another embodiment, a softmax function may be used to convert the previously multiplied value into a normalized value between 0 and 1. At this time, adding up all the normalized values with the Softmax function becomes 1.

That is, each pixel of another temporary attention matrix with size (n×n) output from one query attention module 1000b has a probability of belonging to a specific class in the overall probability, and a normalized probability between 0 and 1. It has value. Another temporary attention matrix may be displayed in different colors for each class to which each pixel belongs, with higher values in the same class being displayed in darker colors, and lower values in the same class being displayed in lighter colors. can be displayed. Here, the class may include positive and negative.

Next, in the final attention matrix generation step (S900b) according to another embodiment, another temporary attention matrix with a size of (n×n) can be sorted based on entropy, and another temporary attention matrix with the selected row is generated. One final attention matrix may be output. For example, there may be d _k selected rows. That is, another final attention matrix may have a size of (d _k × n).

Next, the source attention matrix output step (S1000b) according to another embodiment is (n×d) for another refined feature (Refined_F _{y_2} ) by another source attention matrix output unit (1010b). A value having a size of _k ) and a final attention matrix having a size of (d _k × n) may be multiplied to output another source attention matrix having a size of (n × n).

Next, the target attention matrix output step (S1100b) according to another embodiment includes the second target feature (G x ) output from the second target image (G _x ) by another target attention matrix output unit (1020b). A value with size (n×d _k ) for F _{x_2} ) and another final attention matrix with size (d _k A target attention matrix may be output.

Next, in the similarity calculation step (S1200b) according to another embodiment, the similarity of a pair within another source attention matrix and another target attention matrix may be calculated. And in the learning step (S1300b) according to another embodiment, a loss function can be used to learn to decrease the similarity of positive pairs and to increase the similarity of negative pairs.

Embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented as software, firmware, middleware, or microcode, program code or code segments that perform necessary tasks may be stored in a computer-readable storage medium and executed by one or more processors.

And aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as program modules or components that are executed by a computer. Typically, program modules or components include routines, programs, objects, and data structures that perform specific tasks or implement specific data types. Aspects of the subject matter described herein may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in an order different from the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or in a different configuration. Appropriate results may be achieved through substitution or substitution by elements or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

100.. First constructor
110.. First generator encoder
120.. First generator decoder
200.. First discriminator
300.. Second constructor
310.. Second generator encoder
320.. Second generator decoder
400.. Second discriminator
500.. 1st source encoder
600.. First target encoder
700.. Second source encoder
800.. Second target encoder
900a, 900b.. Convolutional block attention module
910a, 910b.. Channel attention module
920a, 920b.. Spatial attention module
1000a, 1000b.. Query attention module
1010a, 1010b.. Source attention matrix output unit
1020a, 1020b.. Target attention matrix output unit
1100a, 1100b.. Learning Department
S100.. First target image generation step
S200.. Second target image generation step
S300.. First source feature extraction step
S400.. First target feature extraction step
S500.. Second source feature extraction step
S600.. Second target feature extraction step
S700a, S700b.. Refine feature extraction stage
S710a, S710b.. Channel attention stage
S720a, S720b.. Spatial attention stage
S800a, S800b.. Temporary attention matrix creation step
S900a, S900b.. Final attention matrix output stage
S1000a, S1000b.. Source attention matrix output stage
S1100a, S1100b.. Target attention matrix output stage
S1200a, S1200b.. Similarity calculation step
S1300a, S1300b.. learning stage
S1400.. First discrimination step
S1500.. 2nd determination step

Claims (8)

a first source encoder that extracts first source features from a first domain of the first source image;
a first target encoder that extracts a first target feature from a second domain of the first target image;
a second source encoder for extracting second source features from a second domain of the second source image;
a second target encoder extracting second target features from the first domain of the second target image;
A convolutional block attention module that encodes each source feature and extracts each refined feature according to probability values for the channel and pixel position of each source feature output from each source encoder; and
By performing an attention operation on each refined feature using a given query, a temporary attention matrix is generated in which the probability value of belonging to a specific class is displayed for each pixel. Each temporary attention matrix is sorted based on entropy, and then the selected row is selected. A generative adversarial network (GAN)-based image generation device capable of dual learning, including a query attention module that outputs a final attention matrix having . According to clause 1,
a first generator generating the first target image including the second domain from the first source image so that the first target image is input to the first target encoder;
a second generator generating the second target image including the first domain from the second source image so that the second target image is input to the second target encoder;
a first discriminator that determines authenticity between the first source image and the second target image generated from the second generator; and
A generative adversarial network (GAN)-based image generation capable of dual learning, comprising a second discriminator that determines authenticity between the second source image and the first target image generated from the first generator. Device. According to clause 1,
The query attention module is,
a source attention matrix output unit that calculates the value of each refined feature and each final attention matrix and outputs each source attention matrix; and
A generative adversarial network (GAN)-based image generation capable of dual learning, comprising a target attention matrix output unit that calculates the value of each target feature and each final attention matrix and outputs each target attention matrix. Device. According to clause 3,
A learning unit that calculates the similarity of pairs in each source attention matrix and each target attention matrix and learns to reduce the loss value of each final attention matrix using a loss function. A generative adversarial network (GAN)-based image generator capable of learning. A first source feature extraction step of extracting first source features from a first domain of a first source image by a first source encoder;
A first target feature extraction step of extracting a first target feature from a second domain of the first target image by a first target encoder;
A second source feature extraction step of extracting second source features from a second domain of the second source image by a second source encoder;
A second target feature extraction step of extracting a second target feature from the first domain of the second target image by a second target encoder;
A refine feature extraction step in which each source feature is encoded and refined features are extracted according to probability values for the channel and pixel position of each source feature output from each source encoder by a convolutional block attention module;
A temporary attention matrix generation step in which a given query is used by the query attention module to perform an attention operation on each refined feature, thereby generating temporary attention matrices in which each pixel displays a probability value of belonging to a specific class; and
A final attention matrix output step in which each temporary attention matrix is sorted based on entropy by the query attention module and then a final attention matrix with selected rows is output. Based on a generative adversarial network (GAN) capable of double learning, including: How to create an image. According to clause 5,
A source attention matrix output step in which the value of each refined feature and each final attention matrix are calculated by the query attention module to output a source attention matrix, respectively; and
A target attention matrix output step in which the value of each target feature and each final attention matrix are calculated by the query attention module to output the target attention matrix, respectively. A generative method capable of double learning, further comprising: An adversarial neural network (GAN)-based image generation method. According to clause 6,
A similarity calculation step in which the similarity of pairs within each source attention matrix and each target attention matrix is calculated by the learning unit; and
A learning step in which, by the learning unit, a loss function is used to learn to reduce the similarity of positive pairs and to increase the similarity of negative pairs, the method further comprising: A generative adversarial network (GAN)-based image generation method capable of dual learning. According to clause 5,
A first target image generating step of generating a first target image including a second domain from the first source image by a first generator;
A second target image generating step of generating a second target image including a first domain from the second source image by a second generator;
A first determination step in which authenticity between the first source image and the second target image is determined by a first discriminator; and
A generative adversarial network (GAN)-based image capable of dual learning, further comprising a second discriminator in which authenticity between the second source image and the first target image is determined. How to create it.