CN112580782A - Dual-attention Generative Adversarial Network and Image Generation Method Based on Channel Augmentation - Google Patents

Abstract

The invention relates to a channel enhancement-based double-attention generation countermeasure network and an image generation method, wherein the network comprises a generator and a discriminator; the generator comprises a first rolling block and a double-attention machine module; the discriminator comprises a second rolling block and a double-attention machine module; a compression activation operation layer used for acquiring channel attention through compression activation operation is arranged in each of the convolution block I and the convolution block II; the double-attention mechanism module comprises a position attention unit and a channel attention unit which are parallel; the position attention unit establishes relevance among positions based on a self-attention mechanism to obtain position attention characteristics, and the channel attention unit establishes dependency among characteristic channels based on a channel attention mechanism to obtain channel attention characteristics; the dual attention mechanism module fuses the location attention feature and the channel attention feature. The invention can improve the generation performance of the generated countermeasure network, the generated data distribution is closer to the original data distribution, and the generated image quality is better.

Description

Dual-attention Generative Adversarial Network and Image Generation Method Based on Channel Augmentation

technical field

The invention relates to the technical field of image generation, in particular to a dual attention generation confrontation network based on channel enhancement and an image generation method.

Background technique

Generative Adversarial Network (GAN) techniques are mainly applied in the direction of image generation. The generative adversarial network model consists of a generator and a discriminator, wherein the generator can generate images according to the input noise vector or class label, the discriminator is used to discriminate the true and false images, and the generator and discriminator are trained against each other so that the generator learns the distribution of real data , the final generator can be a fake image that is close to the real image. Generative adversarial network technology is widely used in image enhancement, infrared image generation, medical image generation, image super-resolution reconstruction and other directions.

At present, the main methods to improve the image quality of the generative adversarial network model are to modify the network structure, change the loss function, and establish the feature space correlation. Self-attention generative adversarial network model uses a self-attention mechanism that can obtain large-scale spatial correlations, making the structure of generated images more reasonable. Based on the self-attention generative adversarial network, the BigGAN model adjusts and deepens the structure of the generative adversarial network, improves the network learning ability, and further improves the generation quality.

Although the current GAN generation ability is already very strong, it is still difficult to generate images of complex scenes with a wide range of correlations using only the noise vector z and the conditional label y in the case of little prior knowledge. . Existing GAN models have difficulty generating images with complex structural distributions. Although the self-attention mechanism enables the generative adversarial network to generate images with a wide range of correlations, there are still some defects in the target object structure in the generated images. At the same time, it is difficult to generate images containing multi-target object scenes, and the quality of the generated images also needs to be improved.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a generative adversarial network model capable of generating high-quality images for complex scenes in view of at least some of the above deficiencies, while making the structure distribution of the targets in the generated images more reasonable and the images more natural.

In order to achieve the above object, the present invention provides a dual attention generative confrontation network based on channel enhancement, including:

A generator and a discriminator; the generator includes a convolution block one and a double attention mechanism module; the discriminator includes a convolution block two and a double attention mechanism module;

Among them, convolution block 1 and convolution block 2 are both provided with a compressed activation operation layer used to obtain channel attention through compressed activation operation;

The dual attention mechanism module includes a parallel position attention unit and a channel attention unit; the position attention unit establishes the correlation between positions based on the self-attention mechanism, and obtains the position attention feature, and the channel attention unit is based on the self-attention mechanism. The channel attention mechanism establishes the inter-feature inter-channel dependency, and obtains the channel attention feature; the dual attention mechanism module fuses the position attention feature and the channel attention feature.

Preferably, the compression activation operation layer is configured to perform the following operations:

The feature M is averaged at each layer, compressed into a value, and a one-dimensional vector s is obtained. The expression is:

Among them, the number of layers of the feature M is C, the size of each layer is H×W, s _n represents the nth element of the one-dimensional vector s, m _n represents the nth layer of the feature M, n=1,...,C , m _n ^i,j represents the element whose coordinates are (i, j) in m _n ;

The one-dimensional vector s is activated by two-layer nonlinear fully connected network learning, and the weight feature vector w with the weight ratio of each layer is obtained, and the expression is:

w=σ ₂ (W ₂ σ ₁ (W ₁ s))

Among them, W ₁ and W ₂ represent the first and second layer fully connected operations, respectively, and σ ₁ and σ ₂ are the activation functions ReLU and Sigmoid, respectively;

表达式为：Multiply the weight feature vector w into the corresponding layer of feature M to get the calibrated feature

The expression is:

表示特征

的第n层。Among them, w _n represents the nth element of the weight feature vector w,

Representation features

the nth layer.

Preferably, the compression activation operation layer includes an average pooling layer, a 1×1 convolutional layer, a ReLU activation layer, a 1×1 convolutional layer, and a Sigmoid activation layer connected in sequence, and the output and the input are multiplied by the channel to obtain the calibration later features.

Preferably, the first convolutional block includes two linear layers, two batch normalization layers, two ReLU activation layers, two upsampling layers, two 3×3 convolutional layers, and 1×1 convolutional layers and the compressed activation operation layer;

The second convolutional block includes two ReLU activation layers, two 3×3 convolutional layers, 1×1 convolutional layers, two average pooling layers and the compressed activation operation layer.

Preferably, the channel attention unit is configured to perform the following operations:

Recombining feature A to obtain feature A'; wherein, the number of layers of feature A is C, the size of each layer is H×W, the size of feature A' is C×N, and N=H×W;

Multiplying the feature A' and the transpose of the feature A' to get softmax, the feature map Q is obtained. The size of the feature map Q is C×C, and the element q _ji in the feature map Q is expressed as:

为特征A′的转置的第j个特征向量；Among them, {i,j=1,2,...,C}, a' _i is the ith feature vector of feature A',

is the j-th eigenvector of the transpose of feature A';

The feature map Q is multiplied by the feature A' and de-recombined to obtain the channel attention feature T, which is expressed as:

Among them, T _j represents the j-th feature vector of the channel attention feature T, j=1, 2, ..., C, β represents the learning parameter, initialized to 0, and A _j represents the j-th feature vector of the feature A.

Preferably, the location attention unit is configured to perform the following operations:

特征B维度为

重组后维度为

Channel compression is performed on feature A with a 1×1 convolution f(x) function to obtain feature B, and then recombination is performed to obtain feature B'; among them, the number of layers of feature A is C, the size of each layer is H×W, and the compressed The number of layers is

The dimension of feature B is

The reorganized dimension is

特征O维度为

重组后维度为

Channel compression is performed on feature A with a 1×1 convolution g(x) function to obtain feature O, and then recombination is performed to obtain feature O'; the number of layers after compression is

The feature O dimension is

The reorganized dimension is

Multiply the feature B' and the transpose of the feature O' to get softmax to obtain the feature map P, the size of the feature map P is N×N, and the expression of the element p _ji in the feature map P is:

表示特征O’的转置的第j个特征向量，i,j＝1,2,…,N；Among them, b' _i represents the ith feature vector of feature B',

The j-th feature vector representing the transpose of feature O', i,j=1,2,...,N;

Extract feature A with 1×1 convolution h(x) function to obtain feature V, and then recombine to obtain feature V’; the number of layers after extraction is still C;

When the feature V' is multiplied by the feature map P and de-recombined to obtain the position attention feature S, the expression is:

Among them, S _j represents the j-th feature vector of the position attention feature S, j=1,2,...,N, v' _i represents the i-th feature vector representing the feature V', α represents the learning parameter, initialized to 0 , A _j represents the j-th feature vector of feature A.

Preferably, the dual attention mechanism module fuses the position attention feature S and the channel attention feature T through a 3×3 convolution J(x) function and a 3×3 convolution K(x) function, and the expression is:

U=J(S)+K(T)

where U represents the resulting fusion feature.

Preferably, the generator includes sequentially connected linear layers, a first convolution block one, a second convolution block one, a dual attention mechanism module, a third convolution block one, a first activation module, a Tanh Floor;

The input of the generator is the noise vector z and the class condition Condition, the noise vector z obeys the normal distribution, the class condition Condition is embedded in each batch normalization layer of each convolution block 1, and the output of the generator is for falsifying images;

The discriminator includes a first convolution block 2, a dual attention mechanism module, a second convolution block 2, a third convolution block 2, a fourth convolution block 2, and a second activation module connected in sequence. , linear transformation and label embedding layer;

The input of the discriminator is the RGB image and the label y, and the output is the discrimination result of the RGB image.

Preferably, the loss function expression of the discriminator is:

The loss function expression of the generator is:

表示(x，y)服从p_data的概率期望计算，

表示z服从p_z且y服从p_data的概率期望计算。Among them, x represents the image, y is the corresponding class label, p _data is the real data probability distribution, p _z is the noise probability distribution, z represents the one-dimensional noise vector, G(z) represents the mapping process of the generator, D(x, y) represents the mapping process of the discriminator,

Indicates that (x, y) obeys the probability expectation calculation of p _data ,

Denotes the probability expectation calculation that z obeys p _z and y obeys p _data .

The present invention also provides an image generation method, comprising the following steps:

S1. Construct the dual attention generative adversarial network as described in any of the above;

S2, obtaining a training set and inputting the dual attention generative adversarial network for training;

S3. Generate an image using the dual attention generative adversarial network after the training is completed.

The above technical solutions of the present invention have the following advantages: the present invention provides a dual-attention generative adversarial network and an image generation method based on channel enhancement, and the dual-attention generative adversarial network provided by the present invention is based on the existing BigGAN model. Improvements and enhancements, adding a compression activation operation layer used to obtain channel attention through compression activation operations in the convolution block of the generative adversarial network, which can recalibrate the features of the generative adversarial network, that is, enhance some of the features with strong effects. Feature layer, weaken some useless feature layers, make the middle layer features of the network more expressive, improve the performance of the convolution block, and improve the feature learning ability of the generative adversarial network model; and, the dual attention generative adversarial network adopts dual attention. The force mechanism module, the dual attention mechanism module not only contains the location attention unit (the same function as the self-attention mechanism module), but also contains the channel attention unit. After the dual attention mechanism is introduced, the generative adversarial network can simultaneously capture a large range of The correlation information on positions and channels can more comprehensively learn the correlation information between image feature structures, further enhance the correlation of image features, and improve the quality of generated images.

Description of drawings

Figures 1(a) and 1(b) are schematic diagrams of the structure of a dual attention generative adversarial network based on channel enhancement in an embodiment of the present invention; Figure 1(a) is a schematic diagram of the generator structure, and Figure 1(b) is a schematic diagram of the structure of the generator. Schematic diagram of the structure of the discriminator;

2 is a schematic diagram of a compression activation operation flow diagram in an embodiment of the present invention;

Fig. 3 (a) is a schematic diagram of a structure of a convolution block 1 in an embodiment of the present invention, and Fig. 3 (b) is a schematic diagram of a structure of a convolution block 2 in an embodiment of the present invention;

4 is a schematic flowchart of a dual attention mechanism in an embodiment of the present invention;

Fig. 5(a) shows a result graph of an image generated by the BigGAN model; Fig. 5(b) shows a graph of an image generated by a dual attention generative adversarial network based on channel enhancement in an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

As shown in Fig. 1(a) to Fig. 4, an embodiment of the present invention provides a dual-attention generative adversarial network based on channel enhancement. The dual-attention generative adversarial network is improved on the basis of the existing BigGAN model and Improvement, including generator and discriminator, the generator includes convolution block 1 and double attention mechanism module; the discriminator includes convolution block 2 and double attention mechanism module, convolution block 1 and convolution block 2 are both generative confrontation The convolutional block of the network.

Among them, convolution block 1 and convolution block 2 are both provided with a compressed activation operation layer for obtaining channel attention through compressed activation operation. The invention introduces a compression activation mechanism to improve the residual structure convolution block ResBlock of the generative adversarial network. The compression activation operation layer can strengthen some feature layers with strong functions in the feature layer, weaken some useless feature layers, and make the middle of the network. Layer features are more expressive and feature extraction.

The dual attention mechanism module includes parallel position attention unit and channel attention unit. The position attention unit establishes the correlation between positions based on the self-attention mechanism, and obtains the position attention feature. The channel attention unit establishes the feature inter-channel dependency based on the channel attention mechanism, and obtains the channel attention feature; the dual attention mechanism module fuses the position The attention feature S and the channel attention feature T are obtained to obtain the feature U with rich associated information. The present invention introduces a dual attention mechanism, so that the generative adversarial network can obtain the correlation information between a wide range of positions and between channels, so that the structure distribution of the target object of the generated image is more natural.

Preferably, as shown in Figure 2, in the dual-attention generative adversarial network, the compression activation operation layer is used to perform the compression activation (Squeeze-and-Excitation) operation that can obtain channel attention, and the compression activation operation includes the following steps:

Take the average of each layer of the feature M of the input compression activation operation layer, compress it into a value, and obtain a one-dimensional vector s, the expression is:

Among them, the number of layers of the feature M is C, that is, the number of channels is C, the size of each layer is H×W, the length of the obtained one-dimensional vector s is C, and s _n represents the nth element of the one-dimensional vector s, m _n Represents the nth layer of feature M, n=1,...,C, m _n ^i,j represents the element whose coordinates are (i, j) in m _n ;

The one-dimensional vector s is activated by two-layer nonlinear fully connected network learning, and the weight feature vector w with the weight ratio of each layer is obtained, and the expression is:

w=σ ₂ (W ₂ σ ₁ (W ₁ s))

Among them, W ₁ and W ₂ represent the first and second layer fully connected operations, respectively, σ ₁ , σ ₂ are the activation functions ReLU and Sigmoid, respectively, Sigmoid limits the value of w in the range of (0, 1), and the activation function is the neural network The network learns nonlinear relationships;

表达式为：Multiply the weight feature vector w into the corresponding layer of feature M to get the calibrated feature

The expression is:

表示特征

的第n层。Among them, w _n represents the nth element of the weight feature vector w,

Representation features

the nth layer.

Preferably, 1×1 convolution can be used for W ₁ and W ₂ to obtain a compressed activation operation layer (SELayer) including an average pooling layer (Average Pooling), a 1×1 convolution layer (1×1Conv), and ReLU connected in sequence. Activation layer (ReLU), 1×1 convolution layer, Sigmoid activation layer (Sigmoid), the output of the compressed activation operation layer and the input are multiplied by the channel, and the calibrated features are obtained.

In the present invention, a compression activation operation layer SELayer is embedded in the structure of the convolution block ResBlock of the BigGAN model to obtain the SEResBlock.

Figure 3(a) is a schematic diagram of the structure of the SEResBlock in the generator, that is, the SEResBlock(G) in Figure 1(a), that is, the first convolution block. Further, as shown in Figure 3(a), in a preferred embodiment, the convolution block 1 includes two linear layers (Linear), two batch normalization layers (BatchNorm), two ReLU activation layers, two Upsampling layer (Upsample), two 3×3 convolutional layers (3×3Conv), 1×1 convolutional layer (1×1Conv) and compressed activation layer. Among them, the category Condition input by the generator is input into two batch normalization layers respectively through two linear layers; the first batch normalization layer, the first ReLU activation layer, the second upsampling layer, the first 3 The ×3 convolutional layer, the second batch normalization layer, the second ReLU activation layer, and the second 3×3 convolutional layer are connected in sequence, and the input is the output of the previous module; the second 3×3 volume The product layer is then connected to the compression activation layer, and the output of the second 3×3 convolution layer is multiplied by the output of the compression activation layer to obtain the calibrated features; the output of the previous module is sequentially upsampled by the first layer, 1 × 1 convolution layer, and element-wise summation with the calibrated features, as the output of this module (ie, the convolution block 1).

Figure 3(b) is a schematic diagram of the structure of the SEResBlock in the discriminator, that is, the SEResBlock (D) in Figure 1(b), that is, the second convolution block. Compared with the first convolution block, the second convolution block lacks batch normalization. Floor. As shown in Figure 3(b), convolution block 2 includes two ReLU activation layers (ReLU), two 3×3 convolutional layers (3×3Conv), 1×1 convolutional layers (1×1Conv), two An average pooling layer (Average Pooling) and a compressed activation layer. Among them, the first ReLU activation layer, the first 3×3 convolutional layer, the second ReLU activation layer, the second 3×3 convolutional layer, and the second average pooling layer are connected in turn, and the input is the upper The output of a module (that is, the input of the second convolution block); the second average pooling layer is connected to the compression activation layer, and the output of the second average pooling layer is multiplied by the output of the compression activation layer. , the calibrated features are obtained; the output of the previous module goes through the 1×1 convolution layer and the first average pooling layer in turn, and then sums the calibrated features element by element as the module (that is, the convolution block). b) output.

In the present invention, in the first and second convolution blocks, the compression activation operation layer is used after the second 3×3 convolution layer, and the features learned by the entire convolution block are recalibrated without adding too much network. complexity, but also brings performance improvements.

Preferably, as shown in Figure 4, in the dual attention mechanism module of the dual attention generative adversarial network, the channel attention unit (Part I in Figure 4) is used to perform the following operations:

Reshape the feature A of the input dual attention mechanism module to obtain feature A'; the number of layers of feature A is C, the size of each layer is H×W, and the size of feature A' is C×N, N =H×W;

Multiplying the feature A' and the transpose of the feature A' to get softmax, the feature map Q is obtained. The size of the feature map Q is C×C, and the element q _ji in the feature map Q is expressed as:

为特征A′的转置的第j个特征向量，上标“T”表示转置；Among them, {i,j=1,2,...,C}, a' _i is the ith feature vector of feature A',

is the j-th eigenvector of the transpose of feature A', and the superscript "T" means transposition;

Multiply the feature map Q by the feature A' and perform inverse reorganization (inverse Reshape) to obtain the channel attention feature T, which is expressed as:

Among them, T _j represents the j-th feature vector of the channel attention feature T, j=1,2,...,C, β represents the learning parameter, initialized to 0, A _j represents the j-th feature vector of the feature A, j= 1,2,…,C.

Further, as shown in Figure 4, in the dual attention mechanism module, the location attention unit (Part II in Figure 4) is used to perform the following operations:

特征B维度为

重组(Reshape)后维度为

The feature A is channel compressed with the 1×1 convolution f(x) function to obtain the feature B, and then reshaped (Reshape) to obtain the feature B'; the number of layers of the feature A is C, and the size of each layer is H×W. After compression The number of layers, that is, the number of channels is

The dimension of feature B is

The dimension after Reshape is

特征O为

重组(Reshape)后维度为

Channel compression is performed on feature A with a 1×1 convolution g(x) function to obtain feature O, and then reshape is performed to obtain feature O'; the number of channels after compression is

The feature O is

The dimension after Reshape is

Multiply the feature B' and the transpose of the feature O' to get softmax to obtain the feature map P, the size of the feature map P is N×N, and the expression of the element p _ji in the feature map P is:

表示特征O’的转置的第j个特征向量，i,j＝1,2,…,N；Among them, b' _i represents the ith feature vector of feature B',

The j-th feature vector representing the transpose of feature O', i,j=1,2,...,N;

Extract feature A with 1×1 convolution h(x) function to obtain feature V, and then reshape (Reshape) to obtain feature V', the number of layers of feature V after extraction is still C; after the same group, V'= [v' ₁ ,v' ₂ ,...,v' _N ];

Multiply the feature V' and the feature map P and perform inverse reorganization (inverse Reshape) to obtain the position attention feature S, the expression is:

Among them, S _j represents the j-th feature vector of the position attention feature S, j=1,2,...,N, v' _i represents the i-th feature vector representing the feature V', i=1,2,..., N, α represents the learning parameter, initialized to 0, A _j represents the j-th feature vector of feature A.

Further, the dual attention mechanism module fuses the position attention feature S and the channel attention feature T through the 3×3 convolution J(x) function and the 3×3 convolution K(x) function, and the expression is:

U=J(S)+K(T)

Among them, U represents the fusion feature obtained by the dual attention mechanism module. There is rich correlation information in U.

As shown in Figure 1(a) and Figure 1(b), in a preferred embodiment, the generator includes sequentially connected linear layers, the first convolution block one, the second convolution block one, and a dual attention mechanism Module, the third convolution block 1, the first activation module, the Tanh layer; the input of the generator is the noise vector z and the category condition Condition, the noise vector z obeys the normal distribution, and the category condition Condition is embedded in each convolution block For each batch normalization layer, the output of the generator is a fake image.

The discriminator includes the first convolution block two, the dual attention mechanism module, the second convolution block two, the third convolution block two, the fourth convolution block two, the second activation module, the linear Transformation and label embedding layer; the input of the discriminator is the fake image and the label y, and the output is the discrimination result.

Further, as shown in Figure 1(a), the linear layer performs linear calculation on the noise vector z, and then reshapes it into a feature tensor of 4 × 4 × 16ch, where ch can be set to 64, and then passes through the base layer SEResBlock ( G) Learning, after batch normalization (BN), ReLU activation, 3×3 convolution processing of the first activation module, and then activation by the Tanh layer, a fake image with channel number ch=3 is finally generated, and the fake image is RGB The image, where the dual attention mechanism is placed in the middle and back layers of the network features, is in the same position as the self-attention mechanism in the BigGAN model.

As shown in Figure 1(b), the discriminator network structure is composed of SEResBlock (D) and a dual attention mechanism. The structure is opposite to that in the generator. Its function is to discriminate whether the input RGB image and label y are true or not. The final generated features are processed by ReLU activation and global sum pooling of the second activation module, and then judged by the fusion of linear transformation and embedded label y to judge the authenticity of the input RGB image.

Preferably, the loss functions of the generator and discriminator use Hinge Loss. Among them, the loss function expression of the discriminator is:

The loss function expression of the generator is:

表示(x，y)服从p_data的概率期望计算，

表示z服从p_z且y服从p_data的概率期望计算。Among them, x represents the image, y is the corresponding category label, p _data is the real data probability distribution, p _z is the noise probability distribution, z represents the one-dimensional noise vector, G(z) represents the mapping process of the generator, D(x , y) represents the mapping process of the discriminator,

Indicates that (x, y) obeys the probability expectation calculation of p _data ,

Denotes the probability expectation calculation that z obeys p _z and y obeys p _data .

In a preferred embodiment, the present invention sets the learning rate of the generator and the discriminator to 0.0002, adopts a learning rate decay strategy, and the decay rate is 0.9999. The network part adopts the spectrum normalization method to adjust the weights in the network training process, so that the training of the generative adversarial network is more stable. The number of training data for each batch of the network is Batch=64, and the total number of iterations is 10,000.

Further, the present invention adopts Mini-Batch stochastic gradient descent training, firstly training the loss function of the discriminator, and then training the generator. The pseudocode of the specific optimization process is shown in Table 1:

Table 1 Optimization process of generative adversarial network

表示生成器的梯度值，由判别损失方向求导得到，θ_d表示判别器模型的网络参数，

表示判别器的梯度值，生成器损失函数反向求导所得，θ_g表示生成器模型的网络参数。in,

Represents the gradient value of the generator, derived from the direction of the discriminant loss, θ _d represents the network parameters of the discriminator model,

Represents the gradient value of the discriminator, and the generator loss function is derived from the reverse derivation, and θ _g represents the network parameters of the generator model.

Inception Distance(FID)评估指标。FID指标在评估图像时，指标越小，代表生成的图像质量越好。FID计算过程中，首先将生成图像和真实图像经过InceptionV3网络提取特征向量，然后分别计算真实数据服从的正态分布和生成数据服从的正态分布，然后计算两种数据分布间的距离。In particular, in order to qualitatively evaluate the effect of the dual-attention generative adversarial network of the present invention, one can use

Inception Distance (FID) evaluation metric. When the FID indicator is used to evaluate images, the smaller the indicator, the better the quality of the generated image. In the FID calculation process, the generated image and the real image are first extracted through the InceptionV3 network, and then the normal distribution obeyed by the real data and the normal distribution obeyed by the generated data are calculated respectively, and then the distance between the two data distributions is calculated.

The invention uses the ImageNet public data set to compare and verify the performance of the existing BigGAN model and the dual attention generation confrontation network provided by the invention, extracts some categories of the ImageNet public data set, and adjusts it to a resolution of 128×128. The evaluation was carried out using the FID indicator, and the results of some generated images are shown in Figures 5(a) and 5(b). The evaluation results show that the FID index obtained by using the BigGAN model is 17.43, while the FID index obtained by the channel enhancement-based dual attention generative adversarial network (SEDA-GAN) provided by the present invention is 14.89, an increase of 14.57%.

Moreover, the structural changes brought by the present invention are more obvious. Figure 5(a) shows that the BigGAN model randomly generates some samples, and Figure 5(b) shows that the SEDA-GAN model randomly generates some samples. It can be seen that the BigGAN model generates some samples randomly. The position of the organ structure of the goldfish in the sample generated by the model is inaccurate. When there are multiple goldfish, it is confusing, the mouth of the coffee cup is not round, and the structure of the truck and the wooden house also has some defects, and the samples generated by the SEDA-GAN model are in multiple goldfish. In this case, the structure distribution is also more natural, the coffee cup is more rounded, the overall structure of the truck is more natural, and the building structure of the wooden house is also more straight, and the overall visual effect has been improved.

To sum up, the present invention provides a dual attention generative adversarial network based on channel enhancement. On the one hand, the present invention adds a channel correlation learning mechanism to the convolution block structure, recalibrates the features, and improves the model convolution block. The feature learning ability of generative adversarial network enhances the feature expression ability of generative adversarial network. Based on the ResBlock structure of the convolution block of the BigGAN model, a Squeeze-and-Excitation operation that can obtain channel attention is introduced, and a new convolution block SEResBlock is proposed. It has been verified that the generative adversarial network composed of channel-enhanced convolution blocks such as SEResBlock has better generation performance, and the learning speed of data distribution is also improved. On the other hand, the present invention considers that from the perspective of channels, a channel attention mechanism capable of establishing dependencies between feature channels is added to the attention mechanism part, and parallel with the self-attention mechanism, it is constructed into a dual attention mechanism module, which together from the Potential data structure associations that exist between modeled features between locations and channels. In practice, network features not only have correlations between locations, but also associated information on feature channels. Therefore, after the introduction of the dual attention mechanism, the generative adversarial network can simultaneously capture the correlation information on a large range of positions and channels, so that the generative adversarial network can learn more data structure distribution information and make the target structure of the generated image more reasonable. It is very helpful to improve the quality of image generation. Compared with the prior art, after using the channel enhancement and dual attention mechanism, the technical solution proposed by the present invention has a small increase in the total network parameters, and the generation performance of the generative adversarial network is improved. The data distribution generated by the invention is closer to the original data distribution, and the visual effect is better, the quality of the generated image is better, and the structure of the target object is more normal and natural.

The present invention also provides an image generation method, comprising the following steps:

S1. Construct the dual attention generative adversarial network as described in any of the above;

S2, obtaining a training set and inputting the dual attention generative adversarial network for training;

S3. Generate an image using the dual attention generative adversarial network after the training is completed.

In particular, in some preferred embodiments of the present invention, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any of the foregoing implementations is implemented The steps of the image generation method described in the method.

In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image generation method described in any of the above-mentioned embodiments are implemented. .

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, When the computer program is executed, it may include the flow of the above-mentioned image generation method embodiments, and the description will not be repeated here.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A channel-enhanced dual-attention-generating countermeasure network, characterized by: comprises a generator and a discriminator; the generator comprises a first rolling block and a double-attention machine module; the discriminator comprises a second rolling block and a double-attention machine module;

the first convolution block and the second convolution block are both provided with a compression activation operation layer used for acquiring channel attention through compression activation operation;

the double-attention mechanism module comprises a position attention unit and a channel attention unit which are parallel; the position attention unit establishes relevance among positions based on a self-attention mechanism to obtain position attention characteristics, and the channel attention unit establishes dependency among characteristic channels based on a channel attention mechanism to obtain channel attention characteristics; the dual attention mechanism module fuses a location attention feature and a channel attention feature.

2. The dual-attention generating countermeasure network of claim 1,

the compression activation operation layer is used for executing the following operations:

averaging each layer of the characteristics M, and compressing the average to form a value to obtain a one-dimensional vector s, wherein the expression is as follows:

wherein the number of layers of the feature M is C, and the size of each layer is H multiplied by W, s_nN-th element, m, representing a one-dimensional vector s_nAn nth layer representing a feature M, n being 1_n ^i,jRepresents m_nAn element with a middle coordinate of (i, j);

activating the one-dimensional vector s through two-layer nonlinear full-connection network learning to obtain a weight characteristic vector w with each layer of weight proportion, wherein the expression is as follows:

w＝σ₂(W₂σ₁(W₁s))

wherein, W₁、W₂Respectively representing the first and second layer fully connected operation, σ₁、σ₂Respectively, activation functions ReLU and Sigmoid;

multiplying the weight characteristic vector w into the corresponding layer of the characteristic M to obtain the calibrated characteristic

The expression is as follows:

wherein, w_nThe nth element of the weight feature vector w,

representation feature

The nth layer of (1).

3. The dual-attention generating countermeasure network of claim 2,

the compressed activation operation layer comprises an average pooling layer, a 1 × 1 convolution layer, a ReLU activation layer, a 1 × 1 convolution layer and a Sigmoid activation layer which are sequentially connected, and channel multiplication is carried out on output and input to obtain calibrated characteristics.

4. The dual-attention generating countermeasure network of claim 3,

the convolution block I comprises two linear layers, two batch normalization layers, two ReLU active layers, two up-sampling layers, two 3 x 3 convolution layers, a 1 x 1 convolution layer and the compression activation operation layer;

the convolution block two comprises two ReLU active layers, two 3 x 3 convolution layers, a 1 x 1 convolution layer, two average pooling layers and the compressed active operation layer.

5. The dual-attention generating countermeasure network of claim 1,

the channel attention unit is used for executing the following operations:

recombining the characteristic A to obtain a characteristic A'; the number of layers of the feature A is C, the size of each layer is H multiplied by W, the size of the feature A' is C multiplied by N, and N is H multiplied by W;

multiplying the feature A 'by the transpose of the feature A' to obtain softmax, and obtaining a feature map Q, wherein the size of the feature map Q is C multiplied by C, and an element Q in the feature map Q is_jiThe expression is as follows:

wherein { i, j ═ 1,2, …, C }, a'_iThe ith feature vector, a 'of feature A'^T _jA jth feature vector that is a transpose of feature a';

multiplying the feature graph Q and the feature A' and carrying out reverse recombination to obtain a channel attention feature T, wherein the expression is as follows:

wherein, T_jJ-th feature vector representing channel attention feature T, j being 1,2, …, C, β representing learning parameters, is initialized to 0, a_jThe jth feature vector representing feature a.

6. The dual-attention generating countermeasure network of claim 5,

the position attention unit is used for executing the following operations:

performing channel compression on the characteristic A by using a 1 multiplied by 1 convolution f (x) function to obtain a characteristic B, and performing recombination to obtain a characteristic B'; wherein the number of layers of the feature A is C, the size of each layer is H multiplied by W, and the number of layers after compression is

Characteristic B dimension of

Dimension after recombination of

Performing channel compression on the characteristic A by using a 1 × 1 convolution g (x) function to obtain a characteristic O, and performing recombination to obtain a characteristic O'; number of layers after compression

Characteristic dimension O of

Dimension after recombination of

Multiplying the feature B 'by the transpose of the feature O' to obtain softmax, and obtaining a feature map P, wherein the size of the feature map P is NxN, and an element P in the feature map P_jiThe expression is as follows:

wherein, b'_iThe ith feature vector representing feature B',

a transposed jth feature vector representing feature O', i, j ═ 1,2, …, N;

extracting the characteristic A by a 1 multiplied by 1 convolution h (x) function to obtain a characteristic V, and recombining to obtain a characteristic V'; the number of layers after extraction is still C;

multiplying the feature V' by the feature map P and performing inverse recombination to obtain the position attention feature S, wherein the expression is as follows:

wherein S is_jJ-th feature vector representing position attention feature S, j ═ 1,2, …, N, v'_iDenotes the ith feature vector representing the feature V', alpha denotes the learning parameter, initialized to 0, A_jThe jth feature vector representing feature a.

7. The dual-attention generating countermeasure network of claim 6,

the double attention mechanism module fuses the position attention feature S and the channel attention feature T through a 3 x 3 convolution J (x) function and a 3 x 3 convolution K (x) function, and the expression is as follows:

U＝J(S)+K(T)

where U represents the resulting fusion signature.

8. The dual-attention generating countermeasure network of claim 1,

the generator comprises a linear layer, a first rolling block I, a second rolling block I, a double-attention machine module, a third rolling block I, a first activation module and a Tanh layer which are sequentially connected;

the input of the generator is a noise vector z and a class Condition, the noise vector z obeys normal distribution, the class Condition is embedded into each batch normalization layer of each volume block I, and the output of the generator is a forged image;

the discriminator comprises a first rolling block II, a double-attention machine module, a second rolling block II, a third rolling block II, a fourth rolling block II, a second activation module and a linear transformation and label embedding layer which are sequentially connected;

the input of the discriminator is an RGB image and a label y, and the output is a discrimination result of the RGB image.

9. The dual-attention generating countermeasure network of claim 1,

the loss function expression of the discriminator is as follows:

the loss function expression of the generator is as follows:

where x denotes an image, y is the corresponding class label, p_dataFor true data probability distribution, p_zFor the noise probability distribution, z represents a one-dimensional noise vector, G (z) represents the mapping process of the generator, D (x, y) represents the mapping process of the discriminator,

representing (x, y) obeys p_dataThe probability of (a) is expected to be calculated,

representing z obeys p_zAnd y obeys p_dataIs expected to be calculated.

10. An image generation method, comprising the steps of:

s1, constructing a dual-attention generating countermeasure network as claimed in any one of claims 1 to 9;

s2, acquiring a training set and inputting the double-attention force generation countermeasure network for training;

and S3, generating an image by using the double-attention generation confrontation network after training is completed.

Images