CN111696027A - Multi-modal image style migration method based on adaptive attention mechanism - Google Patents

Abstract

The invention discloses a multimodal image style transfer method based on an adaptive attention mechanism, which belongs to the field of computer vision. The method first chooses to use the generative adversarial network as the basic framework. At the same time, it draws on the idea of EM attention mechanism algorithm and channel scale transformation, and improves the EM attention mechanism in the channel domain, so that the network pays more attention to style features. And use the noise to weight the bases in the attention module, so that it can change adaptively, and finally make the style change. At this time, the noise and the image are input into the network at the same time, and the generated adversarial network is used to confront the training algorithm. After training the network, changing the size of the noise enables multimodal style transfer. Through the above method, the present invention makes full use of the advantages of the EM attention mechanism and the generative adversarial network, and proposes an adaptive channel domain EM attention module, which improves the image quality and image diversity of the existing methods after style transfer.

Description

A Multimodal Image Style Transfer Method Based on Adaptive Attention Mechanism

technical field

The invention belongs to the field of computer vision, mainly relates to the problem of multimodal image style transfer, and is mainly applied to the film and television entertainment industry, human-computer interaction and machine vision understanding.

Background technique

Image style transfer refers to the technology of transforming the style of one style of pictures into other different styles while retaining the content of the pictures after analyzing pictures of different styles through computer technology. In the film and television entertainment industry, human-computer interaction and machine vision understanding, there is an increasing demand for image style transfer. For example, the avatar can be turned into a cartoon avatar in real time through the camera; in automatic driving, the style transfer can be used to assist the conversion of the picture to the segmentation picture, etc. The existing image style transfer methods are mainly divided into image-based optimization and model-based optimization methods.

The style transfer method based on image optimization is an early and relatively stable method, and its basic principle can be divided into three steps. The first step is to select a neural network that can extract image features. The second step is to use the neural network to extract features from the original image and the target image, and use the features to design a loss function. The third step is to use the loss function to The derivation operation is performed on the original image, and continuous optimization and iteration are performed to make the style of the original image approach the style of the target image. This type of method does not require a large amount of data, so it is simple and easy to operate; but its disadvantage is that the iteration time is too long, and the image cannot be converted in real time. References: L.A. Gatys, A.S. Ecker, M. Bethge. Image styletransfer using convolutional neural networks, IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414-2423.

The method based on model iteration mainly trains the model through a large number of pictures of different styles, so that the model can learn the mapping function from one picture style and other kinds of picture styles, and then input a style picture into the trained model, and then it can be obtained from Get pictures of different styles but the same content in the model output. The advantage of this method is that there is no need for iterative steps after the model is trained, and image style transfer can be performed in real time, and this type of method can input additional variables to input a category of style pictures, and output multiple different styles at the same time. Variety style picture. However, its disadvantage is that the types of styles that are not available during training cannot be transferred well during testing, and the sample diversity is still lacking during multimodal style transfer. References: A. Yazeed, S. Neil, W. Peter. Latent filterscaling for multimodal unsupervised image-to-image translation. 2019, pp. 1458-1466

In recent years, model-based optimization methods have become more and more mature, and the demand for multimodal style transfer has also increased. Current methods still lack diversity and image quality in multimodal style transfer. Multimodal style transfer means that, given an input image, different styles of images can be output at the same time, as shown in Figure 2, the first image is the input image, and the others are multi-style images output at the same time. Aiming at this field, and considering the above shortcomings, the present invention proposes a multimodal image style transfer method based on an adaptive attention mechanism, and has achieved excellent results.

SUMMARY OF THE INVENTION

The present invention is a multi-modal style transfer method of adaptive channel domain EM attention mechanism, which solves the problem of lack of style diversity in the prior art.

The method first chooses to use the generative adversarial network as the basic framework, normalizes the training image to the size of 256*256*3, and also samples the normal distribution to obtain noise. At the same time, the idea of EM attention mechanism algorithm and channel scale transformation is borrowed, and the EM attention mechanism is improved in the channel domain, so that the network can pay more attention to style features, and use noise to weight the basis in the attention module. , enabling it to adapt adaptively and ultimately to change the style. At this time, the noise and the image are input into the network at the same time, and the generated adversarial network is used to confront the training algorithm. After training the network, changing the size of the noise enables multimodal style transfer. Through the above method, the present invention makes full use of the advantages of the EM attention mechanism and the generative adversarial network, and proposes an adaptive channel domain EM attention module, which improves the image quality and image diversity of the existing methods after style transfer. The overall structure of the algorithm is shown in Figure 1.

For the convenience of describing the content of the present invention, some terms are first defined.

其中μ为正态分布的数学期望，σ²为正态分布的方差，则称其满足正态分布，常记作

Definition 1: Normal distribution. Also known as the normal distribution, also known as the Gaussian distribution, is a very important probability distribution in the fields of mathematics, physics and engineering, and has a significant impact in many aspects of statistics. If the random variable x, its probability density function satisfies

where μ is the mathematical expectation of the normal distribution, and σ ² is the variance of the normal distribution, then it is said to satisfy the normal distribution, which is often denoted as

Definition 2: Generative Adversarial Networks. The generative adversarial network contains two different neural networks, one is called the generator G and the other is called the discriminator D. These two neural networks fight against each other during the training process. The purpose of the discriminator is to distinguish the real data distribution P _data and the generator distribution P _G , and the purpose of the generator is not to let the discriminator distinguish the two distributions, and the final result will be P _data =P _G .

Definition 3: EM algorithm. That is, the expectation maximization algorithm. For the observed data X and the unobservable data Z, the two are collectively referred to as complete data D=(X, Z). The EM algorithm first initializes a model and its parameters, and uses the model to estimate Z, which is called the E step; then uses the estimated Z to update the model, which is called the M step.

Definition 4: Generalized kernel function. It is a function that describes the relationship between points and points, and it is also a function that describes different spatial mapping relationships. There are many options for it, such as the dot product between vectors.

其中f(·，·)表示广义核函数，x表示输入，C(x)表示x的总和，g表示任意变换，结构示意参见图3.Definition 5: Attention mechanism. The attention mechanism usually includes 3 modules, query, key and value. The query and key first perform the correlation operation, and finally perform the weighting operation with the value. The core operator is

where f( , ) represents the generalized kernel function, x represents the input, C(x) represents the sum of x, and g represents an arbitrary transformation. See Figure 3 for the structure diagram.

Definition 6: EM attention mechanism. That is, the combination of the EM algorithm and the attention mechanism, mainly by improving the attention mechanism and adding loop iteration steps.

Definition 7: Adaptive Channel Domain EM Attention Mechanism. It is the method proposed in the present invention, which is an improvement of the EM attention mechanism by changing the scope of attention to the channel field and adding new inputs. See step 3 for details.

Definition 8: softmax function. Or normalized exponential function, it can "compress" a K-dimensional vector x containing any real number into another K-dimensional real vector softmax(x), so that the range of each element is between (0, 1). and the sum of all elements is 1. Its formula can be expressed as:

Definition 9: Relu function. Also known as the modified linear unit, it is a commonly used activation function in artificial neural networks, usually referring to the nonlinear function represented by the ramp function and its variants, and the expression is f(x)=max(0, x).

定义。Definition 10: Tanh function. expression can be used

definition.

Therefore, the technical solution of the present invention is a multi-modal image style transfer method based on an adaptive attention mechanism, and the method includes:

Step 1: Preprocess the dataset;

Obtain the edges2shoes data set, the edges2shoes data set contains shoe outlines and real shoe pictures, a total of 49825 picture teams; then classify the data set, shoe outlines are one type, real shoes are another type, and the order is randomly shuffled; finally Normalize the image pixel values to the range [-1,1];

Step 2: Build a convolutional neural network and a fully connected neural network;

1) The construction of a convolutional neural network includes two sub-networks, one for the generator and the other for the discriminator; the input and output of the generator are pictures, while the input and output of the discriminator are pictures, and the output is a scalar; the first two layers of the generator network are 2 downsampling convolution blocks, followed by 9 residual network blocks, followed by 2 upsampling convolution blocks; the discriminator network sequentially uses 4 downsampling convolution blocks, and two standard convolution blocks; standard The convolution block, the upsampling convolution block, the downsampling convolution block, and the residual network block are shown in Figure 5.

表示维度，假设构建的卷积神经网络中生成器所有通道数目大小为L，全连接网络的输出包含两个部分，第一部为向量

另一部分为向量

其中K为步骤3中基的个数；而其总体包括两层大小均为128维的隐层，中间层使用Relu函数作为激活函数，输出层使用Tanh函数作为损失函数；2) Construct a fully connected network with an input size of 8-dimensional vector

Represents the dimension, assuming that the number of all channels of the generator in the constructed convolutional neural network is L, the output of the fully connected network consists of two parts, the first part is a vector

The other part is a vector

Among them, K is the number of bases in step 3; and its overall includes two hidden layers with a size of 128 dimensions, the middle layer uses the Relu function as the activation function, and the output layer uses the Tanh function as the loss function;

表示第i个通道的N维向量；给定

以及通过正态分布随机采样，初始化一组由K个基向量

组成的矩阵

其中K＜N；则步骤3分为以下三个小步骤进行；第一步是估计隐藏变量

第二步是利用第一步估计的结果来更新基向量矩阵M；第一步和第二步循环迭代直至μ和Z收敛；第三步是利用M和Z来重构X，并利用步骤2中得到的S来对M进行乘法运算；Step 3: Build an adaptive channel domain EM attention module, see Figure 4; corresponding to the process in the Gaussian mixture model, after a picture is sent to the generator in the convolutional neural network, it is output through the convolution block in the generator The obtained feature map is X, the size is C×H×W, where C is the number of channels, H and W are the height and width of the feature map respectively; X is the input image obtained after passing the activation function of the convolution block in the generator ; reshape X to C×N, where N=H×W;

N-dimensional vector representing the ith channel; given

and random sampling through the normal distribution, initialize a set of K basis vectors

composed of matrices

Where K<N; then step 3 is divided into the following three small steps; the first step is to estimate hidden variables

The second step is to use the results estimated in the first step to update the basis vector matrix M; the first and second steps iterate in a loop until μ and Z converge; the third step is to use M and Z to reconstruct X, and use step 2 Multiply M with the S obtained in ;

Step 4: Total Neural Network;

Embed the adaptive channel domain EM attention module in step 3 into the generator in step 2, in a total of 3 different embeddings; the first is the first residual after the second layer downsampling the convolution block Before the network block, the second is replaced at the position of the 5th residual network block, and the third is embedded before the first upsampling convolution block after the last residual network block; features in the output of the fully connected neural network The graph control code d is multiplied into the outputs of all convolutional layers in the generator, while the base control code S is multiplied into the base M in the adaptive channel domain EM attention module obtained in step 3; the output of the generator is used as the discriminator’s Input, the output of the discriminator is the output of the total neural network;

The overall network framework is shown in Figure 1;

Step 5: Design the loss function;

For the pictures obtained in step 1, record the shoe outline category picture as _IA , and the real shoe picture as _IB ; and perform random sampling from the normal distribution to obtain a vector v, the generator in step 2 and the fully connected network are recorded as G, the discriminator is recorded as D; the generator input in G is I _A , the input of the fully connected network is v, the two work together and the output is recorded as G(I _A , v); the input of the discriminator is I _B and G(I _A , v), and their outputs are denoted as D(I _B ) and D(G(I _A , v)), respectively. Then the network loss can be described as:

为判别器的损失函数，

为生成器的损失函数；

分别表示对(I_A，v)和I_B求期望；

is the loss function of the discriminator,

is the loss function of the generator;

respectively represent the expectation of (I _A , v) and I _B ;

Step 6: Train the total neural network, use the loss function constructed in step 5 for network training, fix the parameters of D when updating G, and update D to fix the parameters of G, and update it alternately once per iteration;

Step 7: In the testing phase, the model is trained in step 6, and only the G part of the network is taken; given an input image I _A , and different normal distribution samples v, multiple output images of different styles are obtained.

Further, the specific method of the step 3 is:

这一步是计算每一个基对每一个通道的负责程度，即每个通道属于每个基的可能性；z_ck表示的是μ中第k个基对第c个通道x_c的权值，其中1≤k≤K且1≤c≤C；构建条件于μ_c的x_c的后验概率分布如下：Step 3.1: Estimate hidden variables

This step is to calculate the degree of responsibility of each basis for each channel, that is, the possibility that each channel belongs to each basis; z _ck represents the weight of the k-th basis to the c-th channel x _c in μ, where 1≤k≤K and 1≤c≤C; the posterior probability distribution of x _c conditional on μ _c is constructed as follows:

表示的是广义核函数；则z_ck可以用如下公式进行计算：in

represents the generalized kernel function; then z _ck can be calculated by the following formula:

选择

的形式，则对于第t次迭代，隐藏变量Z采用下面的公式进行计算：Kernel function

choose

, then for the t-th iteration, the hidden variable Z is calculated using the following formula:

Z ^(t) = softmax(X(M ^(t-1) ) ^T )

Step 3.2: Update the basis vector μ: This step is obtained by maximizing the likelihood function of the complete data, which corresponds to the Gaussian mixture model. This step is to use the weights calculated in the first step to perform a weighted sum of the samples. The probability of belonging to a certain basis is used to update the value of the basis; for the t-th iteration, by the weighted sum of X, the update of the basis vector can be expressed as:

Step 3.3: After step 3.1 and step 3.2 are alternately performed T times, go to step 3.3, use M and Z to reconstruct X, and use S obtained in step 2 to multiply μ; for the S obtained in step 2 , whose length is K, which is equal to the number of bases μ; then we can finally use the following formula to reconstruct X:

The innovation of the present invention is:

1) Transform the air domain of the attention mechanism into the channel domain. The attention of the air domain is to regard the pixel as a variable and find the weight of the basis to the pixel, and we convert the air domain to the channel domain, that is, to find the weight of the basis to the channel. value, as shown in Figure 6.

2) Adaptive weighting of the attention mechanism, the weighting of the feature map can change the style of the output picture, but we use the weighting of the basis in the attention to replace the weighting of the feature map, as shown in Figure 7.

3) We introduce this approach into multimodal style transfer and achieve excellent results in experiments.

The improvement in 1) can make the attention mechanism pay more attention to the style, and the improvement in 2) can allow us to change the output style more accurately, and the combination of the two finally improves our experimental results.

Description of drawings

Fig. 1 is the main network structure diagram of the method of the present invention

FIG. 2 is a schematic diagram of a multimodal style transfer result of the present invention.

FIG. 3 is a schematic diagram of the attention mechanism of the present invention.

FIG. 4 is a schematic diagram of the adaptive channel domain EM attention machine mechanism of the present invention.

5 is a schematic diagram of a standard convolution block, an up-sampling convolution block, a down-sampling convolution block and a residual network block of the present invention.

FIG. 6 is a schematic diagram of converting the attention from the spatial domain to the attention from the channel domain according to the present invention.

FIG. 7 is a schematic diagram of an adaptive weighting manner of the present invention.

Detailed ways

Step 1: Preprocess the dataset;

Get the edges2shoes (http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/edges2shoes.tar.gz) dataset, the edges2shoes dataset contains shoe outlines and real shoe pictures, a total of 49825 pairs of pictures; then pair the dataset For classification, the shoe outline is one class, and the real shoe is another class, and the order is randomly shuffled; finally, the pixel values of the picture are normalized to the range [-1, 1].

Step 2: Build a convolutional neural network and a fully connected neural network;

1) The convolutional neural network constructed in this step includes two sub-networks, one is the generator and the other is the discriminator; the input and output of the generator are both pictures, while the input and output of the discriminator are pictures and the output is a scalar; The two layers are 2 downsampling convolution blocks, followed by 9 residual network blocks, followed by 2 upsampling convolution blocks; the discriminator network sequentially uses 4 downsampling convolution blocks, and two standard convolution blocks block; standard convolution block, upsampling convolution block, downsampling convolution block and residual network block are shown in Figure 5.

表示维度，假设构建的卷积神经网络中生成器所有卷积核数目大小为L，全连接网络的输出包含两个部分，第一部为向量

另一部分为向量

其中K为步骤3中基的个数；而其总体包括两层大小均为128维的隐层，中间层使用Relu函数作为激活函数，输出层使用Tanh函数作为损失函数；2) The fully connected network constructed in this step has an input size of 8-dimensional vector

Represents the dimension, assuming that the number of all convolution kernels of the generator in the constructed convolutional neural network is L, the output of the fully connected network consists of two parts, the first part is a vector

The other part is a vector

Among them, K is the number of bases in step 3; and its overall includes two hidden layers with a size of 128 dimensions, the middle layer uses the Relu function as the activation function, and the output layer uses the Tanh function as the loss function;

3) The d in the output of the fully connected neural network is multiplied by all the convolutional outputs in the generator, and S is multiplied by the base M in the adaptive channel domain EM attention module (built in step 3).

表示第i个通道的N维向量；给定

以及通过正态分布随机采样，初始化一组由K个基向量

组成的矩阵

其中K＜N；则步骤3分为以下三个小步骤进行；第一步是估计隐藏变量

第二步是利用第一步估计的结果来更新基向量矩阵M；第一步和第二步循环迭代直至μ和Z收敛；第三步是利用M和Z来重构X，并利用步骤2中得到的S来对M进行乘法运算；Step 3: Build an adaptive channel domain EM attention module, see Figure 4; corresponding to the process in the Gaussian mixture model, after a picture is sent to the generator in the convolutional neural network, it is output through the convolution block in the generator The obtained feature map is X, the size is C×H×W, where C is the number of channels, H and W are the height and width of the feature map respectively; X is the input image obtained after passing the activation function of the convolution block in the generator ; reshape X to C×N, where N=H×W;

N-dimensional vector representing the ith channel; given

and random sampling through the normal distribution, initialize a set of K basis vectors

composed of matrices

Where K<N; then step 3 is divided into the following three small steps; the first step is to estimate hidden variables

The second step is to use the result estimated in the first step to update the basis vector matrix M; the first and second steps iterate in a loop until μ and Z converge; the third step is to use M and Z to reconstruct X, and use step 2 The S obtained in the multiplication operation is performed on M;

Step 4: Overall neural network structure;

Embed the adaptive channel domain EM attention module in step 3 into the generator in step 2, in a total of 3 different embeddings; the first is the first residual after the second layer downsampling the convolution block Before the network block, the second is replaced at the position of the fifth residual network block, and the third is embedded before the first upsampling convolution block after the last residual network block; the overall network framework is shown in Figure 1 ;

Step 5: Design the loss function;

For the pictures obtained in step 1, record the shoe outline category picture as _IA , and the real shoe picture as _IB ; and perform random sampling from the normal distribution to obtain a vector v, the generator in step 2 and the fully connected network are recorded as G, the discriminator is recorded as D; the generator input in G is I _A , the input of the fully connected network is v, the two work together and the output is recorded as G(I _A , v); the input of the discriminator is I _B and G(I _A , v), and their outputs are denoted as D(I _B ) and D(G(I _A , v)), respectively. Then the network loss can be described as:

为判别器的损失函数，

为生成器的损失函数；

分别表示对(I_A，v)和I_B求期望；

is the loss function of the discriminator,

is the loss function of the generator;

respectively represent the expectation of (I _A , v) and I _B ;

Step 6: Train the network, use the loss function constructed in step 5 for network training, fix the parameters of D when updating G, and update D is the opposite, alternately update each iteration, and use 1,000,000 iterations in actual training;

Step 7: In the testing phase, the model is trained in step 6, and only the G part of the network is taken. Given an input image I _A and different normal distribution samples v, multiple output images of different styles can be obtained, and the test of image quality and image diversity can be completed. The experimental results show that on the edges2shoes dataset, the image quality score is 0.15 points higher than the previous 10.32, reaching 10.47 points; the image diversity score is 0.005 points higher than the previous 0.109, reaching 0.114 points.

Further, the specific method of the step 3 is:

这一步是计算每一个基对每一个通道的负责程度，即每个通道属于每个基的可能性；z_ck表示的是μ中第k个基对第c个通道x_c的权值，其中1≤k≤K且1≤c≤C；构建条件于μ_c的x_c的后验概率分布如下：Step 3.1: Estimate hidden variables

This step is to calculate the degree of responsibility of each basis for each channel, that is, the possibility that each channel belongs to each basis; z _ck represents the weight of the k-th basis to the c-th channel x _c in μ, where 1≤k≤K and 1≤c≤C; the posterior probability distribution of x _c conditional on μ _c is constructed as follows:

表示的是广义核函数；则z_ck可以用如下公式进行计算：in

represents the generalized kernel function; then z _ck can be calculated by the following formula:

选择

的形式，则对于第t次迭代，隐藏变量Z采用下面的公式进行计算：Kernel function

choose

, then for the t-th iteration, the hidden variable Z is calculated using the following formula:

Z ^(t) = softmax(X(M ^(t-1) ) ^T )

Step 3.2: Update the basis vector μ: This step is obtained by maximizing the likelihood function of the complete data, which corresponds to the Gaussian mixture model. This step is to use the weights calculated in the first step to perform a weighted sum of the samples. The probability of belonging to a certain basis is used to update the value of the basis; for the t-th iteration, by the weighted sum of X, the update of the basis vector can be expressed as:

Step 3.3: After performing step 3.1 and step 3.2 alternately T times, proceed to step 3.3, use M and Z to reconstruct X, and use S obtained in step 2 to multiply μ: For the S obtained in step 2 , whose length is K, which is equal to the number of bases μ; then we can finally use the following formula to reconstruct X:

Image size: 256*256*3

Learning rate: 0.0002, and decreases linearly with the number of iterations

Training batch size: 1

Iterations: 1000000

Adaptive channel domain EM attention module iteration number T: 3.

Claims (2)

1. A multimodal image style transfer method based on adaptive attention mechanism, the method comprising: Step 1: Preprocess the dataset; Obtain the edges2shoes data set, the edges2shoes data set contains shoe outlines and real shoe pictures, a total of 49825 picture teams; then classify the data set, shoe outlines are one type, real shoes are another type, and the order is randomly shuffled; finally Normalize the image pixel values to the range [-1, 1]; Step 2: Build a convolutional neural network and a fully connected neural network; 1) The construction of a convolutional neural network includes two sub-networks, one for the generator and the other for the discriminator; the input and output of the generator are pictures, while the input and output of the discriminator are pictures, and the output is a scalar; the first two layers of the generator network are 2 downsampling convolution blocks, followed by 9 residual network blocks, and finally 2 upsampling convolution blocks; the discriminator network sequentially uses 4 downsampling convolution blocks, and two standard convolution blocks; 2) Construct a fully connected network with an input size of 8-dimensional vector

Represents the dimension, assuming that the number of all channels of the generator in the constructed convolutional neural network is L, the output of the fully connected network consists of two parts, the first part is a vector

The other part is a vector

Among them, K is the number of bases in step 3; and its overall includes two hidden layers with a size of 128 dimensions, the middle layer uses the Relu function as the activation function, and the output layer uses the Tanh function as the loss function; Step 3: Build an adaptive channel domain EM attention module; corresponding to the process in the Gaussian mixture model, after a picture is sent to the generator in the convolutional neural network, the feature map obtained by the convolution block output in the generator is X, and the size is C×H×W, where C is the number of channels, and H and W are the height and width of the feature map respectively; X is the input image obtained after passing the activation function of the convolution block in the generator; changing X shape to C×N, where N=H×W;

N-dimensional vector representing the ith channel; given

and random sampling through the normal distribution, initialize a set of K basis vectors

composed of matrices

Where K<N; then step 3 is divided into the following three small steps; the first step is to estimate hidden variables

The second step is to use the result estimated in the first step to update the basis vector matrix M; the first and second steps iterate in a loop until μ and Z converge; the third step is to use M and Z to reconstruct X, and use step 2 The S obtained in the multiplication operation is performed on M; Step 4: Total Neural Network; Embed the adaptive channel domain EM attention module in step 3 into the generator in step 2, in a total of 3 different embeddings; the first is the first residual after the second layer downsampling the convolution block Before the network block, the second is replaced at the position of the 5th residual network block, and the third is embedded before the first upsampling convolution block after the last residual network block; features in the output of the fully connected neural network The graph control code d is multiplied into the outputs of all convolutional layers in the generator, while the base control code S is multiplied into the base M in the adaptive channel domain EM attention module obtained in step 3; the output of the generator is used as the discriminator’s Input, the output of the discriminator is the output of the total neural network; Step 5: Design the loss function; For the pictures obtained in step 1, record the shoe outline category picture as _IA , and the real shoe picture as _IB ; and perform random sampling from the normal distribution to obtain a vector v, the generator in step 2 and the fully connected network are recorded as G, the discriminator is recorded as D; the generator input in G is I _A , the input of the fully connected network is v, the two work together and the output is recorded as G(I _A , v); the input of the discriminator is I _B and G(I _A , v), and their outputs are denoted as D(I _B ) and D(G(I _A , v)), respectively. Then the network loss can be described as:

is the loss function of the discriminator,

is the loss function of the generator;

respectively represent the expectation of (I _A , v) and I _B ; Step 6: Train the total neural network, use the loss function constructed in step 5 for network training, fix the parameters of D when updating G, and update D to fix the parameters of G, and update it alternately once per iteration; Step 7: In the testing phase, the model is trained in step 6, and only the G part of the network is taken; given an input image I _A , and different normal distribution samples v, multiple output images of different styles are obtained. 2. a kind of multimodal image style transfer method based on adaptive attention mechanism as claimed in claim 1 is characterized in that the concrete method of described step 3 is: Step 3.1: Estimate hidden variables

This step is to calculate the degree of responsibility of each basis for each channel, that is, the possibility that each channel belongs to each basis; z _ck represents the weight of the k-th basis to the c-th channel x _c in μ, where 1≤k≤K and 1≤c≤C; the posterior probability distribution of x _c conditional on μ _c is constructed as follows:

in

represents the generalized kernel function; then z _ck can be calculated by the following formula:

Kernel function

Choosing the form of exp(a ^T b), then for the t-th iteration, the hidden variable Z is calculated using the following formula: Z ^(t) = softmax(X(M ^(t-1) ) ^T ) Step 3.2: For the t-th iteration, by weighted summation of X, the update of the basis vector can be expressed as:

Step 3.3: After step 3.1 and step 3.2 are alternately performed T times, go to step 3.3, use M and Z to reconstruct X, and use S obtained in step 2 to multiply μ; for the S obtained in step 2 , whose length is K, which is equal to the number of bases μ; then we can finally use the following formula to reconstruct X:

Images