CN110659727B - A Sketch-Based Image Generation Method - Google Patents

Abstract

A sketch-based image generation method, comprising: S1, constructing an adversarial generative model, and the adversarial generative model includes a generator and a discriminator, wherein the generator includes a down-sampling mask residual module, a residual module, and an up-sampling mask residual module Difference module and conditional self-attention module, the discriminator includes more than one sub-discriminatory network of different depths; S2, input one or more training sample sketches into the generator to generate the generated image corresponding to each training sample sketch; S3, train the training sample sketch The real image corresponding to the sample sketch and the generated image are input to the discriminator to calculate the loss function, and the training objective function is calculated according to the loss function; S4, the parameters of the generator and the discriminator are trained according to the training objective function, so that the training objective function is reduced to Minimum; S5, use the trained generator to generate the target generated image corresponding to the target sketch. This method ensures that the generated face images have real local textures and complete face structures.

Description

A Sketch-Based Image Generation Method

technical field

The present disclosure relates to the technical field of image processing, and in particular, to a sketch-based image generation method.

Background technique

Sketch-based image generation is a special case of image-to-image translation that plays an important role in computer graphics. Early sketch-based image generation algorithms used the method of search fusion, which first searched for sketch-related image patches from a large-scale image database, and then fused the image patches. With the rapid development of deep learning in recent years, generative adversarial networks have been increasingly used in image-to-image translation. Isola et al. proposed a general model for image-to-image translation based on supervised training conditional generative adversarial networks. This model is only for dense images, and the generation effect of sparse images such as sketches is not satisfactory.

Due to the diversity, abstraction, and sparseness of sketches, sketch-based face image generation faces great challenges. Existing methods still cannot generate ideal face images, especially when the input sketches do not contain complete face structures (eyes, nose, mouth, etc.), the generated face images often have correspondingly missing parts of the face structure. This is because the network models of the existing methods are all based on convolutional layers, and the perceptual field of the convolutional layer is very limited, and the global perceptual field needs to be achieved through multi-layered convolutional layers, but the multi-layered convolutional layers may make The network cannot easily learn the global structural information; in addition, the realism discriminator used by the generative adversarial network of the existing method is to locally discriminate the image blocks, which can only ensure the local realism of the generated image, and cannot directly discriminate others. Structural integrity of face images.

SUMMARY OF THE INVENTION

(1) Technical problems to be solved

In view of the above problems, the present disclosure provides a sketch-based image generation method, which solves the above technical problems by introducing a self-attention mechanism and a multi-scale discriminator into a conditional generative adversarial network.

(2) Technical solutions

The present disclosure provides a sketch-based image generation method, including: S1, constructing an adversarial generative model, where the adversarial generative model includes a generator and a discriminator, wherein the generator includes a downsampling mask residual module, a residual A difference module, an upsampling mask residual module and a conditional self-attention module, the discriminator includes more than one sub-discriminatory network of different depths; S2, one or more training sample sketches are input into the generator to generate each The generated image corresponding to the training sample sketch; S3, input the training sample sketch and its corresponding real image and generated image to the discriminator to calculate a loss function, and calculate a training objective function according to the loss function; S4, the parameters of the generator and the discriminator are trained according to the training objective function, so that the training objective function is minimized; S5, the trained generator is used to generate the target generation image corresponding to the target sketch.

Optionally, the number of the downsampling mask residual module and the upsampling mask residual module is N, the downsampling mask residual module, the residual module, the first N-1 upsampling modules. The mask residual module, the conditional self-attention module, and the last upsampling mask residual module are connected in sequence, and the outputs of the first to N-1th downsampling mask residual modules are also connected to the Nth to Nth ones, respectively. For the second up-sampling mask residual module, the step S2 includes: inputting the training sample sketch into the first down-sampling mask residual module, and sequentially passing through the down-sampling mask residual module and the residual module , the first N-1 upsampling mask residual modules, the conditional self-attention module and the last upsampling mask residual module are processed, and the last upsampling mask residual module outputs the corresponding training sample sketch. generated image.

Optionally, the processing of the conditional self-attention module includes: concatenating the received input feature map with the training sample sketch to obtain conditional features; respectively mapping the conditional features according to three mapping matrices to obtain three mapping feature maps, the mapping matrix is composed of trainable parameters; processing the three mapping feature maps to obtain response maps; adding the response maps to the input feature maps to obtain output features picture.

Optionally, the mapping feature maps are respectively: _f ([a,x])=Wf[a,x], _g ([a,x])=Wg[a,x], h([a , x])=W _h [a, x], where a is the input feature map, x is the scaled image of the training sample sketch at the same resolution as the feature map, f([a, x]), g ([a, x]), h([a, x]) are the three mapping feature maps respectively, W _f , W _g , W _h are the mapping matrices, a∈R ^C×H×W , x ∈R ^1×H×W , W _f ∈R ^D×(C+1) , W _g ∈R ^D×(C+1) , W _h ∈R ^C×(C+11) , D=C/8, C, H, and W are the channel number, height, and width of the input feature map, respectively.

Optionally, the processing of the three mapping feature maps to obtain a response map includes: performing the mapping feature map f([a, x]) and the mapping feature map g([a, x]) on process to obtain the attention map; process the attention map and the mapping feature map h([a, x]) to obtain the response map.

s_i，j＝f([a，x])^Tg([a，x])。Optionally, the response graph is: r=(r ₁ , r ₂ ,...,r _N )∈R ^C×N , where r is the response graph, N=H×W,

s _i,j = f([a, x]) ^T g([a, x]).

Optionally, the output feature map is: o _j =γr _i +a _j , where γ is a trainable weight parameter, the initial value is 0, o _j is the jth pixel of the output feature map, r _j is the jth pixel of the response map, and a _j is the jth pixel of the input feature map received by the conditional self-attention module.

Optionally, the discriminator is composed of more than one convolutional layer, the number of convolutional layers of different sub-discriminant networks is different, and the hyperparameters of each sub-discriminant network are the same.

Optionally, the loss function includes an adversarial loss function L _adv (G; D), a reconstruction loss function L _L1 (G) and a feature matching loss function L _fm (G), wherein:

其中，x为所述训练样本草图，y为所述真实图像，N_D为所述子判别网络的个数，G(x)为所述训练样本草图对应的生成图像，D_k(x，y)为第k个子判别网络根据训练样本草图和真实图像得到的输出，D_k(x，G(x))为第k个子判别网络根据训练样本草图和生成图像得到的输出，

为在(x，y)的数据分布上求期望，

为在x的数据分布上求期望，Q为选出来的子判别网络的特征层的集合，N_Q为每个子判别网络的元素个数，n_q为第q层特征层的元素个数，

为第k个子判别网络根据生成图像的第q层的中间输出特征图，

为第k个子判别网络根据真实图像的第q层的中间输出特征图，λ为重建损失函数L_L1(G)的权重，μ为特征匹配损失函数L_fm(G)的权重。The training objective function is

Wherein, x is the sketch of the training sample, y is the real image, N _D is the number of the sub-discriminatory networks, G(x) is the generated image corresponding to the sketch of the training sample, D _k (x, y ) is the output obtained by the kth sub-discriminant network according to the training sample sketch and the real image, D _k (x, G(x)) is the output obtained by the kth sub-discriminant network according to the training sample sketch and the generated image,

To find the expectation on the data distribution of (x, y),

In order to find the expectation on the data distribution of x, Q is the set of feature layers of the selected sub-discriminant network, N _Q is the number of elements of each sub-discriminant network, n _q is the number of elements of the qth layer feature layer,

is the intermediate output feature map of the qth layer of the generated image for the kth sub-discriminant network,

is the intermediate output feature map of the kth sub-discriminant network according to the qth layer of the real image, λ is the weight of the reconstruction loss function L _L1 (G), and μ is the weight of the feature matching loss function L _fm (G).

Optionally, the step S4 includes: training parameters other than the self-attention module in the confrontation generation model according to the training objective function; fixing parameters other than the self-attention module in the confrontation generation model, training the self-attention module. The parameters of the attention module; meanwhile, the parameters in the adversarial generative model are trained to minimize the target training function.

(3) Beneficial effects

The sketch-based image generation method provided by the present disclosure has the following beneficial effects:

(1) By introducing a conditional self-attention module into the conditional generative adversarial network, the long-distance dependencies of images can be directly learned;

(2) By introducing a multi-scale discriminator into the conditional generative adversarial network, the authenticity of the generated image can be discriminated at different scales to ensure that the local texture and details of the generated image are realistic, and when the sketch is incomplete The structure of the generated image is complete;

(3) By sharing the parameters of the previous layers in each sub-discrimination network, the number of parameters of the network can be reduced, and it is beneficial to the network convergence.

Description of drawings

FIG. 1 schematically shows a flowchart of a sketch-based image generation method provided by an embodiment of the present disclosure;

2A schematically shows a schematic structural diagram of a generator for constructing an adversarial generation model in the sketch-based image generation method provided by an embodiment of the present disclosure;

FIG. 2B schematically shows a schematic structural diagram of a discriminator for constructing an adversarial generation model in the sketch-based image generation method provided by an embodiment of the present disclosure;

Figure 3 schematically shows a schematic diagram of the conditional self-attention module in the generator shown in Figure 2A.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the specific embodiments and the accompanying drawings.

This embodiment provides a sketch-based image generation method. Referring to FIG. 1 , the method shown in FIG. 1 is described in detail with reference to FIGS. 2A , 2B and 3 , and the method includes the following operations.

S1, build an adversarial generative model. The adversarial generative model includes a generator and a discriminator. The generator includes a downsampling mask residual module, a residual module, an upsampling mask residual module and a conditional self-attention module. The discriminator Include more than one sub-discriminative network of different depths.

In this embodiment, the generator is a network with an "encoding-decoding" structure. The encoding part includes N downsampling mask residual modules, the decoding part includes N upsampling mask residual modules, and a conditional self-attention module is inserted before the last upsampling mask residual module. The conditional self-attention module can effectively learn the long-distance dependency information in the input feature map and help the network learn the structural information of the image. Several residual modules can also be added between the encoding part and the decoding part to enhance the capacity and fitting ability of the network.

Specifically, referring to FIG. 2A, the number of downsampling mask residual modules and upsampling mask residual modules is the same, both are N, the downsampling mask residual module, the residual module, and the first N-1 upsampling mask residual modules. The code residual module, the conditional self-attention module and the last upsampling mask residual module are connected in turn, and the outputs of the first to N-1th downsampling mask residual modules are also connected to the Nth to Nth The two upsampling mask residual modules, namely the decoding part and the encoding part, are connected by jumps to ensure that the lower layer information (ie the output information of the downsampling module) can be directly transferred to the high layer (ie the upsampling module). In the embodiment of the present disclosure, being connected in sequence means that the output of the previous module is connected to the input of the next module, that is, the output feature map of the previous module is used as the input feature map of the latter module.

It can be understood that, the confrontation generation model in the embodiment of the present disclosure may also not include residual modules, that is, the number of residual modules is 0. The greater the number of residual modules, the stronger the fitting ability of the generator. However, when there are too many residual modules, it will affect the training convergence speed and calculation speed of the adversarial generation model. Therefore, it is preferable to set the number of residual modules to 1 within -8.

In the embodiment of the present disclosure, taking N=6 and the number of residual modules as 8 as an example, the parameters of the generator are schematically illustrated. Taking the size of the training sample sketch as 256×256×1 as an example, it is schematically illustrated that every The output size of a module is shown in Table 1. Meanwhile, those skilled in the art can obtain generators composed of down-sampling mask residual modules, up-sampling mask residual modules, residual modules, and conditional self-attention modules with other numbers and parameters according to the description of this embodiment.

Table 1

The input of the discriminator is the generated image and the real image corresponding to the training sample sketch, and the output is the judgment of the realism of the input image. Referring to FIG. 2B , according to an embodiment of the present disclosure, the discriminator is composed of a plurality of sub-discriminatory networks, each of which has more than three convolutional layers, and the weights of the first three convolutional layers in each sub-discriminative network are shared. The hyperparameters of each sub-discriminative network are the same, and only the number of convolutional layers is different. Hyperparameters are preset parameters, such as convolution kernel size, stride, etc. Since the depths of each sub-discriminant network are different, each pixel of the output feature map corresponds to a different perceptual field of the input image, so the sub-discriminatory networks of different depths respectively discriminate the realism of the generated image at different scales; The underlying features of the network are consistent, and each sub-discriminatory network shares the parameters of the previous layers, which can reduce the number of network parameters and is conducive to network convergence.

In the embodiment of the present disclosure, the discriminator is a multi-scale discriminator, and each sub-discrimination network has the same structure and different depths. Taking the sub-discriminant network with the maximum depth as an example, which includes 8 convolution modules, the structure of the multi-scale discriminator is schematically illustrated, as shown in Table 2, which only shows the sub-discriminatory network with the maximum depth. In the embodiment shown in Table 2, the parameters of convolution module 1, convolution module 2, and convolution module 3 are shared by each sub-discriminant network, and on this basis, other convolution modules are added to form a sub-discriminant network. In practical applications, the number of sub-discriminant networks can be selected according to requirements. For example, only two sub-discriminant networks are set up, which are the sub-discriminatory networks composed of convolution modules 1, 2, 3, and 4, and the convolution modules 1, 2, 3, and 3 respectively. The sub-discriminant network composed of 4, 5, 6, 7, and 8.

Table 2

S2, inputting one or more training sample sketches into the generator to generate a generated image corresponding to each training sample sketch.

Specifically, the training sample sketch is input into the first downsampling mask residual module, followed by the downsampling mask residual module, the residual module, the first N-1 upsampling mask residual modules, and the conditional self-attention After processing by the module and the last upsampling mask residual module, the last upsampling mask residual module outputs the generated image corresponding to each training sample sketch.

Referring to Fig. 3, the processing process of the conditional self-attention module is: concatenate the received input feature map with the training sample sketch to obtain the conditional feature; map the conditional feature according to the three mapping matrices to obtain three mapping feature maps, The mapping matrix consists of a large number of trainable parameters; the three mapping feature maps are processed to obtain a response map; the response map is added element-wise to the input feature map to obtain an output feature map.

Specifically, the input feature map a received by the conditional self-attention module is the output feature map of the N-1th upsampling module and the output feature map of the first downsampling module, a∈R ^C×H×W . First, the size of the training sample sketch needs to be modified so that the size of the training sample sketch x is x∈R ^1×H×W , and concatenated in the channel direction to obtain the conditional features [a, x].

The conditional features are mapped according to the three mapping matrices to map the conditional features to three new feature spaces, and three mapped feature maps are obtained, which are:

f([a,x])=W _f [a,x]

g([a,x])=W _g [a,x]

h([a,x])=W _h [a,x]

Among them, a is the input feature map, x is the scaled image of the training sample sketch at the same resolution as the feature map, f([a, x]), g([a, x]), h([a, x]) respectively are three mapping feature maps, W _f , W _g , W _h are mapping matrices, a∈R ^C×H×W , x∈R ^{1 ×H×W} , W _f ∈R ^D×(C+1) , W _g ∈ R ^D×(C+1) , W _h ∈ R ^C×(C+1) , D=C/8, C, H, and W are the number of channels, height and width of the input feature map, respectively. In the embodiment of the present disclosure, instead of explicitly using a matrix to implement W _f , W _g , and W _h , a 1×1 convolution layer is used to implement W _f , W _g , and W _h , respectively.

Further, after processing the mapped feature map f([a, x]) and the mapped feature map g([a, x]), the attention map is obtained, and the attention map and the mapped feature map g([a, x] ) to get the response graph.

Specifically, the mapped feature map f([a, x]) is transposed and multiplied by the mapped feature map g([a, x]), and normalized to obtain the attention map, and the attention map is combined with the map The feature map h([a, x]) is multiplied to get the response map r.

r=(r ₁ , r ₂ ,...,r _N )∈R ^C×N

s_i，j＝f([a，x])^Tg([a，x])。令B＝R^N×N为注意力图，B的每个元素记为b_i，j，表示在合成响应图的第j个像素的时候，当前映射特征图h([a，x])第i个个像素的权重。Among them, N=H×W,

s _i,j = f([a, x]) ^T g([a, x]). Let B=R ^N×N be the attention map, and each element of B is denoted as bi _,j , indicating that when the jth pixel of the response map is synthesized, the current mapping feature map h([a, x]) The ith pixel The weight of each pixel.

Further, the response map is added to the input feature map element by element to form a residual structure, and the output feature map of the conditional self-attention module is obtained:

o _j =γr _j +d _j

Among them, γ is the trainable weight parameter, the initial value is 0, o _j is the j-th pixel of the output feature map, r _j is the j-th pixel of the response map, and a _j is the input feature received by the conditional self-attention module The jth pixel of the graph.

Through the above conditional self-attention module, the generator can gradually learn the long-range dependencies of images.

S3, input the training sample sketch and its corresponding real image and generated image into the discriminator to calculate the loss function, and calculate the training objective function according to the loss function.

In the embodiment of the present disclosure, the loss function includes an adversarial loss function L _adv (G; D), a reconstruction loss function L _L1 (G), and a feature matching loss function L _fm (G), wherein:

为在(x，y)的数据分布上求期望，

为在x的数据分布上求期望，Q为选出来的子判别网络的特征层的集合，N_Q为每个子判别网络的元素个数，n_q为第q层特征层的元素个数，

为第k个子判别网络根据生成图像的第q层的中间输出特征图，

为第k个子判别网络根据真实图像的第q层的中间输出特征图。x is the training sample sketch, y is the real image, N _D is the number of sub-discriminant networks, G(x) is the generated image corresponding to the training sample sketch, D _k (x, y) is the kth sub-discriminant network according to the training sample The output obtained from the sketch and the real image, D _k (x, G(x)) is the output obtained by the kth sub-discriminant network based on the training sample sketch and the generated image,

To find the expectation on the data distribution of (x, y),

In order to find the expectation on the data distribution of x, Q is the set of feature layers of the selected sub-discriminant network, N _Q is the number of elements of each sub-discriminant network, n _q is the number of elements of the qth layer feature layer,

is the intermediate output feature map of the qth layer of the generated image for the kth sub-discriminant network,

is the intermediate output feature map of the qth layer of the real image for the kth sub-discriminative network.

The adversarial loss function _{La adv} (G; D) is used to make the generative adversarial network (ie, the adversarial generative model) complete the adversarial training to ensure the realism of the generated images. The reconstruction loss function L _L1 (G) is used to make the generated image closer to the real image. The feature matching loss function L _fm (G) is used to make the generated image close to the real image in the feature space.

Further, the training objective function of the adversarial generative model can also be calculated according to _{La adv} (G; D), L _L1 (G) and L _fm (G):

Among them, λ and μ are the weights of the reconstruction loss function L _L1 (G) and the feature matching loss function L _fm (G) respectively. For example, λ=100.0 and μ=1.0 are selected in the experiment.

S4, train the parameters of the generator and the discriminator according to the training objective function, so as to minimize the training objective function.

The parameters of the generator and the discriminator are alternately trained, that is, the parameters of the discriminator are fixed first, and the generator is trained for one iteration; when the generator is fixed, the discriminator is trained for one iteration, which is repeated alternately.

Further, in order to better converge against the generative model, operation S4 is divided into three stages, including the following operations:

According to the training objective function, the parameters other than the self-attention module in the adversarial generative model are trained, that is, the model other than the conditional self-attention module is trained; the parameters other than the self-attention module in the adversarial generative model are fixed, and the parameters of the training self-attention module are Only the conditional self-attention module is trained; the parameters in the adversarial generative model are also trained to minimize the training objective function, that is, all trainable parameters in the model are simultaneously trained.

S5, using the trained generator to generate the target generated image corresponding to the target sketch.

The training of the adversarial generative model in the embodiment of the present disclosure is completed according to the above-mentioned operations S1-S4, that is, the target generative image corresponding to the target sketch can be generated according to the generator in the trained adversarial generative model.

The advantages of the method of the present disclosure will be explained below by comparing the sketch-based image generation method of the present disclosure with the existing methods pix2pix and SketchyGAN with better performance.

IS, FID, KID are three commonly used indicators to evaluate the quality of generated images. The higher the IS score, the higher the realism and variety of the generated images; the lower the FID and KID values, the higher the realism of the generated images. Table 3 objectively shows the IS, FID, and KID indicators of the disclosed method and pix2pix and SketchyGAN. It can be seen that the disclosed method still performs better than pix2pix even if the conditional self-attention module or multi-scale discriminator is not used. , SketchyGAN, when using both conditional self-attention module and multi-scale discriminator, its performance is far better than pix2pix, SketchyGAN. Thus, it is also illustrated that both the conditional self-attention module and the multi-scale discriminator in the present disclosure can effectively enhance the generation effect of the adversarial generative model.

table 3

Model IS FID KID pix2pix 2.55±0.20 605.97±13.95 3.05±0.08 SketchyGAN 2.75±0.17 479.09±15.25 2.11±0.09 Method of the present disclosure (removal of conditional self-attention module) 3.73±0.14 451.58±14.91 1.88±0.07 Method of the present disclosure (removal of multi-scale discriminator) 2.83±0.15 409.36±17.04 1.37±0.03 Methods of the present disclosure 2.99±0.21 333.19±16.75 0.91±0.05

In summary, the sketch-based image generation method provided by the present disclosure introduces a conditional self-attention module and a multi-scale discriminator into the conditional generative adversarial network. The conditional self-attention module enables the adversarial generative model to directly learn the length of the image. Distance-dependent information, the multi-scale discriminator ensures the authenticity of the local texture and the integrity of the global structure of the generated image, making the adversarial generative model more robust.

The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present disclosure in detail. It should be understood that the above-mentioned specific embodiments are only specific embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure should be included within the protection scope of the present disclosure.

Claims (10)

1. A sketch-based image generation method, comprising: S1, build an adversarial generative model, the adversarial generative model includes a generator and a discriminator, wherein the generator includes a down-sampling mask residual module, a residual module, an up-sampling mask residual module and a conditional self-attention module, the discriminator includes a plurality of sub-discrimination networks of different depths, and the conditional self-attention module is used to learn the long-distance dependence of the input feature image of the input conditional self-attention module; S2, inputting more than one training sample sketch into the generator to generate a generated image corresponding to each of the training sample sketches; S3, inputting the training sample sketch and its corresponding real image and generated image to the discriminator, to calculate a loss function, and calculate a training objective function according to the loss function; S4, train the parameters of the generator and the discriminator according to the training objective function, so that the training objective function is minimized; S5, using the trained generator to generate a target generation image corresponding to the target sketch. 2 . The sketch-based image generation method according to claim 1 , wherein the number of the downsampling mask residual module and the upsampling mask residual module is N, and the downsampling mask residual module is N. 3 . The difference module, the residual module, the first N-1 upsampling mask residual modules, the conditional self-attention module and the last upsampling mask residual module are connected in sequence, and the first to N-1th downsampling mask The output of the code residual module is also connected to the Nth to second up-sampling mask residual modules respectively, and the step S2 includes: The training sample sketch is input into the first down-sampling mask residual module, followed by the down-sampling mask residual module, the residual module, the first N-1 up-sampling mask residual modules, and the conditional self-attention After the force module and the last upsampling mask residual module are processed, the last upsampling mask residual module outputs the generated image corresponding to each of the training sample sketches. 3. The sketch-based image generation method of claim 1, wherein the processing of the conditional self-attention module comprises: Concatenate the received input feature map with the training sample sketch to obtain conditional features; The conditional features are respectively mapped according to three mapping matrices, to obtain three mapping feature maps, and the mapping matrices are composed of trainable parameters; processing the three mapping feature maps to obtain a response map; The response map is added to the input feature map to obtain an output feature map. 4. The sketch-based image generation method according to claim 3, wherein the mapping feature maps are respectively: f([a,x])=W _f [a,x] g([a,x])=W _g [a,x] h([a,x])=W _h [a,x] Among them, a is the input feature map, x is the scaled image of the training sample sketch at the same resolution as the feature map, f([a,x]), g([a,x]), h([a, x]) are the three mapping feature maps respectively, W _f , W _g , W _h are the mapping matrices, a∈R ^{C ×H×W} , x∈R ^1×H×W , W _f ∈R ^{D ×(C+1)} , W _g ∈R ^D×(C+1) , W _h ∈R ^C×(C+1) , D=C/8, C, H, W are the Number of channels, height and width. 5. The sketch-based image generation method according to claim 4, wherein the processing of the three mapping feature maps to obtain a response map comprises: Process the mapped feature map f([a,x]) and the mapped feature map g([a,x]) to obtain an attention map; The attention map and the mapped feature map h([a,x]) are processed to obtain the response map. 6. The sketch-based image generation method of claim 5, wherein the response graph is: r=(r ₁ ,r ₂ ,...,r _N )∈R ^C×N where r is the response graph, N=H×W,

_si,j = f([a,x]) ^T g([a,x]). 7. The sketch-based image generation method according to claim 6, wherein the output feature map is: o _j =γr _j +a _j Among them, γ is a trainable weight parameter, the initial value is 0, o _j is the j-th pixel of the output feature map, r _j is the j-th pixel of the response map, and a _j is the conditional self-attention The jth pixel of the input feature map received by the module. 8. The sketch-based image generation method according to claim 1, wherein the discriminator is composed of more than one convolutional layer, the number of convolutional layers of different sub-discriminant networks is different, and the hyperparameters of each sub-discriminant network are different. same. 9. The sketch-based image generation method according to claim 1, wherein the loss function comprises an adversarial loss function L _adv (G; D), a reconstruction loss function L _L1 (G) and a feature matching loss function L _fm ( G), where:

The training objective function is

Wherein, x is the sketch of the training sample, y is the real image, _ND is the number of the sub-discriminatory networks, _ND is an integer greater than 1, and G(x) is the corresponding generation of the sketch of the training sample Image, D _k (x, y) is the output obtained by the kth sub-discriminant network based on the training sample sketch and real image, D _k (x, G(x)) is the k-th sub-discriminant network obtained according to the training sample sketch and the generated image Output,

To find the expectation on the data distribution of (x,y),

In order to find the expectation on the data distribution of x, Q is the set of feature layers of the selected sub-discriminant network, N _Q is the number of elements of each sub-discriminant network, n _q is the number of elements of the qth layer feature layer,

is the intermediate output feature map of the qth layer of the generated image for the kth sub-discriminant network,

is the intermediate output feature map of the kth sub-discriminant network according to the qth layer of the real image, λ is the weight of the reconstruction loss function L _L1 (G), and μ is the weight of the feature matching loss function L _fm (G). 10. The sketch-based image generation method according to claim 1, wherein the step S4 comprises: Train parameters other than the self-attention module in the confrontation generative model according to the training objective function; Fixing parameters other than the self-attention module in the confrontation generation model, and training the parameters of the self-attention module; At the same time, parameters in the adversarial generative model are trained to minimize the target training function.

Images