CN116524290A - Image synthesis method based on countermeasure generation network - Google Patents

Abstract

The invention discloses an image synthesis method based on an countermeasure generation network, which comprises the following steps: s1, acquiring a first data set and a second data set, wherein each sample in the first data set comprises a first splicing unit, a second splicing unit and a third splicing unit, each sample in the second data set comprises a fourth splicing unit, a fifth splicing unit and transformation parameters corresponding to the fifth splicing unit, and each splicing unit has the same size; s2, building an countermeasure generation network model and training; s3, inputting the first splicing unit and the second splicing unit to be synthesized into a trained countermeasure generation network model, and taking the first synthesis unit correspondingly output by the discriminator under the unsupervised path as an image synthesis prediction result. The method improves the rationality and diversity of the synthetic image, can obtain a more real and natural synthetic image, and can adapt to complex application scenes.

Description

An image synthesis method based on generative adversarial network

Technical Field

The present invention belongs to the technical field of image processing, and in particular relates to an image synthesis method based on a generative adversarial network.

Background Art

Image synthesis includes four research directions: object placement, image blending, image harmonization, and shadow generation. For the object placement problem, the geometric inconsistencies include but are not limited to: 1) the foreground object is too large or too small; 2) the foreground object is not supported by force, such as floating in the air; 3) the foreground object appears in a semantically inappropriate place, such as a ship appearing inland; 4) there is an unreasonable occlusion relationship between the foreground object and the surrounding objects; 5) the perspective angles of the foreground and background are inconsistent. In summary, the size, position, and shape of the foreground object are unreasonable. Object placement and spatial transformation aim to find a reasonable size, position, and shape for the foreground, thereby avoiding the many unreasonable factors mentioned above. Object placement generally refers to the translation and scaling of the foreground object, while spatial transformation involves relatively complex geometric deformations, such as affine transformation or perspective transformation. For the convenience of description, object placement refers to any geometric deformation.

Learning to place foreground objects on a background scene often appears in applications such as image editing and scene parsing. The original composite image and the foreground mask are input into the model, and the output is adjusted to obtain a more realistic and natural composite image. So far, most of the existing related studies have some thorny problems, including insufficient utilization of the interactive feature relationship between the foreground object and the scene, less prior knowledge involved in the model training process, etc., resulting in unreasonable positions of foreground objects in the synthesized result image. For example, most of the work only considers pasting a foreground object onto another background image and assumes that the foreground object is complete. However, in real applications, it is often necessary to synthesize multiple foreground objects on the same background image, and the foreground object may be incomplete. In addition, the existing technology usually only outputs one position for the same input. However, for a foreground object and a background image, in most cases, the foreground object has multiple reasonable positions on the background image, such as placing a vase on a table in another background image. There are countless reasonable sizes and positions. Therefore, in order to obtain more reasonable and diverse synthesis results, it is necessary to improve the image synthesis algorithm so that it can adapt to complex application scenarios.

Summary of the invention

The purpose of the present invention is to propose an image synthesis method based on a generative adversarial network to address the above problems, thereby improving the rationality and diversity of the synthesized images, obtaining more realistic and natural synthesized images, and being able to adapt to complex application scenarios.

To achieve the above object, the technical solution adopted by the present invention is:

The present invention proposes an image synthesis method based on a generative adversarial network, comprising the following steps:

S1. Obtain a first data set and a second data set, wherein each sample in the first data set includes a first splicing unit [I _fg ,M _fg ], a second splicing unit [I _bg ,M _bg ] and a third splicing unit [I _gt ,M _gt ], and each sample in the second data set includes a fourth splicing unit Fifth splicing unit The transformation parameter corresponding to the fifth splicing unit is t _gt =(t _r ^gt ,t _x ^gt , _ty ^gt ), each splicing unit has the same size, wherein _Ifg is the foreground image, M _fg is the foreground image mask, _Ibg is the background image, M _bg is the background image mask, _Igt is the positive label image, _Mgt is the positive label image mask, represents the negative label synthetic graph, represents the negative label composite image mask, represents the positive label composite image, represents the positive label composite image mask, Indicates the scaling ratio of the foreground object corresponding to the fifth splicing unit, Indicates the x-axis coordinate of the foreground position corresponding to the fifth splicing unit on the background image, Indicates the y-axis coordinate of the foreground position corresponding to the fifth splicing unit on the background image;

S2. Establish a generative adversarial network model and train it. The generative adversarial network model includes a generator, a discriminator, and a priori knowledge extractor. The generator includes a preliminary feature extractor, a multi-scale feature aggregation module, a joint attention module, a Concat function, and a regression block. The multi-scale feature aggregation module includes two parallel first feature extraction units. The first feature extraction unit includes a multi-scale encoder and a feature aggregator connected in sequence. The priori knowledge extractor includes a global feature extractor and an autoencoder connected in sequence. The training process is as follows:

S21, inputting the first concatenation unit [I _fg , M _fg ], the second concatenation unit [I _bg , M _bg ] and the third concatenation unit [I _gt , M _gt ] of each sample in the first data set into a preliminary feature extractor to obtain a first basic feature map F _fg , a second basic feature map F _bg and a third basic feature map F _gt ;

S22, inputting the first basic feature map F _fg and the second basic feature map F _bg into the first feature extraction unit of the multi-scale feature aggregation module in a one-to-one correspondence, obtaining the first multi-scale feature map P _fg and the second multi-scale feature map P _bg correspondingly, and inputting the first multi-scale feature map P _fg and the second multi-scale feature map P _bg into the joint attention module to obtain the global interaction feature Z;

S23, inputting the third basic feature graph F _gt into the global feature extractor of the prior knowledge extractor to obtain a first extracted feature, and encoding the first extracted feature into a priori vector z _p through an automatic encoder;

S24, the random variable _Zu and the prior vector _Zp are respectively fused with the global interaction feature Z through the Concat function, and the first concatenation vector _Zi is obtained to form an unsupervised path and the second concatenation vector _Zj is obtained to form a self-supervised path;

S25. Input the first splicing vector _Zi and the second splicing vector _Zj into the regression block respectively, and predict the first affine transformation parameter _tu = ( _tru ^, _txu , _tyu ) ^and _{the second affine transformation parameter ts = (trs, txs, tys )} ^{correspondingly , tru represents the} _scaling rate of the foreground object under the unsupervised path, _txu represents the x-axis coordinate of the foreground position on the background image under _the unsupervised path, ^tyu represents the y _- ^axis coordinate of the foreground position on the background image under the unsupervised path, _trs represents the scaling rate of the foreground object under the self-supervised path, _txs represents the x- ^axis coordinate of the ^foreground position on the background image under the self-supervised path, and _tys represents the y _- axis coordinate of the foreground position on the background image under the ^self -supervised path;

S26. Perform affine transformation on the input foreground image ^{and the foreground image mask according to the first affine transformation parameter tu = (tr u, tx u, ty u) and the second affine transformation parameter ts = (trs, txs, tys} _{) ,} ^respectively _, ^to _obtain ^a _{first affine transformation} ^result _and ^a second affine transformation result, the first affine transformation result including the foreground image after affine transformation under the unsupervised path. and foreground mask The second affine transformation result includes the foreground image after affine transformation under the self-supervised path and foreground mask

S27, synthesize the first affine transformation result and the second affine transformation result with the background image respectively, and obtain a first synthesized image And the second composite image It is expressed as follows:

S28, the first synthesis unit and the second synthesis unit Input the discriminator and train the discriminator using the second data set;

S29, calculate the joint loss to update the network parameters and obtain the trained adversarial generation network model;

S3, input the first splicing unit [I _fg ,M _fg ] and the second splicing unit [I _bg ,M _bg ] to be synthesized into the trained adversarial generative network model, and input the first synthesis unit corresponding to the output of the discriminator under the unsupervised path As the image synthesis prediction result.

Preferably, the preliminary feature extractor adopts the VGG16 network model.

Preferably, the preliminary feature extractor performs the following operations:

F ₁ = MaxPool(Conv64(Conv64(Input1))), with size H/2×W/2×64;

F ₂ =MaxPool(Conv128(Conv128(F ₁ ))), with size H/4×W/4×128;

F ₃ =MaxPool(Conv256(Conv256(Conv256(F ₂ )))), with size H/8×W/8×256;

_Ffg = MaxPool(Conv512(Conv512(Conv512( _F3 )))), with size H/16×W/16×512;

Among them, Maxpool represents the maximum pooling operation, ConvX represents the convolution operation with the number of channels as X, Input1 represents the first splicing unit [I _fg ,M _fg ] or the second splicing unit [I _bg ,M _bg ] or the third splicing unit [I _gt ,M _gt ], and the height, width and number of channels of each splicing unit are 4.

Preferably, the multi-scale encoder performs the following operations:

P ₁ =ReLU(BatchNorm(Conv512(Input2))), with size H ₁ ×W ₁ ×512;

P ₂ =ReLU(BatchNorm(Conv512(P ₁ ))), with size H ₁ ×W ₁ ×512;

P ₃ =ReLU(BatchNorm(ConvC(P ₂ ))), with size H ₁ ×W ₁ ×C;

_{PS1 =} AdaptiveAvgPool{ _S1 × _S1 }( _P3 ), with size _S1 × _S1 ×C;

PS2 ₌ AdaptiveAvgPool{ _S2 × _S2 }( _P3 ), with size _S2 × _S2 ×C;

PS3 ₌ AdaptiveAvgPool{S ₃ ×S ₃ }(P ₃ ), with size S ₃ ×S ₃ ×C;

Wherein, ReLU represents the ReLU activation function, BatchNorm represents the normalization operation, ConvX represents the convolution operation with the number of channels being X, AdaptiveAvgPool{n×n} represents adaptively pooling the height×width of the corresponding input image to n×n, S ₁ , S ₂ , and S ₃ are the first preset size, the second preset size, and the third preset size respectively, Input2 represents the first basic feature map F _fg or the second basic feature map F _bg , the height of each basic feature map is H ₁ , the width is W ₁ , the number of channels is C, H/16＝H ₁ , W/16＝W ₁ , and the height of each splicing unit is H and the width is W;

The feature aggregator performs the following operations:

_{PS1 =} Reshape( _P3 ), with size 1×( _{S1* S1} )×C;

PS2 ₌ Reshape( _P3 ), with size 1×( _{S2* S2} )×C;

PS3 ₌ Reshape( _P3 ), with size 1×( _{S3* S3} )×C;

_Pg = Concat(Concat( _PS1 , _PS2 ), _PS3 ), with size 1×(S1 _* S1 _{+ S2*} S2 _{+ S3* S3} )×C;

Among them, Reshape represents a reshape function, Concat represents a Concat function, and _Pg represents the output feature of the first feature extraction unit corresponding to Input2, that is, the first multi-scale feature map _Pfg or the second multi-scale feature map _Pbg .

Preferably, the global feature extractor performs the following operations:

Z ₁ =ReLU(BatchNorm(Conv512(F _fg ))), with size H ₁ ×W ₁ ×512;

Z ₂ =ReLU(BatchNorm(Conv512(Z ₁ ))), with size H ₁ ×W ₁ ×512;

Z ₃ =ReLU(BatchNorm(ConvC(Z ₂ ))), with size H ₁ ×W ₁ ×C;

Z _{4 =} AdaptiveAvgPool{1×1}(P ₃ ), with size 1×1×C;

The autoencoder performs the following operations:

h = ReLU(FC1024(Z ₄ )), with size 1×1×1024;

mu = FC512(h), size is 1 × 1 × 512;

Logvar=FC512(h), size is 1×1×512;

Z _p = mu + e ^Logvar/2 , with dimensions of 1 × 1 × 512;

Among them, FCY is a fully connected layer that maps the number of channels of the corresponding input image to Y.

Preferably, the joint attention module performs the following operations:

_Qfg = ConvC/8( _Pfg ), size is H×W×C/8;

_Kfg = ConvC/4( _Pfg ), size is H×W×C/4;

V _fg =ConvC/4(P _fg ), size is H×W×C;

Q _bg =ConvC/8(P _bg ), size is H×W×C/8;

K _bg =ConvC/4(P _bg ), size is H×W×C/4;

V _bg =ConvC/4(P _bg ), size is H×W×C;

The first dimension and the second dimension of Q _fg , K _fg , V _fg , Q _bg , K _bg , and V _bg are respectively integrated by the Reshpe function to obtain Q' _fg , K' _fg , V' _fg , Q' _bg , K' _bg , and V' _bg , which are expressed as follows:

The size of Q' _fg is HW × C/8;

The size of K' _fg is HW × C/4;

V' _fg has dimensions of HW × C;

The size of Q' _bg is HW × C/8;

The size of _K'bg is HW×C/4;

V' _bg size is HW × C;

The third dimension of _Q'fg and _Q'bg is concatenated and expressed as follows:

Q _cat = Concat(Q' _fg ,Q' _bg ), size is HW×C/4,

Perform attention calculation on Q _cat to obtain X _fg and X _bg , which are expressed as follows:

_Xfg = Softmax( _Qcat * _KfgT ⁾ _Vfg + _Pfg , size is HW×C;

X _bg = Softmax(Q _cat *K _bg ^T) V _bg +P _bg , size is HW×C;

Z = AdaptiveAvgPool{1×1}(Conv512(Concat( _Xfg , _Xbg ))), with size 1×1×C;

Among them, ConvX represents a convolution operation with X channels.

Preferably, the regression block performs the following operations:

t _u =FC3(FC1024(ReLU(FC1024(Z _i ))));

t _s =FC3(FC1024(ReLU(FC1024(Z _j ))));

Among them, FCY is a fully connected layer that maps the number of channels of the corresponding input image to Y, and ReLU represents the ReLU activation function.

Preferably, the affine transformation adopts Spatial Transformer Network.

Preferably, the discriminator performs the following operations:

R = Sigmoid(Conv1(

LeakyReLU(Conv512(

LeakyReLU(Conv256(

LeakyReLU(Conv128(

LeakyReLU(Conv64(Input)))))))))

Among them, Sigmoid represents the Sigmoid activation function, ConvX represents the convolution operation with the number of channels X, LeakyReLU represents the LeakyReLU activation function, R represents the output feature of the discriminator, and Input represents the input feature of the discriminator.

Preferably, the joint loss is expressed as follows:

in,

In the formula, θ _G represents the learning parameters in the generator G, θ _D represents the learning parameters in the discriminator D, Generate loss function for adversarial on unsupervised path, is the adversarial generation loss function on the self-supervised path, _Lkld (G) is the KL divergence loss function, _Lrec (G) is the reconstruction loss function, _Lbce (D) is the cross entropy loss function, represents the mean of the distribution of the prior vector z _p , represents the variance of the distribution of the prior vector z _p , D _KL represents the calculated KL divergence, N(a ₁ ,b ₁ ) represents the distribution with mean a ₁ and variance b ₁ , if a ₁ ＝0, b ₁ ＝1, then it is a normal distribution.

Compared with the prior art, the present invention has the following beneficial effects:

Based on the foreground image and background image and their masks, this application proposes a new end-to-end framework, namely, a generative adversarial network based on joint attention. Specifically, a multi-scale feature aggregation module is designed in the generator to extract multi-scale information from the background and foreground. After the multi-scale features of the foreground image and the background image are extracted, the joint attention module is used to extract the global feature interaction information of the foreground object and the background image, and based on these feature information, the affine transformation parameters of the foreground image are predicted. According to the predicted parameters, the foreground image and its mask are affine transformed and placed at the corresponding position of the background image to complete the synthesis of the input foreground image and background image. In addition, during the training process, a self-supervised route is added to learn prior knowledge from the positive label synthetic image, so as to further guide the generator to find the credible position of the foreground target in the background image, which can have advantages over other existing methods in terms of the rationality and diversity of the position in the result, and can obtain more realistic and natural synthetic images, and can adapt to complex application scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an image synthesis method based on a generative adversarial network according to the present invention;

FIG2 is a schematic diagram of the structure of a generative adversarial network model according to the present invention;

FIG3 is a schematic diagram of the structure of the joint attention module of the present invention;

FIG. 4 is a comparison diagram of the synthetic graph effects of the method of the present invention and the method of the prior art.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

It should be noted that when a component is referred to as being "connected" to another component, it may be directly connected to the other component or there may be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

As shown in Figure 1-4, an image synthesis method based on a generative adversarial network includes the following steps:

S1. Obtain a first data set and a second data set, wherein each sample in the first data set includes a first splicing unit [I _fg ,M _fg ], a second splicing unit [I _bg ,M _bg ] and a third splicing unit [I _gt ,M _gt ], and each sample in the second data set includes a fourth splicing unit Fifth splicing unit The transformation parameter corresponding to the fifth splicing unit is t _gt =(t _r ^gt ,t _x ^gt , _ty ^gt ), each splicing unit has the same size, wherein _Ifg is the foreground image, M _fg is the foreground image mask, _Ibg is the background image, M _bg is the background image mask, _Igt is the positive label image, _Mgt is the positive label image mask, represents the negative label synthetic graph, represents the negative label composite image mask, represents the positive label composite image, represents the positive label composite image mask, Indicates the scaling ratio of the foreground object corresponding to the fifth splicing unit, Indicates the x-axis coordinate of the foreground position corresponding to the fifth splicing unit on the background image, Indicates the y-axis coordinate of the foreground position corresponding to the fifth splicing unit on the background image.

In this embodiment, the adversarial generative network model includes a generator, a discriminator, and a priori knowledge extractor. In the training stage, the model training architecture is shown in FIG2. The input of the adversarial generative network model is the concatenation of the foreground image and its foreground image mask [I _fg , M _fg ] (first concatenation unit), the concatenation of the background image and its background image mask [I _bg , M _bg ] (second concatenation unit), and the concatenation of the positive label image and its positive label image mask [I _gt , M _gt ] (third concatenation unit), wherein the size of I _fg , I _bg and I _gt is H×W×3, and the size of M _fg , M _bg and M _gt is H×W×1. After concatenation, the size of [I _fg , M _fg ], [I _bg , M _bg ] and [I _gt , M _gt ] is H×W×4. In the training phase, the generator consists of an unsupervised path and a self-supervised path (the two paths are independent of each other). When the model is training, for the result image generated on the self-supervised path, the label result image (i.e., the second data set) is used to calculate the loss function, while on the unsupervised path, the label result image is not introduced for training. Its output is 2 sets of lists consisting of 3 transformation parameters respectively. The input foreground image and its foreground image mask are affine transformed according to the transformation parameters predicted by the model, and then synthesized into the background image, thereby generating two final synthesis units respectively. in and The dimensions are H×W×3, and The size of is H×W×1. Finally, the synthesis unit generated by the two paths is All the sizes are H×W×4 and are input to the discriminator. The generator and discriminator are trained separately according to the adversarial generation training model.

In the test phase, the model architecture used is the unsupervised path part of the generator in Figure 2. The input is [I _fg ,M _fg ] and [I _bg ,M _bg ], and the output is a list of 3 transformation parameters. The input foreground image and its mask are affine transformed according to the transformation parameters predicted by the model, and then synthesized into the background image, thus generating the final synthesis unit. Since the modules used in the testing phase are part of the model in the training phase, this paper mainly introduces the training process of the adversarial generative network model.

S2. Establish a generative adversarial network model and train it. The generative adversarial network model includes a generator, a discriminator, and a priori knowledge extractor. The generator includes a preliminary feature extractor, a multi-scale feature aggregation module, a joint attention module, a Concat function, and a regression block. The multi-scale feature aggregation module includes two parallel first feature extraction units. The first feature extraction unit includes a multi-scale encoder and a feature aggregator connected in sequence. The priori knowledge extractor includes a global feature extractor and an autoencoder connected in sequence. The training process is as follows:

S21. Input the first concatenated unit [I _fg , M _fg ], the second concatenated unit [I _bg , M _bg ] and the third concatenated unit [I _gt , M _gt ] of each sample in the first data set into a preliminary feature extractor to obtain a first basic feature map F _fg , a second basic feature map F _bg and a third basic feature map F _gt , respectively.

In one embodiment, the preliminary feature extractor adopts the VGG16 network model.

In one embodiment, the preliminary feature extractor performs the following operations:

F ₁ = MaxPool(Conv64(Conv64(Input1))), with size H/2×W/2×64;

F ₂ =MaxPool(Conv128(Conv128(F ₁ ))), with size H/4×W/4×128;

F ₃ =MaxPool(Conv256(Conv256(Conv256(F ₂ )))), with size H/8×W/8×256;

_Ffg = MaxPool(Conv512(Conv512(Conv512( _F3 )))), with size H/16×W/16×512;

Among them, Maxpool represents the maximum pooling operation, ConvX represents the convolution operation with the number of channels as X, Input1 represents the first splicing unit [I _fg ,M _fg ] or the second splicing unit [I _bg ,M _bg ] or the third splicing unit [I _gt ,M _gt ], and the height, width and number of channels of each splicing unit are 4.

Specifically, the preliminary feature extractor can refer to the VGG architecture, such as the VGG16 network model, etc. Taking [I _fg ,M _fg ] as an example, for the input [I _fg ,M _fg ], the size is H×W×4, and the corresponding first basic feature map F _fg is H ₁ ×W ₁ ×C. Let H/16＝H ₁ , W/16＝W ₁ , and the second basic feature map F _bg and the third basic feature map F _gt are similarly sized to H ₁ ×W ₁ ×C. Among them, the convolution layers of the preliminary feature extractor are all 3*3 convolution kernels, and X in ConvX represents the number of channels. The sliding step size of each convolution layer is stride=1, padding=1; Maxpool refers to the maximum pooling operation. In the VGG16 network model, the 2*2 maximum pooling method is adopted; padding refers to filling the matrix with n circles on the outside, and padding=1 means filling 1 circle. For example, a 5×5 matrix becomes 7×7 after filling one circle.

S22. Input the first basic feature map F _fg and the second basic feature map F _bg into the first feature extraction unit of the multi-scale feature aggregation module in a one-to-one correspondence to obtain the first multi-scale feature map P _fg and the second multi-scale feature map P _bg accordingly, and input the first multi-scale feature map P _fg and the second multi-scale feature map P _bg into the joint attention module to obtain the global interaction feature Z.

In one embodiment, the multi-scale encoder performs the following operations:

P ₁ =ReLU(BatchNorm(Conv512(Input2))), with size H ₁ ×W ₁ ×512;

P ₂ =ReLU(BatchNorm(Conv512(P ₁ ))), with size H ₁ ×W ₁ ×512;

P ₃ =ReLU(BatchNorm(ConvC(P ₂ ))), with size H ₁ ×W ₁ ×C;

_{PS1 =} AdaptiveAvgPool{ _S1 × _S1 }( _P3 ), with size _S1 × _S1 ×C;

PS2 ₌ AdaptiveAvgPool{ _S2 × _S2 }( _P3 ), with size _S2 × _S2 ×C;

PS3 ₌ AdaptiveAvgPool{S ₃ ×S ₃ }(P ₃ ), with size S ₃ ×S ₃ ×C;

Wherein, ReLU represents the ReLU activation function, BatchNorm represents the normalization operation, ConvX represents the convolution operation with the number of channels being X, AdaptiveAvgPool{n×n} represents adaptively pooling the height×width of the corresponding input image to n×n, S ₁ , S ₂ , and S ₃ are the first preset size, the second preset size, and the third preset size respectively, Input2 represents the first basic feature map F _fg or the second basic feature map F _bg , the height of each basic feature map is H ₁ , the width is W ₁ , the number of channels is C, H/16＝H ₁ , W/16＝W ₁ , and the height of each splicing unit is H and the width is W;

The feature aggregator performs the following operations:

_{PS1 =} Reshape( _P3 ), with size 1×( _{S1* S1} )×C;

PS2 ₌ Reshape( _P3 ), with size 1×( _{S2* S2} )×C;

PS3 ₌ Reshape( _P3 ), with size 1×( _{S3* S3} )×C;

_Pg = Concat(Concat( _PS1 , _PS2 ), _PS3 ), with size 1×(S1 _* S1 _{+ S2*} S2 _{+ S3* S3} )×C;

Among them, Reshape represents a reshape function, Concat represents a Concat function, and _Pg represents the output feature of the first feature extraction unit corresponding to Input2, that is, the first multi-scale feature map _Pfg or the second multi-scale feature map _Pbg .

The first basic feature map F _fg and the second basic feature map F _bg obtained, both of which have a size of H ₁ ×W ₁ ×C, are input into the multi-scale feature aggregator, thereby obtaining the first multi-scale feature map P _fg and the second multi-scale feature map P _bg that aggregate multiple scale information in each image. Taking the first basic feature map F _fg as an example, the multi-scale encoder pre-sets multiple scales S ₁ , S ₂ and S ₃ , firstly performs further feature extraction on the first basic feature map F _fg to obtain P ₃ , and then uses AdaptiveAvgPool to pool P ₃ to multiple pre-set scales. In order to aggregate the information in the feature maps of multiple scales obtained in the previous step, so as to perform global feature interaction of background and foreground objects at multiple scales in the subsequent step, a feature aggregator is designed. By further adjusting the image scale of each scale, and then splicing and aggregating their feature maps in the second dimension, the first multi-scale feature map P _fg can be obtained. The acquisition process of the second multi-scale feature map P _bg is similar.

In one embodiment, the joint attention module performs the following operations:

_Qfg = ConvC/8( _Pfg ), size is H×W×C/8;

_Kfg = ConvC/4( _Pfg ), size is H×W×C/4;

V _fg =ConvC/4(P _fg ), size is H×W×C;

Q _bg =ConvC/8(P _bg ), size is H×W×C/8;

K _bg =ConvC/4(P _bg ), size is H×W×C/4;

V _bg =ConvC/4(P _bg ), size is H×W×C;

The first dimension and the second dimension of Q _fg , K _fg , V _fg , Q _bg , K _bg , and V _bg are respectively integrated by the Reshpe function to obtain Q' _fg , K' _fg , V' _fg , Q' _bg , K' _bg , and V' _bg , which are expressed as follows:

The size of Q' _fg is HW × C/8;

The size of K' _fg is HW × C/4;

V' _fg has dimensions of HW × C;

The size of Q' _bg is HW × C/8;

The size of _K'bg is HW×C/4;

V' _bg size is HW × C;

The third dimension of _Q'fg and _Q'bg is concatenated and expressed as follows:

Q _cat = Concat(Q' _fg ,Q' _bg ), size is HW×C/4,

Perform attention calculation on Q _cat to obtain X _fg and X _bg , which are expressed as follows:

_Xfg = Softmax( _Qcat * _KfgT ⁾ _Vfg + _Pfg , size is HW×C;

X _bg = Softmax(Q _cat *K _bg ^T) V _bg +P _bg , size is HW×C;

Z = AdaptiveAvgPool{1×1}(Conv512(Concat( _Xfg , _Xbg ))), with size 1×1×C;

Among them, ConvX represents a convolution operation with X channels.

For the first multi-scale feature map _Pfg and the second multi-scale feature map _Pbg obtained, the global feature interaction information between the foreground image and the background image at multiple scales is extracted through the joint attention module, which greatly promotes the prediction of the reasonable position of the foreground object in the background image. As shown in Figure 3, when the joint attention module is trained, the parameters of each convolution layer are different, so the output results are also different. The convolution layers of the joint attention module are all 1*1 convolution kernels. The sliding step size of each convolution layer is stride=1, and padding=0. And adjust the dimensions of Q _fg , K _fg , V _fg , Q _bg , K _bg , V _bg through the Reshpe function, integrate the first dimension and the second dimension, and obtain Q' _fg , K' _fg , V' _fg , Q' _bg , K' _bg , V' _bg in turn. Then, Q' _fg and Q' _bg are spliced and combined at the channel level, that is, the third dimension. Then, Q _cat is respectively calculated with K _fg , K _bg and V _fg , V _bg of the foreground image and the background image to obtain X _fg and X _bg with a size of HW×C. Finally, the feature map obtained by the joint attention of both sides is combined, that is, X _fg and X _bg are spliced, and convolution and pooling operations are performed to obtain the global interaction feature Z with a size of 1×1×C. In Figure 3,

S23. Input the third basic feature map F _gt into the global feature extractor of the prior knowledge extractor to obtain a first extracted feature, and encode the first extracted feature into a priori vector z _p through an automatic encoder.

In one embodiment, the global feature extractor performs the following operations:

Z ₁ =ReLU(BatchNorm(Conv512(F _fg ))), with size H ₁ ×W ₁ ×512;

Z ₂ =ReLU(BatchNorm(Conv512(Z ₁ ))), with size H ₁ ×W ₁ ×512;

Z ₃ =ReLU(BatchNorm(ConvC(Z ₂ ))), with size H ₁ ×W ₁ ×C;

Z _{4 =} AdaptiveAvgPool{1×1}(P ₃ ), with size 1×1×C;

The autoencoder performs the following operations:

h = ReLU(FC1024(Z ₄ )), with size 1×1×1024;

mu = FC512(h), size is 1 × 1 × 512;

Logvar=FC512(h), size is 1×1×512;

Z _p = mu + e ^Logvar/2 , with dimensions of 1 × 1 × 512;

Among them, FCY is a fully connected layer that maps the number of channels of the corresponding input image to Y.

During training, the reasonable position information of the foreground object is extracted from the positive label map, and the extracted information is encoded into a priori vector Z _p and combined with the autoregressive process of the generator to provide prior knowledge guidance for the generator training. For the third basic feature map F _gt obtained, global feature extraction is first performed to obtain Z ₄ , and then a VAE-based autoencoder is used to encode Z ₄ into a priori vector Z _p , where FC represents a fully connected layer, such as FC1024 (Z ₄ ) means that the number of channels of Z ₄ is mapped from the original C to 1024, and the FC used in different places corresponds to different fully connected layers.

S24. The random variable _Zu and the prior vector _Zp are respectively fused with the global interaction feature Z through the Concat function, and the first concatenated vector _Zi is obtained to form an unsupervised path and the second concatenated vector _Zj is obtained to form a self-supervised path.

S25. Input the first splicing vector _Zi and the second splicing vector _Zj into the regression block respectively, and predict the first affine transformation parameter t _u = (t _r ^u , t _x ^u , _ty ^u ) and the second affine transformation parameter t _s = (t _r ^s , t _x ^s , _ty ^s ) accordingly, t _r ^u represents the scaling rate of the foreground object under the unsupervised path, t _x ^u represents the x-axis coordinate of the foreground position on the background image under the unsupervised path, t _y ^u represents the y-axis coordinate of the foreground position on the background image under the unsupervised path, t _r ^s represents the scaling rate of the foreground object under the self-supervised path, t _x ^s represents the x-axis coordinate of the foreground position on the background image under the self-supervised path, and _ty ^s represents the y-axis coordinate of the foreground position on the background image under the self-supervised path.

In one embodiment, the regression block performs the following operations:

t _u =FC3(FC1024(ReLU(FC1024(Z _i ))));

t _s =FC3(FC1024(ReLU(FC1024(Z _j ))));

Among them, FCY is a fully connected layer that maps the number of channels of the corresponding input image to Y, and ReLU represents the ReLU activation function.

Specifically, for the unsupervised path in the generator, a random variable _Zu (size is 1×1×512) is introduced into the global interaction feature Z before autoregression to make the synthesis result of the model diverse; for the self-supervised path in the generator, a prior vector _Zp (size is 1×1×512) is introduced into the global interaction feature Z before autoregression to guide the generator to make reasonable foreground object placement predictions, where:

The first concatenation vector _Zi is expressed as:

_Zi = Concat(Z, _Zu ), with size 1×1×(C+512);

The second splicing vector _Zj is expressed as:

Z _j =Concat(Z,Z _p ), with size 1×1×(C+512);

Then, the first splicing vector _Zi is input into the regression block to predict the first affine transformation parameter _tu ⁼ ( _tru , _txu ^, _tyu ), and the second splicing vector _Zj is input ^into the regression block to predict ^the second affine transformation parameter _ts = ( _{trs , txs} ^{, tys} ).

S26. Perform affine transformation on ^the input foreground image ^{and the foreground image mask according to the first affine transformation parameter tu = (tru, txu, tyu) and the second affine transformation parameter ts = (trs, txs, tys} _{) , respectively ,} ^to _obtain ^a _{first affine transformation} ^{result and} a second affine transformation result, the first affine transformation result including the foreground image after affine transformation under the unsupervised path. and foreground mask The second affine transformation result includes the foreground image after affine transformation under the self-supervised path and foreground mask

In one embodiment, the affine transformation uses a Spatial Transformer Network, and performs an affine transformation on the input foreground image and its foreground image mask according to the transformation parameters predicted by the generator, wherein the affine transformation principle is based on the Spatial Transformer Network.

Take the affine transformation of the input image according to _ts as an example, where _trs∈ [0,1], define the height h= _trsH and width ^w = _trsW ^{after scaling} , and assume that the coordinates of the upper left vertex of the foreground object placed on the background image are (x,y), so define _txs =x/Ww, ^tys =y/ _Hh . Then, perform affine transformation on the foreground image ^Ifg and _its foreground image mask _Mfg . Taking the affine transformation of the foreground image as an example, the formula is:

Among them, (x _fg ,y _fg ) represents the coordinate point on the foreground image, and (x′ _fg ,y′ _fg ) is the coordinate point of the foreground image after affine transformation. After the foreground image and the foreground image mask of the unsupervised path and the self-supervised path are affine transformed, the first affine transformation result and the second affine transformation result are obtained respectively. The first affine transformation result is the foreground image after affine transformation under the unsupervised path. and foreground mask The second affine transformation result is the foreground image after affine transformation under the self-supervised path and foreground mask

S27, synthesize the first affine transformation result and the second affine transformation result with the background image respectively, and obtain a first synthesized image And the second composite image It is expressed as follows:

S28, the first synthesis unit and the second synthesis unit The discriminator is input and trained using the second dataset.

This application is based on the adversarial generative network training model. All synthetic images generated by the generator must be input into the discriminator for discriminant training. The input of the discriminator is or The output is a value R∈{0,1}. If it is 1, it means that the discriminator thinks that the foreground object position of the composite image is reasonable, and 0 means it is unreasonable. In addition to being trained by the composite image generated by the generator, the discriminator is also trained by the second data set. The input of the discriminator is or The output is a value R∈{0,1}, where 1 indicates that the position of the foreground object in the discriminator's synthetic image is reasonable, and 0 indicates that it is unreasonable.

In one embodiment, the discriminator performs the following operations:

R = Sigmoid(Conv1(

LeakyReLU(Conv512(

LeakyReLU(Conv256(

LeakyReLU(Conv128(

LeakyReLU(Conv64(Input)))))))))

Among them, Sigmoid represents the Sigmoid activation function, ConvX represents the convolution operation with the number of channels X, LeakyReLU represents the LeakyReLU activation function, R represents the output feature of the discriminator, and Input represents the input feature of the discriminator.

S29. Calculate the joint loss to update the network parameters and obtain the trained adversarial generative network model.

In one embodiment, the combined loss is expressed as follows:

in,

In the formula, θ _G represents the learning parameters in the generator G, θ _D represents the learning parameters in the discriminator D, Generate loss function for adversarial on unsupervised path, is the adversarial generation loss function on the self-supervised path, _Lkld (G) is the KL divergence loss function, _Lrec (G) is the reconstruction loss function, _Lbce (D) is the cross entropy loss function, represents the mean of the distribution of the prior vector z _p , represents the variance of the distribution of the prior vector z _p , D _KL represents the calculated KL divergence, N(a ₁ ,b ₁ ) represents the distribution with mean a ₁ and variance b ₁ , if a ₁ ＝0, b ₁ ＝1, then it is a normal distribution.

Among them, when training the adversarial generative network model, the parameters of each network module are updated by calculating the joint loss. The adversarial generative loss function on the unsupervised path is used The generator can be trained to generate synthetic images on the unsupervised path that the discriminator considers reasonable. The adversarial generation loss function on the self-supervised path is used. The generator can be trained so that the synthetic images generated on the self-supervised path are considered reasonable by the discriminator. The KL divergence loss function L _kld (G) can make the distribution of the prior vector learned from the positive label synthetic image tend to Gaussian distribution. The reconstruction loss function L _rec (G) is used to make the transformation parameters predicted on the self-supervised path tend to the transformation parameters of the positive label. The cross entropy loss L _bce (D) is used to train the discriminator through the positive label synthetic image and the negative label synthetic image to improve the discriminant ability of the discriminator.

S3, input the first splicing unit [I _fg ,M _fg ] and the second splicing unit [I _bg ,M _bg ] to be synthesized into the trained adversarial generative network model, and input the first synthesis unit corresponding to the output of the discriminator under the unsupervised path As the image synthesis prediction result.

The final test results of the technical solution of this application on the OPA dataset are shown in Table 1:

Table 1

Technical Solution FID LPLIPS TERSE 46.88 0 PlaceNet 37.01 0.161 GracoNet 28.10 0.207 This application 23.21 0.270

The FID indicator refers to the difference between the synthetic image generated by the technical solution and the positive label synthetic image in the test set. The LPLIPS indicator refers to the difference between the 10 synthetic images generated by the technical solution for the same input, that is, the diversity of the synthetic images generated by the technical solution. Figure 4 shows a comparison of the synthetic image effects of the technical solution of the present application and other technical solutions. It can be seen that the synthetic effect of the technical solution of the present application is better than other existing technical solutions, including TERSE (reference S.Tripathi, S.Chandra, A.Agrawal, A.Tyagi, J.M.Rehg, and V.Chari, "Learning to generate synthetic data via compositing," in CVPR, pp.461–470, 2019.), PlaceNet (reference L.Zhang, T.Wen, J.Min, J.Wang, D.Han, and J.Shi, "Learning object placement by inpainting for compositional dataaugmentation," in ECCV, pp.566–581, 2020.), and GracoNet (reference S.Zhou, L.Liu, L.Niu, and L.Zhang, "Learning object placement via dual-path graph completion," in ECCV, pp.373–389, 2022.).

The technical features of the above-mentioned embodiments can be combined arbitrarily, and the specific steps are not limited in the text. Those skilled in the art can adjust the order according to actual needs. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-described embodiments only express the more specific and detailed embodiments described in this application, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this application, which all belong to the protection scope of this application. Therefore, the protection scope of the patent application shall be based on the attached claims.

Claims (10)

1. An image synthesis method based on a generative adversarial network, characterized in that: the image synthesis method based on a generative adversarial network comprises the following steps: S1. Acquire a first data set and a second data set, wherein each sample in the first data set includes a first splicing unit [I _fg , M _fg ], a second splicing unit [I _bg , M _bg ] and a third splicing unit [I _gt , M _gt ], and each sample in the second data set includes a fourth splicing unit Fifth splicing unit The transformation parameter corresponding to the fifth splicing unit is t _gt =(t _r ^gt ,t _x ^gt , _ty ^gt ), each splicing unit has the same size, wherein _Ifg is the foreground image, M _fg is the foreground image mask, _Ibg is the background image, M _bg is the background image mask, _Igt is the positive label image, _Mgt is the positive label image mask, represents the negative label synthetic graph, represents the negative label composite image mask, represents the positive label composite image, represents the positive label composite image mask, Indicates the scaling ratio of the foreground object corresponding to the fifth splicing unit, Indicates the x-axis coordinate of the foreground position corresponding to the fifth splicing unit on the background image, Indicates the y-axis coordinate of the foreground position corresponding to the fifth splicing unit on the background image; S2. Establish a generative adversarial network model and perform training. The generative adversarial network model includes a generator, a discriminator, and a priori knowledge extractor. The generator includes a preliminary feature extractor, a multi-scale feature aggregation module, a joint attention module, a Concat function, and a regression block. The multi-scale feature aggregation module includes two parallel first feature extraction units. The first feature extraction unit includes a multi-scale encoder and a feature aggregator connected in sequence. The priori knowledge extractor includes a global feature extractor and an autoencoder connected in sequence. The training process is as follows: S21, inputting the first concatenation unit [I _fg , M _fg ], the second concatenation unit [I _bg , M _bg ] and the third concatenation unit [I _gt , M _gt ] of each sample in the first data set into a preliminary feature extractor to obtain a first basic feature map F _fg , a second basic feature map F _bg and a third basic feature map F _gt ; S22, inputting the first basic feature map F _fg and the second basic feature map F _bg into the first feature extraction unit of the multi-scale feature aggregation module in a one-to-one correspondence, obtaining the first multi-scale feature map P _fg and the second multi-scale feature map P _bg correspondingly, and inputting the first multi-scale feature map P _fg and the second multi-scale feature map P _bg into the joint attention module to obtain the global interaction feature Z; S23, inputting the third basic feature graph F _gt into the global feature extractor of the prior knowledge extractor to obtain a first extracted feature, and encoding the first extracted feature into a priori vector z _p through an automatic encoder; S24, the random variable _Zu and the prior vector _Zp are respectively fused with the global interaction feature Z through the Concat function, and the first concatenation vector _Zi is obtained to form an unsupervised path and the second concatenation vector _Zj is obtained to form a self-supervised path; S25. Input the first splicing vector _Zi and the second splicing vector _Zj into the regression block respectively, and predict the first affine transformation parameter _tu = ( _tru ^, _txu , _tyu ) ^and _{the second affine transformation parameter ts = (trs, txs, tys )} ^{correspondingly , tru represents the} _scaling rate of the foreground object under the unsupervised path, _txu represents the x-axis coordinate of the foreground position on the background image under _the unsupervised path, ^tyu represents the y _- ^axis coordinate of the foreground position on the background image under the unsupervised path, _trs represents the scaling rate of the foreground object under the self-supervised path, _txs represents the x- ^axis coordinate of the ^foreground position on the background image under the self-supervised path, and _tys represents the y _- axis coordinate of the foreground position on the background image under the ^self -supervised path; S26. Perform an affine transformation on the input foreground image and the foreground image mask according to the first affine transformation parameter _tu = ( _tr ^u , _tx ^u , _ty ^u ) and the second affine transformation parameter _ts = ( _trs , _txs , _tys ), respectively, to obtain a ^{first affine} transformation result and a second affine transformation result correspondingly, wherein the ^first affine transformation result includes the foreground image after affine transformation under the unsupervised path. and foreground mask The second affine transformation result includes the foreground image after affine transformation under the self-supervised path and foreground mask S27, synthesize the first affine transformation result and the second affine transformation result with the background image respectively, and obtain a first synthesized image And the second composite image It is expressed as follows: S28, the first synthesis unit and the second synthesis unit Input the discriminator and train the discriminator using the second data set; S29, calculate the joint loss to update the network parameters and obtain the trained adversarial generation network model; S3, input the first splicing unit [I _fg ,M _fg ] and the second splicing unit [I _bg ,M _bg ] to be synthesized into the trained adversarial generative network model, and input the first synthesis unit corresponding to the output of the discriminator under the unsupervised path As the image synthesis prediction result. 2. The image synthesis method based on the generative adversarial network as described in claim 1 is characterized in that the preliminary feature extractor adopts the VGG16 network model. 3. The image synthesis method based on the adversarial generative network according to claim 1, characterized in that: the preliminary feature extractor performs the following operations: F ₁ = MaxPool(Conv64(Conv64(Input1))), with size H/2×W/2×64; F ₂ =MaxPool(Conv128(Conv128(F ₁ ))), with size H/4×W/4×128; F ₃ =MaxPool(Conv256(Conv256(Conv256(F ₂ )))), with size H/8×W/8×256; _Ffg = MaxPool(Conv512(Conv512(Conv512( _F3 )))), with size H/16×W/16×512; Among them, Maxpool represents the maximum pooling operation, ConvX represents the convolution operation with the number of channels as X, Input1 represents the first splicing unit [I _fg ,M _fg ] or the second splicing unit [I _bg ,M _bg ] or the third splicing unit [I _gt ,M _gt ], and the height, width and number of channels of each splicing unit are 4. 4. The image synthesis method based on a generative adversarial network as claimed in claim 1, characterized in that: The multi-scale encoder performs the following operations: P ₁ =ReLU(BatchNorm(Conv512(Input2))), with size H ₁ ×W ₁ ×512; P ₂ =ReLU(BatchNorm(Conv512(P ₁ ))), with size H ₁ ×W ₁ ×512; P ₃ =ReLU(BatchNorm(ConvC(P ₂ ))), with size H ₁ ×W ₁ ×C; _{PS1 =} AdaptiveAvgPool{ _S1 × _S1 }( _P3 ), with size _S1 × _S1 ×C; PS2 ₌ AdaptiveAvgPool{ _S2 × _S2 }( _P3 ), with size _S2 × _S2 ×C; PS3 ₌ AdaptiveAvgPool{S ₃ ×S ₃ }(P ₃ ), with size S ₃ ×S ₃ ×C; Wherein, ReLU represents the ReLU activation function, BatchNorm represents the normalization operation, ConvX represents the convolution operation with the number of channels being X, AdaptiveAvgPool{n×n} represents adaptively pooling the height×width of the corresponding input image to n×n, S ₁ , S ₂ , and S ₃ are the first preset size, the second preset size, and the third preset size respectively, Input2 represents the first basic feature map F _fg or the second basic feature map F _bg , the height of each basic feature map is H ₁ , the width is W ₁ , the number of channels is C, H/16＝H ₁ , W/16＝W ₁ , and the height of each splicing unit is H and the width is W; The feature aggregator performs the following operations: _{PS1 =} Reshape( _P3 ), with size 1×( _{S1* S1} )×C; PS2 ₌ Reshape( _P3 ), with size 1×( _{S2* S2} )×C; PS3 ₌ Reshape( _P3 ), with size 1×( _{S3* S3} )×C; _Pg = Concat(Concat( _PS1 , _PS2 ), _PS3 ), with size 1×(S1 _* S1 _{+ S2*} S2 _{+ S3* S3} )×C; Among them, Reshape represents a reshape function, Concat represents a Concat function, and _Pg represents the output feature of the first feature extraction unit corresponding to Input2, that is, the first multi-scale feature map _Pfg or the second multi-scale feature map _Pbg . 5. The image synthesis method based on the adversarial generative network as claimed in claim 4, characterized in that: The global feature extractor performs the following operations: Z ₁ =ReLU(BatchNorm(Conv512(F _fg ))), with size H ₁ ×W ₁ ×512; Z ₂ =ReLU(BatchNorm(Conv512(Z ₁ ))), with size H ₁ ×W ₁ ×512; Z ₃ =ReLU(BatchNorm(ConvC(Z ₂ ))), with size H ₁ ×W ₁ ×C; Z _{4 =} AdaptiveAvgPool{1×1}(P ₃ ), with size 1×1×C; The autoencoder performs the following operations: h = ReLU(FC1024(Z ₄ )), with size 1×1×1024; mu = FC512(h), size is 1 × 1 × 512; Logvar=FC512(h), size is 1×1×512; Z _p = mu + e ^Logvar/2 , with dimensions of 1 × 1 × 512; Among them, FCY is a fully connected layer that maps the number of channels of the corresponding input image to Y. 6. The image synthesis method based on the adversarial generative network according to claim 4, characterized in that: the joint attention module performs the following operations: _Qfg = ConvC/8( _Pfg ), size is H×W×C/8; _Kfg = ConvC/4( _Pfg ), size is H×W×C/4; V _fg =ConvC/4(P _fg ), size is H×W×C; Q _bg =ConvC/8(P _bg ), size is H×W×C/8; K _bg =ConvC/4(P _bg ), size is H×W×C/4; V _bg =ConvC/4(P _bg ), size is H×W×C; The first dimension and the second dimension of Q _fg , K _fg , V _fg , Q _bg , K _bg , and V _bg are respectively integrated by the Reshpe function to obtain Q' _fg , K' _fg , V' _fg , Q' _bg , K' _bg , and V' _bg , which are expressed as follows: The size of Q' _fg is HW × C/8; The size of K' _fg is HW × C/4; V' _fg has dimensions of HW × C; The size of Q' _bg is HW × C/8; The size of _K'bg is HW×C/4; V' _bg size is HW × C; The third dimension of _Q'fg and _Q'bg is concatenated and expressed as follows: Q _cat = Concat(Q' _fg ,Q' _bg ), size is HW×C/4, Perform attention calculation on Q _cat to obtain X _fg and X _bg , which are expressed as follows: _Xfg = Softmax( _Qcat * _KfgT ⁾ _Vfg + _Pfg , size is HW×C; X _bg = Softmax(Q _cat *K _bg ^T) V _bg +P _bg , size is HW×C; Z = AdaptiveAvgPool{1×1}(Conv512(Concat( _Xfg , _Xbg ))), with size 1×1×C; Among them, ConvX represents a convolution operation with X channels. 7. The image synthesis method based on the adversarial generative network according to claim 1, characterized in that: the regression block performs the following operations: t _u =FC3(FC1024(ReLU(FC1024(Z _i )))); t _s =FC3(FC1024(ReLU(FC1024(Z _j )))); Among them, FCY is a fully connected layer that maps the number of channels of the corresponding input image to Y, and ReLU represents the ReLU activation function. 8. The image synthesis method based on the generative adversarial network as described in claim 1 is characterized in that the affine transformation adopts the Spatial Transformer Network. 9. The image synthesis method based on the adversarial generative network according to claim 1, characterized in that: the discriminator performs the following operations: R = Sigmoid(Conv1( LeakyReLU(Conv512( LeakyReLU(Conv256( LeakyReLU(Conv128( LeakyReLU(Conv64(Input))))))))) Among them, Sigmoid represents the Sigmoid activation function, ConvX represents the convolution operation with the number of channels X, LeakyReLU represents the LeakyReLU activation function, R represents the output feature of the discriminator, and Input represents the input feature of the discriminator. 10. The image synthesis method based on the adversarial generative network according to claim 1, characterized in that: the joint loss is expressed as follows: in, In the formula, θ _G represents the learning parameters in the generator G, θ _D represents the learning parameters in the discriminator D, Generate loss function for adversarial on unsupervised path, is the adversarial generation loss function on the self-supervised path, _Lkld (G) is the KL divergence loss function, _Lrec (G) is the reconstruction loss function, _Lbce (D) is the cross entropy loss function, represents the mean of the distribution of the prior vector z _P , represents the variance of the distribution of the prior vector z _P , D _KL represents the calculated KL divergence, N(a ₁ ,b ₁ ) represents the distribution with mean a ₁ and variance b ₁ , if a ₁ ＝0, b ₁ ＝1, it is a normal distribution.