CN114399427B - Word effect migration method based on loop generation countermeasure network - Google Patents
Word effect migration method based on loop generation countermeasure network Download PDFInfo
- Publication number
- CN114399427B CN114399427B CN202210018956.9A CN202210018956A CN114399427B CN 114399427 B CN114399427 B CN 114399427B CN 202210018956 A CN202210018956 A CN 202210018956A CN 114399427 B CN114399427 B CN 114399427B
- Authority
- CN
- China
- Prior art keywords
- image
- word
- word effect
- effect
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000000694 effects Effects 0.000 title claims abstract description 306
- 238000013508 migration Methods 0.000 title claims abstract description 144
- 230000005012 migration Effects 0.000 title claims abstract description 144
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000012549 training Methods 0.000 claims abstract description 81
- 238000012545 processing Methods 0.000 claims abstract description 5
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 230000004913 activation Effects 0.000 claims description 31
- 238000004364 calculation method Methods 0.000 claims description 31
- 238000010606 normalization Methods 0.000 claims description 27
- 238000005457 optimization Methods 0.000 claims description 27
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 21
- 238000013461 design Methods 0.000 claims description 14
- 230000004927 fusion Effects 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 10
- 230000003213 activating effect Effects 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012067 mathematical method Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention relates to a word effect migration method based on a loop generation countermeasure network. Comprising the following steps: s1: processing the data set to obtain a font image and word effect image pair, and preprocessing each image to obtain a training data set; s2: designing a word effect style migration network based on a loop generation countermeasure network, wherein the network consists of a word effect removal sub-network and a word effect migration sub-network; s3: designing a loss function for training the network designed in the step S2; s4: training a word effect style migration network based on a loop generation countermeasure network using the training data set; s5: inputting the new target word effect image and the word effect image to be migrated into a trained word effect migration network, and outputting the image with the target word effect after the word effect migration. The invention can realize word effect style migration and generate high-quality and strong-structural word effect migration images.
Description
Technical Field
The invention relates to the field of image processing and computer vision, in particular to a word effect migration method based on a cyclic generation countermeasure network.
Background
In recent years, style migration technology has been unprecedented, especially, new network models are proposed and new mathematical methods are introduced, and more style migration methods emerge. For an artistic image, everyone has a unique insight into the artistic style. Therefore, it is difficult to uniformly describe and express an artistic style possessed by one artistic image, and after all, the concept and definition of the artistic style are ambiguous. This is a more difficult problem to describe in order to convert the style of a piece of content image into the artistic style of a given style image. The development of computer vision and the upgrading of hardware have made possible the implementation of this problem, and thus many researchers have begun to invest in research on style migration.
Font special effects are a combination of visual elements, such as outlines, colors and textures of words, that can significantly improve the artistry of words, often used in the design industry, but due to their special complexity they are usually created manually by humans. Word effect style migration is a subtask of style migration, which is a technique that applies a font-specific effect to different glyphs. Since the fonts in the character effect style migration are highly structured, most of the style migration methods are not suitable for the character effect style migration, but the character effect style migration can design a reference standard image. In recent years, some new attempts have been made to style migration, however, the lack of data limits the performance of style migration models. In 2020, yang et al proposed a new word effect data set, i.e., TE141K data set, and a baseline method for generating a challenge network (TET-GAN) by word effect migration, which greatly motivated the development of word effect style migration tasks.
The word effect style migration has important practical significance, and for Chinese characters with huge quantity and base numbers, people can manually realize word effect images of all common words, and huge labor and time cost can be consumed. The word effect style migration algorithm just solves the problem, and a developer only needs to design a few representative word effect samples, so that the words for applying the word effect to other fonts can be automatically generated through the word effect style migration algorithm, thereby avoiding repetitive labor and saving a great amount of time.
The invention provides a word effect style migration method based on a circularly generated countermeasure network. According to the method, the network is trained to complete two tasks of word effect migration and word effect removal, so that the network can learn to decouple and recombine the character pattern features and the word effect features of the characters, and high-quality word effect style migration is completed. The method trains the network using a loop-generated countermeasure and introduces a attentive mechanism to strengthen the supervision of the glyph edge structure by the model. The word effect style migration method based on the circularly generated countermeasure network can effectively complete the word effect style migration task, and the generated word effect image is improved in visual effect and performance index.
Disclosure of Invention
The invention aims to provide a word effect migration method based on a loop generation countermeasure network, which can realize high-quality word effect style migration.
In order to achieve the above purpose, the technical scheme of the invention is as follows: a method for word effect migration based on a loop generation countermeasure network, comprising the steps of:
S1, processing a data set to obtain a font image and a word effect image pair, and preprocessing each image to obtain a training data set;
Step S2, designing a word effect style migration network based on a loop generation countermeasure network, wherein the network consists of a word effect removal subnet and a word effect migration subnet;
Step S3, designing a loss function for training the network designed in the step S2;
s4, training a word effect style migration network based on a loop generation countermeasure network by using a training data set;
and S5, inputting the new target word effect image and the word effect image to be migrated into a trained word effect style migration network, and outputting the image with the target word effect after the word effect migration.
In an embodiment of the present invention, the step S1 is specifically implemented as follows:
S11, dividing each image in the original input data set into an independent font image and a word effect image with the same size by using the font image and the word effect image with the same size; combining the font image and the word effect image from the same original image into a font image and word effect image pair;
step S12, the two images of the font image and the word effect image pair are subjected to the same random overturn and are cut into a font image and a word effect image with H multiplied by W at the center, and the relationship between the font image and the word effect image pair is maintained;
step S13, normalizing all font images and word effect images with H multiplied by W, giving an image I (I, j), and calculating a normalized image The formula of (2) is as follows:
where (i, j) denotes the position of the pixel.
In an embodiment of the present invention, the step S2 is specifically implemented as follows:
s21, designing a generation network and a discrimination network of a word effect removal subnet, and respectively generating probabilities of a word effect removal image and a discrimination image being a real image by using the designed network;
And S22, designing a generation network and a discrimination network of the character effect migration sub-network, and respectively generating probabilities of the character effect migration image and the discrimination image being real images by using the designed network.
In an embodiment of the present invention, the step S21 is specifically implemented as follows:
Step S211, designing a generating network of the word effect removing sub-network, wherein the input of the generating network is a pair of a font image and a word effect image, the font image is represented by A s, and the word effect image is represented by A e; the generating network is composed of encoders E x and E y, a shared layer S 1 for extracting image features, and a generator G 1 for reconstructing features into word effect removed images
Step S212, design encoders E y and E x, whose inputs are a e and a s, respectively; the encoder E y consists of five convolution blocks, each of which consists of an activation layer, a convolution layer, a normalization layer and an attention module in sequence; the activation layer adopts LeakyReLU activation functions, the convolution layer uses convolution with a convolution kernel of 4×4, a step size of 2 and a filling of 1, the normalization layer uses an example for normalization, and the attention module uses a convolution attention module (CBAM); image characteristics of size C x H x W obtained by inputting image into encoder E y The encoder E x and the encoder E y have the same structure, and c×h×w image characteristics obtained by inputting an image to the encoder E x
Step S213, designing a shared layer S 1,S1 to be a network with a Unet structure; features of size C.times.H.times.W obtained in step S212AndCan be used as the input of the sharing layer S 1, the input is that f 0,f0 sequentially passes through three convolution blocks which are the same as those in S212 to respectively obtain the characteristic f 1 with the size of C multiplied by H/2 multiplied by W/2, the characteristic f 2 with the size of C multiplied by H/4 multiplied by W/4 and the characteristic f 3;f3 with the size of C multiplied by H/8 multiplied by W/8, and the like, and the method comprises the steps of Deconvolution blocks composed of a deconvolution layer, a normalization layer and an attention module in sequence obtain a characteristic f 4 of C multiplied by H/4 multiplied by W/4; The activation layer in the deconvolution block adopts a ReLU activation function, the deconvolution layer uses deconvolution with a convolution kernel of 4 multiplied by 4, a step length of 2 and a filling of 1, the normalization layer uses an example for normalization, and the attention module uses a convolution attention module (CBAM); the feature f 2 and the feature f 4 are spliced in the channel dimension to obtain a feature with the size of 2C x H/4 x W/4, and the feature is subjected to deconvolution to obtain a feature f 5 with the size of C x H/2 x W/2; The feature f 1 and the feature f 5 are spliced in the channel dimension to obtain a feature with the size of 2C multiplied by H/2 multiplied by W/2, and the feature is subjected to deconvolution to obtain a feature f 6 with the size of C multiplied by H multiplied by W; the feature f 0 and the feature f 6 are spliced in the channel dimension to obtain an output feature f s with the size of 2C multiplied by H multiplied by W; When the input features areThe output characteristic f s is then noted asWhen the input features areThe output characteristic f s is then noted as
Step S214, a design generator G 1; the input to generator G 1 is the characteristics generated in step S213Or (b)The generator G 1 consists of five deconvolution blocks identical to those in step S213, a deconvolution layer with a convolution kernel of 3 x 3, a step size of 1, a padding of 1, and a tanh activation function; each deconvolution block reduces the channel number of the feature by half and improves the width and the height of the feature to two times; when the input features areFeatures of size 2C H WThe feature with the size of C/16 multiplied by 32H multiplied by 32W is obtained through five deconvolution blocks, and finally the feature is subjected to deconvolution layer and tanh activation function to obtain A3-channel word effect removal image with the same size as the input word effect image A e When the input features areFeatures of size 2C H WThe feature with the size of C/16 multiplied by 32H multiplied by 32W is obtained through five deconvolution blocks, and finally the feature is subjected to deconvolution layer and tanh activation function to obtain A3-channel font reconstruction image with the same size as the input font image A s
Step S215, designing a discrimination network of a word effect removal subnet, wherein the discrimination network consists of a discriminator D 1, and the discriminator D 1 adopts a structure of a Markov discriminator in PatchGAN; the discriminator D 1 consists of 5 convolution blocks which are sequentially composed of a convolution layer, an instance normalization layer and an activation layer, wherein the convolution layer uses convolution with a convolution kernel of 4 multiplied by 4, a step length of 2 and a filling of 1, the normalization layer uses instance normalization, and the activation layer uses LeakyReLU activation functions; the inputs to the arbiter D 1 are the inputs to the arbiter A e andThe result obtained by splicing the false sample or the result obtained by splicing the A e and the A s in the channel dimension, namely the real sample, is output as an N multiplied by 1 tensor, wherein N=32, and the probability and the calculation loss of the image as the real image are judged by the average value of the tensor.
In an embodiment of the present invention, the step S22 is specifically implemented as follows:
Step S221, designing a generating network of a word effect migration subnet, wherein the inputs of the generating network are a word pattern image A s to be migrated and a target word effect image B e; the generating network is composed of an encoder E x、Ez, a sharing layer S 2 and a generator G 2, wherein the encoder and the sharing layer are used for extracting image features, and the generator reconstructs the features into word-effect migration images
Step S222, designing the encoder E z,Ez, and the encoders E x and E y in step S212 to have the same structure, and inputting the target byte image B e into the encoder to obtain image characteristics with the size of C×H×W
Step S223, designing a shared layer S 2,S2 to be a network with a Unet structure; the structure of the shared layer S 2 is the same as that of the shared layer S 1 in step S213, and the input characteristic is a characteristic of the size of c×h×w output from the encoder in step S222Features of size 2C x H x W
Step S224, designing a generator G 2, the structure of the generator G 2 is the same as the structure of the generator G 1 in step S214, but the input is the output feature of the shared layer S 1 And sharing layer S 2 output featuresStitching features in the channel dimension, feature size 4C x H x W, output 3-channel and same size word-effect migrated image as input image B e
Step S225, designing a discrimination network of the word effect migration sub-network, wherein the discrimination network consists of a discriminator D 2, a discriminator D 2 adopts the structure of a Markov discriminator in PatchGAN, and the structures of a discriminator D 2 and a discriminator D 1 in S215 are the same; the input to arbiter D 2 is the word effect migration image generated by A s、Be and generator G 2 The output of the concatenation in the channel dimension, i.e. the false sample, or the concatenation in the channel dimension, i.e. the true sample, of the a s、Be and the standard effect migration image a e is an nxnx1 tensor, where n=32, and the probability that the image is the true image and the calculation loss are determined by the mean of this tensor.
In an embodiment of the present invention, the step S3 is specifically implemented as follows:
step S31, designing a total optimization target of the whole network, wherein the optimization target is as follows:
Wherein, Representing the loss function of the training image encoding and reconstruction,Representing the training handwriting effect removal subnet loss function,The loss function of the training handwriting migration subnet is represented, E, G, D respectively represents an encoder (E x、Ey、Ez), a generator (G 1、G2) and a discriminator (D 1、D2);
step S32, designing a loss function of training image coding and reconstruction, The calculation formula of (2) is as follows:
Where x represents a glyph image, G 1(S1(Ex (x))) represents an image encoded and reconstructed image, λ rec represents a weight, E x represents an encoder E x,S1 represents a shared layer S 1,G1 represents a generator G 1,|| ||1 that is an absolute value taking operation;
step S33, designing a loss function for removing the subnet by training handwriting effect, The calculation formula of (2) is as follows:
Wherein, Indicating a loss of the characteristic(s),Indicating that the word effect removes the pixel loss,Representing the word effect removal condition against the loss of resistance, λ dfeat,λdpix and λ dadv are the weights of each of the three parts;
The feature loss directs the encoder to remove the word effect from the word effect image, retaining only the glyph information, and the formula for λ dfeat is as follows:
Wherein x represents a font image, y represents a word effect image, E x and E y represent two encoders E x and E y,S1 represent that the shared layer S 1,|| ||1 is an absolute value taking operation;
the word effect removal pixel loss is a word effect removal image that directs the generator to generate a more similar content glyph image, The calculation formula of (2) is as follows:
Where x represents a glyph image, G 1 represents a generator G 1, y represents a word effect image, G 1(S1(Ey (y))) represents an image from which a word effect is removed, and || 1 is an absolute value taking operation;
the word effect removal condition is used to improve the quality of the generated result, enable the generator to learn to "fool" the arbiter, also enable the arbiter to distinguish between the real image and the generated image, employ the condition in WGAN-GP against the loss, The calculation formula of (2) is as follows:
where lambda gp is the weight of the gradient penalty, Is the result of the fusion of the font image and the generated word effect removing image,Representation ofD 1 represents the gradient of the arbiter D 1,The calculation formula of (2) is as follows:
wherein epsilon represents a fusion weight parameter;
step S34, designing a loss function of the training handwriting migration subnetwork, The calculation formula of (2) is as follows:
Wherein, Representing a word effect migrate pixel loss,Representing the word effect migration condition against the loss of resistance, lambda spix、λsadv is the weight of each of the two parts;
the effect migrate pixel loss is to instruct the generator to generate an image that is closer to the target effect image, The calculation formula of (2) is as follows:
Where x represents a glyph image, y represents a reference image corresponding to x and having a target word effect, z represents a target word effect image, E x represents an encoder E x,Ez represents a style encoder E z,S1 and S 2 represent two shared layers S 1 and S 2,G2 represent a word effect image generator G 2,G2(concat(S1(Ex(x)),S2(Ez (z))) respectively), and a word effect migration generated image;
word effect migration condition combat penalty Also, using the design in WGAN-GP, the calculation formula is as follows:
where lambda gp is the weight of the gradient penalty, Is the result of the fusion of the standard word effect image and the generated word effect image,Representation ofD 2 represents the gradient of the arbiter D 2,The calculation formula of (2) is as follows:
Where ε represents the fusion weight parameter.
In an embodiment of the present invention, the step S4 is specifically implemented as follows:
Step S41, randomly selecting two pairs of images with the same word effect from a data set, wherein the font image and the word effect image of the first image pair are respectively marked as A s and A e, and the font image and the word effect image of the second image pair are respectively marked as B s and B e;
s42, training image coding and reconstruction; input glyph image A s, through encoder E x, shared layer S 1, and generator G 1, generates an encoded and reconstructed image Calculating the loss in the step S32, calculating the gradient of each parameter in E x、S1 and G 1 by using a back propagation method, and updating the parameters by using an Adam optimization method;
Step S43, training word effect removing the discrimination network of the sub-network; input word effect image A e, obtaining word effect removed image through encoder E y, shared layer S 1 and generator G 1 Calculating word effect removal conditions against resistance lossCalculating the gradient of each parameter in the discriminator D 1 by using a back propagation method, and updating the parameters by using an Adam optimization method;
step S44, training word effect removing the generation network of the sub-network; input word effect image A e, obtain feature S 1(Ey(Ae) through encoder E y, shared layer S 1), and obtain word effect removed image through generator G 1 Inputting a font image a s, obtaining a feature S 1(Ex(As by the encoder E x and the sharing layer S 1); calculating a total loss function of a word-effect removal subnetCalculating gradients of the parameters in E y、S1 and G 1 by using a back propagation method, and updating the parameters by using an Adam optimization method;
Step S45, training the handwriting migration subnetwork in two stages, wherein each stage is used for training a judging network and a generating network of the handwriting migration subnetwork; judging a first stage of a network of the word effect migration subnetwork; input font image A s, obtained by encoder E x, shared layer S 1 Input word effect image B e, obtain characteristics through encoder E z, sharing layer S 2 Will beAndConnecting the inputs as generator G 2 in the channel dimension, generating a word-effect migration imageCalculating word effect migration conditions against resistance lossCalculating the gradient of each parameter in the discriminator D 2 by using a back propagation method, and updating the parameters by using an Adam optimization method; training a first stage of generating a handwriting migration subnet; the input image is the same as the training discrimination network, and the character effect migration image is generatedComputing total loss function of word effect migration subnetworkCalculating gradients of the parameters in E x、Ez、S1、S2 and G 2 by using a back propagation method, and updating the parameters by using an Adam optimization method;
Step S46, training a second stage of the network discrimination of the word effect migration subnetwork; input font image B s, obtained by encoder E x, shared layer S 1 Inputting the word effect migration image generated in step S45Obtaining features by encoder E z, shared layer S 2 Will beAndConnecting the inputs as generator G 2 in the channel dimension, generating a word-effect migration imageCalculating word effect migration conditions against resistance lossCalculating the gradient of each parameter in the discriminator D 2 by using a back propagation method, and updating the parameters by using an Adam optimization method; training a second stage of generating a network of the handwriting migration subnetwork; the input image is identical to the training discriminator, and a word effect migration image is generatedComputing total loss function of word effect migration subnetworkCalculating gradients of the parameters in E x、Ez、S1、S2 and G 2 by using a back propagation method, and updating the parameters by using an Adam optimization method;
In step S47, the above steps are a complete iteration for two pairs of images, two hundred and fifty iterations are needed in the whole training process, and in each iteration process, a plurality of pairs of images with two pairs of the same word effect are randomly sampled and used as a batch for training, the previous hundred iterations do not perform the training of the second stage in step S46, and the remaining hundred and fifty iterations perform the training of all steps.
In an embodiment of the present invention, the step S5 is specifically implemented as follows:
Inputting the font image to be migrated, and obtaining font characteristics through the encoder E x and the sharing layer S 1 Inputting target word effect image, obtaining word effect characteristic by encoder E z and sharing layer S 2 Will beAndThe input as generator G 2 is connected in the channel dimension, generating a word effect migration image having the word effect of the word effect image and the glyph of the glyph image.
Compared with the prior art, the invention has the following beneficial effects: the invention aims to solve the problem that the traditional style migration model can not separate the fonts and the word effects well and generate high-quality word effect migration images. The invention provides a word effect style migration method based on a cyclic generation countermeasure network, which simultaneously trains the network to complete two tasks of word effect migration and word effect removal, so that the network can learn the character pattern characteristics and the word effect characteristics of the decoupled and recombined characters. And training the network in a manner of circularly generating the countermeasure, introducing a attention mechanism to strengthen the supervision of the model on the edge structure of the character pattern, and ensuring that the network can generate a character effect migration image with high quality and strong structure.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the invention.
FIG. 2 is a word effect migration model according to an embodiment of the present invention.
Fig. 3 is a training encoding and reconstruction process according to an embodiment of the present invention.
Fig. 4 is a training handwriting removal process according to an embodiment of the present invention.
Fig. 5 is a training handwriting migration subnet process according to an embodiment of the invention.
FIG. 6 is a cycle generation countermeasure training process in accordance with an embodiment of the present invention.
Detailed Description
The technical scheme of the invention is specifically described below with reference to the accompanying drawings.
The invention provides a word effect style migration method based on a loop generation countermeasure network, which is shown in figure 1 and comprises the following steps:
S1, processing a data set to obtain a font image and a word effect image pair, and preprocessing each image to obtain a training data set;
Step S2, designing a word effect style migration network based on a loop generation countermeasure network, wherein the network consists of a word effect removal subnet and a word effect migration subnet;
Step S3, designing a loss function for training the network designed in the step S2;
s4, training a word effect style migration network based on a loop generation countermeasure network by using a training data set;
and S5, inputting the new target word effect image and the word effect image to be migrated into a trained word effect migration network, and outputting the image with the target word effect after the word effect migration.
Further, step S1 includes the steps of:
S11, dividing each image in the original input data set into an independent font image and a word effect image with the same size by using the font image and the word effect image with the same size; and combining the font image and the word effect image from the same original image into a font image and word effect image pair.
And step S12, the two images of the font image and the word effect image pair are subjected to the same random overturn and are cut into the font image and the word effect image with the size of H multiplied by W at the center, and the relationship between the font image and the word effect image pair is maintained.
And step S13, normalizing all the font images and the character effect images with the size of H multiplied by W. Given image I (I, j), a normalized image is calculatedThe formula of (2) is as follows:
where (i, j) denotes the position of the pixel.
Further, step S2 includes the steps of:
And S21, designing a generation network and a discrimination network of the word effect removal sub-network, and respectively generating probabilities of the word effect removal image and the discrimination image being real images by using the designed network.
And S22, designing a generation network and a discrimination network of the character effect migration sub-network, and respectively generating probabilities of the character effect migration image and the discrimination image being real images by using the designed network.
Further, step S21 includes the steps of:
Step S211, designing a generating network of the word effect removing sub-network, wherein the input of the network is a pair of a font image and a word effect image, the font image is denoted by a s, and the word effect image is denoted by a e. The generating network is composed of encoders E x and E y, a shared layer S 1 for extracting image features, and a generator G 1 for reconstructing features into word effect removed images
Step S212, design encoders E y and E x, whose inputs are a e and a s, respectively. The encoder E y consists of five convolution blocks, each of which consists of an activation layer, a convolution layer, a normalization layer and an attention module in sequence. The activation layer uses LeakyReLU activation functions, the convolution layer uses convolution with a convolution kernel of 4 x 4, a step size of 2, and a fill of 1, the normalization layer uses instance normalization, and the attention module uses convolution attention module (CBAM). Image characteristics of size C x H x W obtained by inputting image into encoder E y The encoder E x and the encoder E y have the same structure, and c×h×w image characteristics obtained by inputting an image to the encoder E x
Step S213, design sharing layer S 1,S1 is a network of Unet structure. Features of size C.times.H.times.W obtained in step S212AndCan be used as the input of the sharing layer S 1, the input is that f 0,f0 sequentially passes through three convolution blocks which are the same as those in S212 to respectively obtain the characteristic f 1 with the size of C multiplied by H/2 multiplied by W/2, the characteristic f 2 with the size of C multiplied by H/4 multiplied by W/4 and the characteristic f 3.f3 with the size of C multiplied by H/8 multiplied by W/8, and the like, and the method comprises the steps of Deconvolution layer, normalization layer and attention module form deconvolution blocks in sequence to obtain characteristic f 4 of size c×h/4×w/4. The active layer in the deconvolution block adopts a ReLU active function, the deconvolution layer uses deconvolution with a convolution kernel of 4×4, a step size of 2 and a filling of 1, the normalization layer uses an example for normalization, and the attention module uses a convolution attention module (CBAM). Feature f 2 and feature f 4 are concatenated in the channel dimension to yield a feature of size 2c×h/4×w/4, which is deconvoluted to yield a feature of size c×h/2×w/2 f 5 by the deconvolution block described above. Feature f 1 and feature f 5 are concatenated in the channel dimension to yield a feature of size 2c×h/2×w/2, which is deconvoluted to yield a feature f 6 of size c×h×w by the deconvolution block described above. Feature f 0 and feature f 6 are spliced in the channel dimension to yield an output feature f s of the size 2c×h×w. When the input features areThe output characteristic f s is then noted asWhen the input features areThe output characteristic f s is then noted as
Step S214, design generator G 1. The input to generator G 1 is the characteristics generated in step S213Or (b)The generator G 1 consists of five deconvolution blocks identical to those in step S213, a deconvolution kernel of 3 x3, step size 1, padding 1, and tanh activation function. Each deconvolution block reduces the number of channels of the feature by half, doubling the width and height of the feature. When the input features areFeatures of size 2C H WThe feature with the size of C/16 multiplied by 32H multiplied by 32W is obtained through five deconvolution blocks, and finally the feature is subjected to deconvolution layer and tanh activation function to obtain A3-channel word effect removal image with the same size as the input word effect image A e When the input features areFeatures of size 2C H WThe feature with the size of C/16 multiplied by 32H multiplied by 32W is obtained through five deconvolution blocks, and finally the feature is subjected to deconvolution layer and tanh activation function to obtain A3-channel font reconstruction image with the same size as the input font image A s
In step S215, a discrimination network for removing the sub-network is designed, and the discrimination network is composed of a discriminator D 1, and a structure of a markov discriminator in PatchGAN is adopted as a discriminator D 1. The discriminator D 1 consists of 5 convolutions consisting of a convolutions layer, an instance normalization layer, an activation layer in sequence, the convolutions layer uses convolutions with a convolution kernel of 4 x 4, a step size of 2, and a fill of 1, the normalization layer uses instance normalization, and the activation layer uses LeakyReLU activation functions. The inputs to the arbiter D 1 are the inputs to the arbiter A e andThe result of stitching in the channel dimension (i.e. false sample) or the result of stitching a e and a s in the channel dimension (i.e. true sample) is output as a tensor of n×n×1, where n=32, and the probability of the image being a true image and the computational loss are determined by the mean of this tensor.
Further, step S22 includes the steps of:
Step S221, designing a generation network of a word effect migration subnet, where the inputs of the network are a word pattern image a s to be migrated and a target word effect image B e. The generating network is composed of an encoder E x、Ez, a sharing layer S 2 and a generator G 2, wherein the encoder and the sharing layer are used for extracting image features, and the generator reconstructs the features into word-effect migration images
Step S222, designing the encoder E z.Ez, and the encoders E x and E y in step S212 to have the same structure, and inputting the target byte image B e into the encoder to obtain image characteristics with the size of C×H×W
Step S223, designing the shared layer S 2,S2 is a network of Unet structure. The structure of the shared layer S 2 is the same as that of the shared layer S 1 in step S213, and the input characteristic is a characteristic of the size of c×h×w output from the encoder in step S222Features of size 2C x H x W
Step S224, design generator G 2. The structure of generator G 2 is the same as that of generator G 1 in step S214, but the input is the output feature of shared layer S 1 And sharing layer S 2 output featuresStitching features in the channel dimension, feature size 4C x H x W, output 3-channel and same size word-effect migrated image as input image B e
In step S225, a discrimination network of the word effect migration subnetwork is designed, the discrimination network is composed of a discriminator D 2, a discriminator D 2 adopts the structure of a markov discriminator in PatchGAN, and the structures of a discriminator D 2 and a discriminator D 1 in S215 are the same. The input to arbiter D 2 is the word effect migration image generated by A s、Be and generator G 2 The output of the concatenation in the channel dimension (i.e. false samples) or the concatenation of a s、Be and standard effect migration image a e in the channel dimension (i.e. true samples) is an N x1 tensor, where n=32, the mean of this tensor is used to determine the probability that the image is a true image and the computational loss.
Further, step S3 includes the steps of:
step S31, designing a total optimization target of the whole network. The optimization targets are as follows:
Wherein, Representing the loss function of the training image encoding and reconstruction,Representing the training handwriting effect removal subnet loss function,The loss function of the training effect migration subnet is represented by E, G, D, which respectively represent an encoder (E x、Ey、Ez), a generator (G 1、G2) and a discriminator (D 1、D2).
Step S32, designing a loss function of training image coding and reconstruction.The calculation formula of (2) is as follows:
Where x represents a glyph image, G 1(S1(Ex (x))) represents an image encoded and reconstructed image, λ rec represents a weight, E x represents an encoder E x,S1 represents a shared layer S 1,G1, and generator G 1,|| ||1 is operated with absolute value.
And step S33, designing a loss function for removing the subnet by training handwriting effect.The calculation formula of (2) is as follows:
Wherein, Indicating a loss of the characteristic(s),Indicating that the word effect removes the pixel loss,Meaning that the word effect removal condition is against the loss of resistance, λ dfeat,λdpix and λ dadv are the weights of each of the three parts.
The feature loss directs the encoder to remove the word effect from the word effect image, retaining only the glyph information, and the formula for λ dfeat is as follows:
Where x represents a glyph image, y represents a word effect image, E x and E y represent two encoders E x and E y,S1 represent that the shared layer S 1,|| ||1 is an absolute value taking operation.
The word effect removal pixel loss is a word effect removal image that directs the generator to generate a more similar content glyph image,The calculation formula of (2) is as follows:
Where G 1 denotes a generator G 1, y denotes a word effect image, G 1(S1(Ey (y))) represents a character effect removed image, the absolute 1 is an absolute value operation.
The word effect removal condition is used to improve the quality of the generated result, enable the generator to learn to "fool" the arbiter, also enable the arbiter to distinguish between the real image and the generated image, employ the condition in WGAN-GP against the loss,The calculation formula of (2) is as follows:
where lambda gp is the weight of the gradient penalty, Is the result of the fusion of the font image and the generated word effect removing image,Representation ofD 1 represents the arbiter D 1.The calculation formula of (2) is as follows:
Where ε represents the fusion weight parameter.
And step S34, designing a loss function of the training effect migration subnet.The calculation formula of (2) is as follows:
Wherein, Representing a word effect migrate pixel loss,Representing the word effect migration condition against the loss of resistance, lambda spix、λsadv is the weight of each of the two parts.
The effect migrate pixel loss is to instruct the generator to generate an image that is closer to the target effect image,The calculation formula of (2) is as follows:
Where x represents a glyph image, y represents a reference image corresponding to x and having a target effect, z represents a target effect image, E x represents an encoder E x,Ez represents a style encoder E z,S1 and S 2 represent two shared layers S 1 and S 2,G2 represent an effect image generator G 2,G2(concat(S1(Ex(x)),S2(Ez (z))) respectively) represents an effect migration generated image.
Word effect migration condition combat penaltyAlso, using the design in WGAN-GP, the calculation formula is as follows:
where lambda gp is the weight of the gradient penalty, Is the result of the fusion of the standard word effect image and the generated word effect image,Representation ofD 2 represents the arbiter D 2.The calculation formula of (2) is as follows:
Where ε represents the fusion weight parameter.
Further, step S4 includes the steps of:
Step S41, two pairs of images with the same word effect are randomly selected from the data set, wherein the font image and the word effect image of the first image pair are respectively marked as A s and A e, and the font image and the word effect image of the second image pair are respectively marked as B s and B e.
Step S42, training image coding and reconstruction. Input glyph image A s, through encoder E x, shared layer S 1, and generator G 1, generates an encoded and reconstructed imageAnd calculates the loss in step S32, calculates the gradient of each parameter in E x、S1 and G 1 using the back propagation method, and updates the parameters using Adam optimization method.
And S43, training word effect to remove the discrimination network of the subnet. Input word effect image A e, obtaining word effect removed image through encoder E y, shared layer S 1 and generator G 1 Calculating word effect removal conditions against resistance lossThe gradient of each parameter in the arbiter D 1 is calculated using the back propagation method and the parameters are updated using Adam optimization.
And S44, training word effect removing the generation network of the sub-network. Input word effect image A e, obtain feature S 1(Ey(Ae) through encoder E y, shared layer S 1), and obtain word effect removed image through generator G 1 Input glyph image a s, feature S 1(Ex(As is obtained by encoder E x, shared layer S 1). Calculating a total loss function of a word-effect removal subnetThe gradient of each parameter in E y、S1 and G 1 was calculated using the back-propagation method and the parameters were updated using the Adam optimization method.
And step S45, training the handwriting migration subnetwork in two stages, wherein each stage is used for training the discrimination network and the generation network of the handwriting migration subnetwork. And judging the first stage of the network of the word effect migration subnet. Input font image A s, obtained by encoder E x, shared layer S 1 Input word effect image B e, obtain characteristics through encoder E z, sharing layer S 2 Will beAndConnecting the inputs as generator G 2 in the channel dimension, generating a word-effect migration imageCalculating word effect migration conditions against resistance lossThe gradient of each parameter in the arbiter D 2 is calculated using the back propagation method and the parameters are updated using Adam optimization. And a first stage of generating a network for training the handwriting migration subnetwork. The input image is the same as the training discrimination network, and the character effect migration image is generatedComputing total loss function of word effect migration subnetworkThe gradient of each parameter in E x、Ez、S1、S2 and G 2 was calculated using the back-propagation method and the parameters were updated using the Adam optimization method.
And S46, training a second stage of the network of the word effect migration subnet. Input font image B s, obtained by encoder E x, shared layer S 1 Inputting the word effect migration image generated in step S45Obtaining features by encoder E z, shared layer S 2 Will beAndConnecting the inputs as generator G 2 in the channel dimension, generating a word-effect migration imageCalculating word effect migration conditions against resistance lossThe gradient of each parameter in the arbiter D 2 is calculated using the back propagation method and the parameters are updated using Adam optimization. And a second stage of generating a network for training the handwriting migration subnetwork. The input image is identical to the training discriminator, and a word effect migration image is generatedComputing total loss function of word effect migration subnetworkThe gradient of each parameter in E x、Ez、S1、S2 and G 2 was calculated using the back-propagation method and the parameters were updated using the Adam optimization method.
In step S47, the above steps are a complete iteration for two pairs of images, two hundred and fifty iterations are needed in the whole training process, and in each iteration process, a plurality of pairs of images with two pairs of the same word effect are randomly sampled and used as a batch for training, the previous hundred iterations do not perform the training of the second stage in step S46, and the remaining hundred and fifty iterations perform the training of all steps.
Further, step S5 is implemented as follows:
Inputting the font image to be migrated, and obtaining font characteristics through the encoder E x and the sharing layer S 1 Inputting target word effect image, obtaining word effect characteristic by encoder E z and sharing layer S 2 Will beAndThe input as generator G 2 is connected in the channel dimension, generating a word effect migration image having the word effect of the word effect image and the glyph of the glyph image.
The above is a preferred embodiment of the present invention, and all changes made according to the technical solution of the present invention belong to the protection scope of the present invention when the generated functional effects do not exceed the scope of the technical solution of the present invention.
Claims (2)
1. A method for word effect migration based on a loop generation countermeasure network, comprising the steps of:
S1, processing a data set to obtain a font image and a word effect image pair, and preprocessing each image to obtain a training data set;
Step S2, designing a word effect style migration network based on a loop generation countermeasure network, wherein the network consists of a word effect removal subnet and a word effect migration subnet;
Step S3, designing a loss function for training the network designed in the step S2;
s4, training a word effect style migration network based on a loop generation countermeasure network by using a training data set;
S5, inputting the new target character effect image and the character pattern image to be migrated into a trained character effect style migration network, and outputting an image with the target character effect after character effect migration;
the step S1 is specifically implemented as follows:
S11, dividing each image in the original input data set into an independent font image and a word effect image with the same size by using the font image and the word effect image with the same size; combining the font image and the word effect image from the same original image into a font image and word effect image pair;
step S12, the two images of the font image and the word effect image pair are subjected to the same random overturn and are cut into a font image and a word effect image with H multiplied by W at the center, and the relationship between the font image and the word effect image pair is maintained;
Step S13, normalizing all font images and word effect images with H multiplied by W, giving an image I (I, j), and calculating a normalized image The formula of (2) is as follows:
Wherein (i, j) represents the position of the pixel;
the step S2 is specifically implemented as follows:
s21, designing a generation network and a discrimination network of a word effect removal subnet, and respectively generating probabilities of a word effect removal image and a discrimination image being a real image by using the designed network;
Step S22, designing a generation network and a discrimination network of a word effect migration subnet, and respectively generating probabilities that a word effect migration image and a discrimination image are real images by using the designed network;
the step S21 is specifically implemented as follows:
Step S211, designing a generating network of the word effect removing sub-network, wherein the input of the generating network is a pair of a font image and a word effect image, the font image is represented by A s, and the word effect image is represented by A e; the generating network is composed of encoders E x and E y, a shared layer S 1 for extracting image features, and a generator G 1 for reconstructing features into word effect removed images
Step S212, design encoders E y and E x, whose inputs are a e and a s, respectively; the encoder E y consists of five convolution blocks, each of which consists of an activation layer, a convolution layer, a normalization layer and an attention module in sequence; the activation layer adopts LeakyReLU activation functions, the convolution layer uses convolution with a convolution kernel of 4×4, a step size of 2 and a filling of 1, the normalization layer uses an example for normalization, and the attention module uses a convolution attention module (CBAM); image characteristics of size C x H x W obtained by inputting image into encoder E y The encoder E x and the encoder E y have the same structure, and c×h×w image characteristics obtained by inputting an image to the encoder E x
Step S213, designing a shared layer S 1,S1 to be a network with a Unet structure; features of size C.times.H.times.W obtained in step S212AndAll are used as the input of the sharing layer S 1, the input f 0,f0 is sequentially processed by three convolution blocks which are the same as those in S212 to respectively obtain the characteristic f 1 with the size of C multiplied by H/2 multiplied by W/2, the characteristic f 2 with the size of C multiplied by H/4 multiplied by W/4 and the characteristic f 3;f3 with the size of C multiplied by H/8 multiplied by W/8, and the like, and the characteristic f 1 with the size of C multiplied by H/2 multiplied by W/2, the characteristic f 2 with the size of C multiplied by H/8 multiplied by W/8 is obtained by the activating layer, Deconvolution blocks composed of a deconvolution layer, a normalization layer and an attention module in sequence obtain a characteristic f 4 of C multiplied by H/4 multiplied by W/4; The activation layer in the deconvolution block adopts a ReLU activation function, the deconvolution layer uses deconvolution with a convolution kernel of 4 multiplied by 4, a step length of 2 and a filling of 1, the normalization layer uses an example for normalization, and the attention module uses a convolution attention module (CBAM); the feature f 2 and the feature f 4 are spliced in the channel dimension to obtain a feature with the size of 2C x H/4 x W/4, and the feature is subjected to deconvolution to obtain a feature f 5 with the size of C x H/2 x W/2; The feature f 1 and the feature f 5 are spliced in the channel dimension to obtain a feature with the size of 2C multiplied by H/2 multiplied by W/2, and the feature is subjected to deconvolution to obtain a feature f 6 with the size of C multiplied by H multiplied by W; the feature f 0 and the feature f 6 are spliced in the channel dimension to obtain an output feature f s with the size of 2C multiplied by H multiplied by W; When the input features areThe output characteristic f s is then noted asWhen the input features areThe output characteristic f s is then noted as
Step S214, a design generator G 1; the input to generator G 1 is the characteristics generated in step S213Or (b)The generator G 1 consists of five deconvolution blocks identical to those in step S213, a deconvolution layer with a convolution kernel of 3 x 3, a step size of 1, a padding of 1, and a tanh activation function; each deconvolution block reduces the channel number of the feature by half and improves the width and the height of the feature to two times; when the input features areFeatures of size 2C H WThe feature with the size of C/16 multiplied by 32H multiplied by 32W is obtained through five deconvolution blocks, and finally the feature is subjected to deconvolution layer and tanh activation function to obtain A3-channel word effect removal image with the same size as the input word effect image A e When the input features areFeatures of size 2C H WThe feature with the size of C/16 multiplied by 32H multiplied by 32W is obtained through five deconvolution blocks, and finally the feature is subjected to deconvolution layer and tanh activation function to obtain A3-channel font reconstruction image with the same size as the input font image A s
Step S215, designing a discrimination network of a word effect removal subnet, wherein the discrimination network consists of a discriminator D 1, and the discriminator D 1 adopts a structure of a Markov discriminator in PatchGAN; the discriminator D 1 consists of 5 convolution blocks which are sequentially composed of a convolution layer, an instance normalization layer and an activation layer, wherein the convolution layer uses convolution with a convolution kernel of 4 multiplied by 4, a step length of 2 and a filling of 1, the normalization layer uses instance normalization, and the activation layer uses LeakyReLU activation functions; the inputs to arbiter D 1 are A e andThe result obtained by splicing in the channel dimension, namely a false sample or the result obtained by splicing A e and A s in the channel dimension, namely a real sample, is output as an N multiplied by 1 tensor, wherein N=32, and the probability and the calculation loss of the image as the real image are judged by the average value of the tensor;
The step S22 is specifically implemented as follows:
Step S221, designing a generating network of a word effect migration subnet, wherein the inputs of the generating network are a word pattern image A s to be migrated and a target word effect image B e; the generating network is composed of an encoder E x、Ez, a sharing layer S 2 and a generator G 2, wherein the encoder and the sharing layer are used for extracting image features, and the generator reconstructs the features into word-effect migration images
Step S222, designing the encoder E z,Ez, and the encoders E x and E y in step S212 to have the same structure, and inputting the target byte image B e into the encoder to obtain image characteristics with the size of C×H×W
Step S223, designing a shared layer S 2,S2 to be a network with a Unet structure; the structure of the shared layer S 2 is the same as that of the shared layer S 1 in step S213, and the input characteristic is a characteristic of the size of c×h×w output from the encoder in step S222Features of size 2C x H x W
Step S224, designing a generator G 2, the structure of the generator G 2 is the same as the structure of the generator G 1 in step S214, but the input is the output feature of the shared layer S 1 And sharing layer S 2 output featuresStitching features in the channel dimension, feature size 4C x H x W, output 3-channel and same size word-effect migrated image as input image B e
Step S225, designing a discrimination network of the word effect migration sub-network, wherein the discrimination network consists of a discriminator D 2, a discriminator D 2 adopts the structure of a Markov discriminator in PatchGAN, and the structures of a discriminator D 2 and a discriminator D 1 in S215 are the same; the input to arbiter D 2 is the word effect migration image generated by A s、Be and generator G 2 Splicing in the channel dimension, namely a false sample, or splicing in the channel dimension, namely a real sample, of the A s、Be and the standard word effect migration image A e, wherein the output is an N multiplied by 1 tensor, N=32, and the probability and the calculation loss of the image as the real image are judged through the mean value of the tensor;
The step S3 is specifically implemented as follows:
step S31, designing a total optimization target of the whole network, wherein the optimization target is as follows:
Wherein, Representing the loss function of the training image encoding and reconstruction,Representing the training handwriting effect removal subnet loss function,The loss function of the training handwriting migration subnet is represented, E, G, D respectively represents an encoder (E x、Ey、Ez), a generator (G 1、G2) and a discriminator (D 1、D2);
step S32, designing a loss function of training image coding and reconstruction, The calculation formula of (2) is as follows:
where x represents a glyph image, G 1(S1(Ex (x))) represents an image encoded and reconstructed image, λ rec represents a weight, E x represents an encoder E x,S1 represents a shared layer S 1,G1 represents a generator G 1,||||1 that is an absolute value taking operation;
step S33, designing a loss function for removing the subnet by training handwriting effect, The calculation formula of (2) is as follows:
Wherein, Indicating a loss of the characteristic(s),Indicating that the word effect removes the pixel loss,Representing the word effect removal condition against the loss of resistance, λ dfeat,λdpix and λ dadv are the weights of each of the three parts;
The feature loss directs the encoder to remove the word effect from the word effect image, retaining only the glyph information, and the formula for λ dfeat is as follows:
Wherein x represents a font image, y represents a word effect image, E x and E y represent two encoders E x and E y,S1 represent that the shared layer S 1,||||1 is an absolute value taking operation;
the word effect removal pixel loss is a word effect removal image that directs the generator to generate a more similar content glyph image, The calculation formula of (2) is as follows:
Where x represents a glyph image, G 1 represents a generator G 1, y represents a word effect image, G 1(S1(Ey (y))) represents an image from which a word effect is removed, and || 1 is an absolute value taking operation;
the word effect removal condition is used to improve the quality of the generated result, enable the generator to learn to "fool" the arbiter, also enable the arbiter to distinguish between the real image and the generated image, employ the condition in WGAN-GP against the loss, The calculation formula of (2) is as follows:
where lambda gp is the weight of the gradient penalty, Is the result of the fusion of the font image and the generated word effect removing image,Representation ofD 1 represents the gradient of the arbiter D 1,The calculation formula of (2) is as follows:
wherein epsilon represents a fusion weight parameter;
step S34, designing a loss function of the training handwriting migration subnetwork, The calculation formula of (2) is as follows:
Wherein, Representing a word effect migrate pixel loss,Representing the word effect migration condition against the loss of resistance, lambda spix、λsadv is the weight of each of the two parts;
the effect migrate pixel loss is to instruct the generator to generate an image that is closer to the target effect image, The calculation formula of (2) is as follows:
Where x represents a glyph image, y represents a reference image corresponding to x and having a target word effect, z represents a target word effect image, E x represents an encoder E x,Ez represents a style encoder E z,S1 and S 2 represent two shared layers S 1 and S 2,G2 represent a word effect image generator G 2,G2(concat(S1(Ex(x)),S2(Ez (z))) respectively), and a word effect migration generated image;
word effect migration condition combat penalty Also, using the design in WGAN-GP, the calculation formula is as follows:
where lambda gp is the weight of the gradient penalty, Is the result of the fusion of the standard word effect image and the generated word effect image,Representation ofD 2 represents the gradient of the arbiter D 2,The calculation formula of (2) is as follows:
wherein epsilon represents a fusion weight parameter;
The step S4 is specifically implemented as follows:
Step S41, randomly selecting two pairs of images with the same word effect from a data set, wherein the font image and the word effect image of the first image pair are respectively marked as A s and A e, and the font image and the word effect image of the second image pair are respectively marked as B s and B e;
s42, training image coding and reconstruction; input glyph image A s, through encoder E x, shared layer S 1, and generator G 1, generates an encoded and reconstructed image Calculating the loss in the step S32, calculating the gradient of each parameter in E x、S1 and G 1 by using a back propagation method, and updating the parameters by using an Adam optimization method;
Step S43, training word effect removing the discrimination network of the sub-network; input word effect image A e, obtaining word effect removed image through encoder E y, shared layer S 1 and generator G 1 Calculating word effect removal conditions against resistance lossCalculating the gradient of each parameter in the discriminator D 1 by using a back propagation method, and updating the parameters by using an Adam optimization method;
step S44, training word effect removing the generation network of the sub-network; input word effect image A e, obtain feature S 1(Ey(Ae) through encoder E y, shared layer S 1), and obtain word effect removed image through generator G 1 Inputting a font image a s, obtaining a feature S 1(Ex(As by the encoder E x and the sharing layer S 1); calculating a total loss function of a word-effect removal subnetCalculating gradients of the parameters in E y、S1 and G 1 by using a back propagation method, and updating the parameters by using an Adam optimization method;
Step S45, training the handwriting migration subnetwork in two stages, wherein each stage is used for training a judging network and a generating network of the handwriting migration subnetwork; judging a first stage of a network of the word effect migration subnetwork; input font image A s, obtained by encoder E x, shared layer S 1 Input word effect image B e, obtain characteristics through encoder E z, sharing layer S 2 Will beAndConnecting the inputs as generator G 2 in the channel dimension, generating a word-effect migration imageCalculating word effect migration conditions against resistance lossCalculating the gradient of each parameter in the discriminator D 2 by using a back propagation method, and updating the parameters by using an Adam optimization method; training a first stage of generating a handwriting migration subnet; the input image is the same as the training discrimination network, and the character effect migration image is generatedComputing total loss function of word effect migration subnetworkCalculating gradients of the parameters in E x、Ez、S1、S2 and G 2 by using a back propagation method, and updating the parameters by using an Adam optimization method;
Step S46, training a second stage of the network discrimination of the word effect migration subnetwork; input font image B s, obtained by encoder E x, shared layer S 1 Inputting the word effect migration image generated in step S45Obtaining features by encoder E z, shared layer S 2 Will beAndConnecting the inputs as generator G 2 in the channel dimension, generating a word-effect migration imageCalculating word effect migration conditions against resistance lossCalculating the gradient of each parameter in the discriminator D 2 by using a back propagation method, and updating the parameters by using an Adam optimization method; training a second stage of generating a network of the handwriting migration subnetwork; the input image is identical to the training discriminator, and a word effect migration image is generatedComputing total loss function of word effect migration subnetworkCalculating gradients of the parameters in E x、Ez、S1、S2 and G 2 by using a back propagation method, and updating the parameters by using an Adam optimization method;
In step S47, the above steps are a complete iteration for two pairs of images, two hundred and fifty iterations are needed in the whole training process, and in each iteration process, a plurality of pairs of images with two pairs of the same word effect are randomly sampled and used as a batch for training, the previous hundred iterations do not perform the training of the second stage in step S46, and the remaining hundred and fifty iterations perform the training of all steps.
2. The method for word effect migration based on loop generation countermeasure network according to claim 1, wherein the step S5 is specifically implemented as follows:
Inputting the font image to be migrated, and obtaining font characteristics through the encoder E x and the sharing layer S 1 Inputting target word effect image, obtaining word effect characteristic by encoder E z and sharing layer S 2 Will beAndThe input as generator G 2 is connected in the channel dimension, generating a word effect migration image having the word effect of the word effect image and the glyph of the glyph image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210018956.9A CN114399427B (en) | 2022-01-07 | 2022-01-07 | Word effect migration method based on loop generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210018956.9A CN114399427B (en) | 2022-01-07 | 2022-01-07 | Word effect migration method based on loop generation countermeasure network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114399427A CN114399427A (en) | 2022-04-26 |
| CN114399427B true CN114399427B (en) | 2024-06-28 |
Family
ID=81229329
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210018956.9A Active CN114399427B (en) | 2022-01-07 | 2022-01-07 | Word effect migration method based on loop generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114399427B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112307714A (en) * | 2020-11-03 | 2021-02-02 | 武汉理工大学 | Character style migration method based on double-stage deep network |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10621760B2 (en) * | 2018-06-15 | 2020-04-14 | Adobe Inc. | Synthesizing new font glyphs from partial observations |
| CN110097615B (en) * | 2018-12-06 | 2021-01-19 | 北京大学 | Stylized and de-stylized artistic word editing method and system |
| CN110570346B (en) * | 2019-08-19 | 2022-11-04 | 西安理工大学 | Method for performing style migration on calligraphy based on cyclic generation countermeasure network |
| CN112633430B (en) * | 2020-12-25 | 2022-10-14 | 同济大学 | Chinese font style migration method |
| CN113792743A (en) * | 2021-08-24 | 2021-12-14 | 西安理工大学 | Ancient book Chinese character image denoising method based on progressive confrontation generation |
-
2022
- 2022-01-07 CN CN202210018956.9A patent/CN114399427B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112307714A (en) * | 2020-11-03 | 2021-02-02 | 武汉理工大学 | Character style migration method based on double-stage deep network |
Non-Patent Citations (1)
| Title |
|---|
| 基于生成式对抗网络的字体风格迁移方法;白海娟;周未;王存睿;王磊;;大连民族大学学报;20190515(第03期);第250-256页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114399427A (en) | 2022-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110503598B (en) | Font style migration method for generating countermeasure network based on conditional cycle consistency | |
| CN105930308B (en) | Non-negative matrix factorization method based on low-rank recovery | |
| CN111861906B (en) | Pavement crack image virtual augmentation model establishment and image virtual augmentation method | |
| CN109948692B (en) | Computer-generated picture detection method based on multi-color space convolutional neural network and random forest | |
| CN113140018B (en) | Method for training countermeasure network model, method for establishing word stock, device and equipment | |
| CN112084328A (en) | Scientific and technological thesis clustering analysis method based on variational graph self-encoder and K-Means | |
| CN113298734B (en) | Image restoration method and system based on mixed hole convolution | |
| CN111986075A (en) | Style migration method for target edge clarification | |
| CN112163401A (en) | Chinese character font generation method based on compressed and excited GAN network | |
| CN113920516A (en) | Calligraphy character skeleton matching method and system based on twin neural network | |
| CN107240100B (en) | Image segmentation method and system based on genetic algorithm | |
| CN117788629A (en) | Image generation method, device and storage medium with style personalization | |
| CN114399427B (en) | Word effect migration method based on loop generation countermeasure network | |
| CN115859011B (en) | Matrix operation method, device, unit and electronic equipment | |
| CN110348023A (en) | A kind of method, apparatus, storage medium and the electronic equipment of Chinese text participle | |
| CN114693675A (en) | Multi-channel HE staining pathological image segmentation method introducing gating axial self-attention | |
| CN113688715A (en) | Facial expression recognition method and system | |
| CN113111906A (en) | Method for generating confrontation network model based on condition of single pair image training | |
| Minarno et al. | Generation of Batik Patterns Using Generative Adversarial Network with Content Loss Weighting. | |
| Yoon et al. | Layer-Wise Sparse Training of Transformer via Convolutional Flood Filling | |
| CN116071275B (en) | Face image restoration method based on online knowledge distillation and pretraining priori | |
| CN116993998A (en) | Contour detection method based on biological vision multi-level interaction mechanism | |
| CN117456286B (en) | Ginseng grading method, device and equipment | |
| CN116740343B (en) | Image segmentation method and device based on quantum classical mixed neural network | |
| CN114627196B (en) | Latent variable space decoupling method based on variational automatic encoder |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |