CN117058266A - Handwriting word generation method based on skeleton and outline - Google Patents

Abstract

The invention discloses a method for generating calligraphy characters based on skeleton and outline, which includes the following steps: establishing a model; the model uses the CycleGAN model as a backbone network, and the CycleGAN model includes two sets of generative adversarial networks, and the model also includes Con, Ske , IPaD and SCF; Step 2, train the model; the source domain style Chinese character image is used as the original image input model, and the original image is converted into the target style image through the first group of generative adversarial networks, and the second group of generative adversarial networks is used to convert the original image into a target style image. The target style images output by the first set of generative adversarial networks are converted into reconstructed images. During the training process, the model is optimized by calculating the loss of the entire model; Step 3: Obtain the optimized model for automatic generation of calligraphy fonts. The present invention introduces an effective skeleton-contour fusion module to fuse skeleton information and contour information, and can achieve high-quality content style performance in the absence of accurate paired font samples.

Description

A calligraphy character generation method based on skeleton and outline

Technical field

The invention belongs to the field of computer vision technology, and specifically relates to a calligraphy character generation method based on skeleton and outline.

Background technique

Chinese calligraphy is an art form based on Chinese characters and is written primarily with a brush. In recent years, with the rapid development of artificial intelligence technology, research on the automatic generation of Chinese calligraphy has gradually emerged, committed to the digital protection and inheritance of cultural heritage, and established a widely applicable Chinese calligraphy text database. However, the automatic generation of calligraphy Chinese characters is technically quite challenging, mainly reflected in the following two aspects: 1. The shapes of calligraphy characters are diverse, and the overall shapes of calligraphy fonts are also very different. 2. Most calligraphy characters are traditional characters, and their structures are more complex than simplified characters.

To address the above two challenges, existing Chinese character generation methods are usually considered as an image-to-image conversion problem. In the existing technology, some use the Pix2Pix model to generate Chinese fonts, and realize the generation of calligraphy fonts by building a deep neural network model that directly generates calligraphy characters from standard font characters. Another existing technology builds an effective calligraphy generation model LF-Font, which extracts content and style representation by utilizing pairs of characters and components. However, these models require paired data for training and a large number of paired samples are collected. It is often impractical and cumbersome, especially for certain font generation problems, such as ancient calligraphy fonts, which makes it difficult for existing techniques to obtain sufficient paired fonts in small sample cases, making it difficult for these models to obtain accurate and reliable results.

In order to solve the problem of data pairing, some technicians use CycleGAN models to generate Chinese fonts based on unpaired data, such as the deformable generation model DG-Font. This technique introduces certain stroke encodings to alleviate the problem of mode collapse. Some existing techniques also propose their semi-supervised variants by using a small number of paired samples as supervision, and others utilize multiple square-block transformations. To capture the glyph structure of Chinese characters, other existing technologies use the outline of Chinese characters to obtain global information.

Although these supervised, unsupervised, and self-supervised models are very effective for general Chinese font generation, these existing techniques are still ineffective when applied to Chinese calligraphy generation due to the diverse shapes of Chinese characters and the very different styles of different font pieces. It is unsatisfactory, especially the difficulty in producing high-quality performance of content and style, which is also the key to the generation of Chinese calligraphy. Some of the above techniques still require a certain amount of paired data to provide significant supervision for generating results, but collecting the amount of paired data is very difficult. However, simply using the skeleton or outline of text to generate fonts often has some flaws in the style or content, and still cannot meet the needs for automatic generation of Chinese calligraphy fonts.

Contents of the invention

The purpose of the present invention is to provide a method for generating calligraphy characters based on skeleton and outline, which is used to solve the problem in the existing technology that the generated Chinese calligraphy fonts are difficult to produce high-quality content and style expression without sufficient paired font supervision. technical problem.

The method for generating calligraphy characters based on skeleton and outline includes the following steps.

Step 1: Establish a model; the model uses the CycleGAN model as the backbone network, and the CycleGAN model includes two sets of generative adversarial networks.

Step 2: Train the model; the Chinese character font style input to the model is the source domain style, and the Chinese character image in the source domain style is the source domain image. The Chinese character images in the source domain style are collected as training samples. The calligraphy font image that needs to be generated is The font style is the target style, and the calligraphy font image of the target style is the target domain image. The target domain image is collected to form a calligraphy data set; the source domain image is used as the original image input model during training, and the original image is converted into For the target style image, the target style image output by the first group of generative adversarial networks is converted into a reconstructed image through the second group of generative adversarial networks. The font style of the target style image should be consistent with the target style, and the font style of the reconstructed image should be consistent with the source The domain style is consistent. During the training process, the model is optimized by calculating the loss of the entire model. The optimization goal is to minimize the loss of the entire model.

Step 3: Obtain the optimized model for automatic generation of calligraphy fonts.

Among them, both sets of generative adversarial networks include a contour extraction module Con, a skeleton extraction module Ske, and a skeleton-contour fusion module SCF. The model also includes an inexact pairing data module IPaD.

In the second step, both groups of generative adversarial networks extract skeleton information and contour information respectively through the contour extraction module Con and the skeleton extraction module Ske, and fuse the skeleton information and contour information through the skeleton-contour fusion module SCF in the generator. It is spliced with the image input to the generator, and then processed by the corresponding generator to generate the image.

The inexact pairing data module IPaD automatically recognizes the characters in the calligraphy data set and records them as identification tags, and then performs inexact pairing in the calligraphy data set based on the target style image. During pairing, it allows the use of wrong identification tags for the relevant calligraphy data sets, thus obtaining Inexact pairing data.

The losses of the entire model include the first-generation adversarial loss L _advy , the second-generation adversarial loss L _advx , cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con and imprecise pairing loss L _inex .

Preferably, in step one, the first group of generative adversarial networks includes the constructed generator G _y and the discriminator D _y , and the second group of generative adversarial networks includes the constructed generator G _x and the discriminator D _x ; The generator G _y is used to convert the original image into a target style image, and the discriminator D _y is used to determine whether the font style between the generated target style image and the target domain image is consistent; the second group of generative adversarial networks uses the opposite The process reconstructs the results output by the first set of generative adversarial networks, that is, the target style image is converted into a reconstructed image of the source domain style through the generator 2 G _x , and the discriminator 2 D _x is used to distinguish between the generated reconstructed image and Whether the font style between source domain images is consistent.

Preferably, in the second step, in the first group of generative adversarial networks, the original image of the source domain image x as input is processed by the skeleton extraction module Ske and the contour extraction module Con respectively, and the corresponding skeleton information sx and contour information cx are extracted. , the skeleton information sx and the contour information cx are fused through the skeleton-contour fusion module SCF; the original image x is input to the generator G _y , and during the processing, the generator G _y combines the original image x with the skeleton-contour fusion module SCF The obtained skeleton features E _asx and contour features E _bcx are spliced at the channel level, and the target style image is generated after processing. , collect the target domain image y to form the target domain data set Y , and convert the target style image/> and the target domain image y in the target domain data set Y are respectively input into the discriminator D _y to determine whether the results returned by the discriminator D _y are consistent, so as to evaluate the target style image/> authenticity.

Preferably, after inputting the skeleton information and contour information of a given Chinese character to the skeleton-contour fusion module SCF, the skeleton-contour fusion module SCF first inputs them into the corresponding skeleton encoder and contour encoder to generate the corresponding skeleton feature E _sx and contour feature E _cx ; then add the encoded skeleton feature E _sx and contour feature E _cx to obtain the feature E _scx and use the SoftMax function to obtain the normalized feature c ^Z ; based on the normalized feature c ^Z , use the attention weight The formula calculates the weight a _c of the corresponding skeleton feature E _sx and the weight b _c of the contour feature E _cx ; finally, multiply the calculated weights a _c and b _c by the corresponding skeleton feature E _sx and contour feature E _cx to obtain the fusion The weighted skeleton feature E _asx and the fused weighted contour feature E _bcx are calculated as follows:

,/>

Among them, a _c , b _c and c c in ^Z all represent the calculation on channel c , and A and B are matrices of two learnable parameters.

Preferably, in the second group of generative adversarial networks, the target style image Then through the skeleton extraction module Ske and the contour extraction module Con, the corresponding skeleton information is extracted/> and contour information/> , skeleton information/> and contour information/> The two are fused through the skeleton-contour fusion module SCF; target style image/> Input generator two G _x . During the processing, generator two G _x converts the target style image/> The corresponding skeleton features and corresponding contour features obtained by fusion with the skeleton-contour fusion module SCF are spliced at the channel level, and reconstructed to generate a reconstructed image consistent with the style of the source domain/> ;Collect the source domain image x to form the source domain data set X , and reconstruct the image/> After inputting the source domain image x in the source domain _data set X to the discriminator D _x , it is judged whether the results returned by the two discriminators D authenticity.

Preferably, in the second step, in the first group of generative adversarial networks in the CycleGAN model, the difference in font style between the target style image and the target domain image is calculated by the discriminator D _y , that is, the first generation adversarial The loss L _advy is used to optimize the generator one G _y ; the input of the second group of generative adversarial networks is based on the output of the generator one G _y in the first group of generative adversarial networks, and the discriminator two D _x in the second group of generative adversarial networks The difference in font style between the source domain image and the reconstructed image is calculated, that is, the second-generation adversarial loss L _advx , which is used to optimize the generator two G _x .

Cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con , and imprecise pairing loss L _inex all correspond to the optimized generator two G _x and generator one G _y ; the cycle consistency loss L _cyc is Original image x and reconstructed image in source domain style The loss between; the skeleton consistency loss L _ske is the skeleton information sx of the original image x and the reconstructed image/> Skeleton information extracted from The loss between; the contour consistency loss L _con is the contour information cx of the original image x and the reconstructed image/> Contour information extracted from The loss between, the imprecise pairing loss L _inex is the imprecise pairing data y _inex and the target style image corresponding to the imprecise pairing data/> between losses.

Preferably, the second generation adversarial loss L _advx , the first generation adversarial loss L _advy , cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con , and imprecise pairing _loss Linex The calculation formula is as follows:

Among them, E _{x ~ X} [ ] represents the expected value of the data in [ ] under the given source domain image x distribution in the source domain data set X , Represented in a given set of reconstructed images/> Reconstructed image in/> The expected value of the data in [ ] under the distribution, log D _x ( x ) represents the probability that the discriminator D _x recognizes the source domain image x as the source domain image, log (1- log D _x (/> )) means that the discriminator D _x will reconstruct the image/> The probability of being recognized as not being a source domain image; E _{y ~ Y} [ ] represents the expected value of the data in [ ] under the given target domain image y distribution in the target domain data set Y , /> Represents a collection of images in a given target style/> Target style image in/> The expected value of the data in [ ] under the distribution, log D _y ( y ) represents the probability that the discriminator D _y recognizes the target domain image y as the target domain image, log (1- log D _y (/> )) means that the discriminator D _y converts the target style image/> The probability of being recognized as an image that is not the target domain;/> Represents a source domain image x in a given source domain data set X and a given set of reconstructed images /> Reconstructed image in/> The expected value of the norm of the data in || || ₁ under the distribution, Ske ( x ) and Ske (/> ) respectively represent the source domain image x and reconstructed image /> through the skeleton extraction module Ske Process the results obtained, Con ( x ) and Con (/> ) respectively represent the source domain image x and reconstructed image /> through the contour extraction module Con Processing results;/> Represents reconstructed image/> The set of Y _inex represents the set of inexact paired data y _inex , /> Represents the target style image corresponding to the imprecisely paired data, /> Represents target style images corresponding to imprecisely paired data/> collection of Represents the inexact paired data y _inex in the given set Y _inex and the given set /> Correspondence to target style image of imprecise paired data/> The expected value of the norm of the data in || || ₁ under the distribution.

Preferably, the model loss of the entire model The calculation formula is as follows:

In this formula, λ _cyc , λ _ske , λ _con , and λ _inex are four possible parameters corresponding to the cycle consistency loss L _cyc , the skeleton consistency loss L _ske , the contour consistency loss L _con , and the inaccurate pairing loss L _inex respectively. The tuned hyperparameter represents the weight of the corresponding loss in the entire model loss.

The present invention has the following advantages: since calligraphy fonts are more complex and include multiple calligraphy style features such as connected strokes, stroke sharpness, thickness, etc., these features are difficult to characterize using skeletons, stroke codes or other components alone. Therefore, this scheme introduces silhouettes to represent these stylistic features. The pure outline information cannot determine the content of the character, so an effective skeleton-contour fusion module is introduced to fuse the skeleton information and outline information. This solution also uses the imprecise paired data module IPaD to automatically identify characters in the calligraphy data set and record them as identification tags to obtain an imprecise paired data set. The imprecise paired dataset is used to compute the image-level loss between the generated image and the corresponding imprecise paired image in the imprecise paired dataset. Based on the above technical characteristics, this solution can comprehensively utilize skeleton or outline information, and can automatically generate Chinese calligraphy fonts without requiring a large number of paired samples, and can achieve high-quality content style expression.

Description of the drawings

Figure 1 is a model flow chart of a calligraphy character generation method based on skeleton and outline according to the present invention.

Figure 2 is a schematic diagram of the workflow of the skeleton-contour fusion module SCF in the present invention.

Figure 3 is a comparison diagram of Chinese character generation results between the present invention and the prior art.

Figure 4 is a comparison of the block script fonts and the calligraphy fonts of Yu Youren and Zhu Suiliang.

Figure 5 is a rendering of the present invention's conversion of four different sets of Chinese characters in block script fonts into calligraphy fonts of Master Bada Shanren, Huang Tingjian, Zhu Suiliang and Master Hongyi respectively.

Detailed ways

The specific embodiments of the present invention will be further described in detail by describing the embodiments with reference to the accompanying drawings to help those skilled in the art have a more complete, accurate and thorough understanding of the inventive concepts and technical solutions of the present invention.

As shown in Figures 1 and 2, the present invention provides a calligraphy character generation method based on skeleton and outline, which includes the following steps.

Step 1. Build the model.

CycleGAN model, namely cycle generative adversarial model, is an unsupervised learning model. The CycleGAN model contains two groups of generative adversarial networks. The first group includes the constructed generator G _y and the discriminator D _y . The second group of generative adversarial networks includes the constructed generator G _x and the discriminator D _x . In this scheme, the generator G _y is used to convert the original image into a target style image, and the discriminator D _y is used to determine whether the font style between the generated target style image and the target domain image is consistent, that is, to determine the target style image. authenticity. The second group of generative adversarial networks uses the opposite process to reconstruct the output results of the first group of generative adversarial networks, that is, the target style image is converted into a reconstructed image of the source domain style through the generator two G _x , and the discriminator two D _x It is used to determine whether the font style between the generated reconstructed image and the source domain image is consistent, that is, to determine the authenticity of the reconstructed image.

The generators in the above generative adversarial networks include encoders, converters and decoders. In the first group of generative adversarial networks in the CycleGAN model, the difference in font style between the target style image and the target domain image is calculated through the discriminator D _y . The loss of the discriminator D _y is combined with the loss of the generator G _y to form The first-generation adversarial loss L _advy is used to optimize the generator- G _y ; the input of the second group of generative adversarial networks is based on the output of the generator- G _y in the first group of generative adversarial networks. The discriminator 2 D _x calculates the difference in font style between the source domain _{image and the reconstructed image. The loss of the discriminator 2 D x is} combined _with the loss of the generator 2 G Generator two G _x . During training, the discriminator D _y and the discriminator D _x are generally trained first, and then the corresponding generators are optimized based on the two generations of adversarial losses obtained by the discriminator processing. The training of the generator in the conventional CycleGAN model is essentially the process of minimizing the adversarial losses of the above two generations. The training process can also alternately train the discriminator and generator.

In this solution, the CycleGAN model is used as the basic model to learn two mappings between the source domain and the target domain. The CycleGAN model can introduce cycle consistency loss to help overcome the limitations of paired data. The model established in this solution uses the CycleGAN model as the backbone network. The CycleGAN model contains two sets of generative adversarial networks. Both sets of generative adversarial networks include the contour extraction module Con, the skeleton extraction module Ske, and the skeleton-contour fusion module SCF. In the contour extraction module, since calligraphy feature images are usually represented in gray, the contour information can be easily extracted by using the famous Canny operator. In the skeleton extraction module, some existing skeleton schemes with the same simple rules are used (such as the paper: Jie Zhou, Yefei Wang, Yiyang Yuan, Qing Huang, and Jinshan Zeng, “Sgce-font: Skeleton guided channel expansion for chinese font generation," arXivpreprint arXiv:2211.14475, 2022.) to effectively extract skeleton information. In addition, this model also has an inexact paired data module IPaD. For the inexact paired data module, the existing Chinese Character Recognition method (CCR) is used. For example, papers: Jinshan Zeng, Ruiying Xu, Yu Wu, Hongwei Li, and Jiaxing Lu, “Zero-shot chinese character recognition with stroke and radical-level decompositions,” in Proceedings of the International Joint Conference on Neural Networks, 2023.) to automatically recognize characters in calligraphy datasets and record as recognition Label, after generating the target style image, perform similarity matching based on the target style image. The difference between the imprecise matching data module IPaD and the existing technology is that it allows the use of wrong identification labels for the relevant calligraphy data sets during matching, that is, the matching result is a Chinese character that is similar but different from the original image. Although some calligraphy Chinese characters are recognized incorrectly here, they can still provide some important reference information for related calligraphy Chinese characters.

Compared with Simplified Chinese fonts, calligraphy fonts are more complex and include multiple calligraphy style features such as connected strokes, stroke sharpness, thickness, etc. These features are difficult to characterize using skeletons, stroke encodings, or other components alone. Therefore, silhouettes are introduced to represent these stylistic features. The pure outline information cannot determine the content of the character, so an effective skeleton-contour fusion module is introduced to fuse the skeleton information and outline information. The architecture of the skeleton-contour fusion module is shown in Figure 2.

Step 2: Train the model.

The above model integrates the skeleton-contour fusion module SCF and the inexact paired data module IPaD. The proposed model integrates the skeleton and outline information of Chinese characters to provide comprehensive structural supervision information.

The basic workflow of training includes: training the model; the Chinese character font style input to the model is the source domain style, the Chinese character image in the source domain style is the source domain image, and the Chinese character images in the source domain style are collected as training samples, which need to be generated The font style of the calligraphy font image is the target style, and the calligraphy font image of the target style is the target domain image. The target domain image is collected to form a calligraphy data set; the source domain image is used as the original image input model during training, and the first set of generative adversarial networks is used to generate the calligraphy font image. The original image is converted into a target style image, and the target style image output by the first group of generative adversarial networks is converted into a reconstructed image through the second group of generative adversarial networks. The font style of the target style image should be consistent with the target style, and the font of the reconstructed image The style should be consistent with the source domain style. During the training process, the model is optimized by calculating the loss of the entire model. The optimization goal is to minimize the loss of the entire model. At the same time, the inexact pairing data module IPaD automatically recognizes the characters in the calligraphy data set and records them as identification tags, and then performs inexact pairing in the calligraphy data set based on the target style image, that is, the wrong identification tags are allowed to be used for the relevant calligraphy data set during pairing.

Specifically, in the first group of generative adversarial networks , the original image of the source domain image The contour information cx is fused through the skeleton-contour fusion module SCF. The skeleton-contour fusion module SCF is a kind of cross-attention module. After inputting the skeleton information and contour information of a Chinese character to the skeleton-contour fusion module SCF, the skeleton-contour fusion module SCF first inputs them into the relevant encoder (i.e. Corresponding skeleton encoder and contour encoder) to generate the corresponding skeleton feature E _sx and contour feature E _cx ; then add the encoded skeleton feature E _sx and contour feature E _cx to obtain the feature E _scx and use the SoftMax function to obtain the normalization Characterization c ^Z . Based on the normalized feature c ^Z , the attention weight formula is used to calculate the weight a _c of the corresponding skeleton feature E _sx and the weight b _c of the contour feature E _cx . Finally, the calculated weights a _c and b _c are multiplied by the corresponding skeleton features E _sx and contour features E _cx to obtain the fused weighted skeleton feature E _asx and the fused weighted contour feature E _bcx , where the calculation formula is described as follows :

,/>

Among them, a _c , b _c and c c in ^Z all represent the calculation on channel c , and A and B are matrices of two learnable parameters.

The original image x is input to the generator G _y . During the processing, the generator G _y splices the original image x with the skeleton feature E _asx and the contour feature E _bcx obtained by the skeleton-contour fusion module SCF at the channel level. After processing Generate target style image , and then evaluate the target style image/> through the discriminator D _y authenticity, that is, the target style image/> and the target domain image are respectively input into the discriminator- D _y to determine whether the results returned by the two discriminators- D _y are consistent.

Then in the second set of generative adversarial networks, the target style image Then through the skeleton extraction module Ske and the contour extraction module Con, the corresponding skeleton information is extracted/> and contour information/> , skeleton information/> and contour information/> The two are fused through the skeleton-contour fusion module SCF. Target style image/> Input generator two G _x . During the processing, generator two G _x converts the target style image/> The corresponding skeleton features and corresponding contour features obtained by fusion with the skeleton-contour fusion module SCF are spliced at the channel level, and reconstructed to generate a reconstructed image consistent with the style of the source domain/> . The reconstructed image is then evaluated in the discriminator D _x The authenticity will reconstruct the image/> After inputting the discriminator D _x with the source domain data set X , it is judged whether the results returned by the two discriminators D _x are consistent.

According to the workflow described above, the model losses proposed in this scheme include cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con , imprecise pairing loss L _inex , and two generations of adversarial loss L _advx and L _advy six main components. Among the two generations of adversarial losses, L _advx corresponds to the second generation adversarial loss of generator two G _x and discriminator two D _x , and L _advy corresponds to the first generation adversarial loss of generator one G _y and discriminator one D _y ;The cycle consistency loss L _cyc is the original image x and the reconstructed image of the source domain style loss of time. The above two generations of adversarial loss and cycle consistent loss are the loss functions of the CycleGAN model itself. During training, the model is optimized by minimizing the loss function and the corresponding model training is completed.

Since the solution of this application also extracts the skeleton information and contour information of the image respectively, this model also has contour consistency loss and skeleton consistency loss. The skeleton consistency loss L _ske is the skeleton information sx of the original image x and the reconstructed image Skeleton information extracted from The loss between; the contour consistency loss L _con is the contour information cx of the original image x and the reconstructed image/> Contour information extracted from loss of time. Finally, because this solution uses the inexact pairing data module IPaD for the target domain data set to perform inexact pairing on the calligraphy data set, the loss also includes imprecise pairing loss.

The target style image generated by generator one G _y If an accurate pairing cannot be achieved from the target domain image, an inexact pairing is performed, that is, the wrong identification label is allowed to be used for the relevant calligraphy data set during pairing, thereby obtaining the imprecise pairing data y _inex , at this time the corresponding target style image /> That is, the target style image corresponding to the inaccurate pairing data y _inex /> . The imprecise pairing loss L _inex is the imprecise pairing data y _inex and the target style image corresponding to the imprecise pairing data/> between losses. Among the above losses, cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con and imprecise pairing loss L _inex are all used to optimize generator one G _y and generator two G _x . The calculation formula of the above loss function is as follows:

Among them, E _{x ~ X} [ ] represents the expected value of the data in [ ] under the given source domain image x distribution in the source domain data set X , Represented in a given set of reconstructed images/> Reconstructed image in/> The expected value of the data in [ _] under the distribution _, log D _x ( x ) represents the probability that the discriminator D _x recognizes the source domain image ), the smaller the second-generation adversarial loss. log (1- log D _x (/> )) means that the discriminator D _x will reconstruct the image/> The probability of being recognized as not a source domain image; as the training process optimizes the generator two G _x , the smaller the loss of the generator two G _x , it indicates the reconstructed image/> The smaller the difference in font style between the source image x and the source domain image x , the smaller the difference between log (1- log D _x (/> )) is smaller, the discriminator D _x will reconstruct the image/> The probability of correct identification is also smaller, which results in a larger loss for the discriminator D _x and a smaller second-generation adversarial loss. E _{y ~ Y} [ ] represents the expected value of the data in [ ] under the given target domain image y distribution in the target domain data set Y , Represents a collection of images in a given target style/> Target style image in/> The expected value of the data in [ ] under the distribution, log D _y ( y ) represents the probability that the discriminator D _y will recognize the target domain image y as the target domain image. The smaller the loss of the discriminator D _y , log D _y ( y ), the smaller the first-generation adversarial loss. log (1- log D _y (/> )) means that the discriminator D _y converts the target style image/> The probability of being recognized as an image that is not the target domain; as the training process optimizes the generator G _y , the smaller the loss of the generator G _y , it indicates the target style image/> The smaller the difference in font style between the target domain image y and the target domain image y, the smaller the difference between log (1- log D _y (/> )) is smaller, the discriminator D _y will target style image/> The probability of correct identification is also smaller, which results in a larger loss of the discriminator D _y and a smaller first-generation adversarial loss. /> Represents a source domain image x in a given source domain data set X and a given set of reconstructed images /> Reconstructed image in/> The expected value of the norm of the data in || || ₁ under the distribution, Ske ( x ) and Ske (/> ) respectively represent the source domain image x and reconstructed image /> through the skeleton extraction module Ske Process the results obtained, Con ( x ) and Con (/> ) respectively represent the source domain image x and reconstructed image /> through the contour extraction module Con Processing results;/> Represents reconstructed image/> The set of Y _inex represents the set of inexact paired data y _inex , /> Represents the target style image corresponding to the imprecisely paired data, /> Represents target style images corresponding to imprecisely paired data/> collection of Represents the inexact paired data y _inex in the given set Y _inex and the given set /> Correspondence to target style image of imprecise paired data/> The expected value of the norm of the data in || || ₁ under the distribution.

In the CycleGAN model, the entire model (the model is the model of the calligraphy character generation method based on skeleton, contour and imprecise paired data, the English abbreviation SCI-Font, where S, C, and I correspond to the skeleton extraction module Ske and the contour extraction module in sequence Model loss for Con and Inexact Paired Data module IPaD) , the relationship between the losses of all generators G and the losses of all discriminators D can be described by the following expression:

in, Indicates that the greater the loss of the discriminator D in the model, the smaller the loss of the generator G. The meaning of this expression is to find the value that can be used within the range of the losses of all generators G and the losses of all discriminators D. obtain the minimum value. At this time, the loss of all generators G is the minimum value, and the loss of all discriminators D is the maximum value, and at the same time/> Obtain the optimal solution. Based on this relationship, when training the model, the model loss fed back during training is used to optimize the model and reduce the model loss.

Combined with other loss functions, the model loss of the entire model The calculation formula is as follows:

In this formula, λ _cyc , λ _ske , λ _con , and λ _inex are four possible parameters corresponding to the cycle consistency loss L _cyc , the skeleton consistency loss L _ske , the contour consistency loss L _con , and the inaccurate pairing loss L _inex respectively. The adjusted hyperparameters represent the weight of the corresponding loss in the entire model loss. The hyperparameters are optimized and a set of optimal hyperparameters are selected to improve the performance and effect of learning.

Step 3: Obtain the optimized model for automatic generation of calligraphy fonts.

Based on the above models and training methods, the model of this scheme integrates the skeleton and outline information of Chinese characters, and then uses it as an explicit representation to enhance the latent content style representation produced by the decoder, which can effectively capture the content and style characteristics of calligraphy fonts. Noting the difficulty in collecting paired data, we used some automatic Chinese character recognition technology to generate imprecise paired data sets to further supervise model performance. Inexact paired data can better supervise the relationship between the source domain and the target domain. Glyph differences, although some calligraphy Chinese characters are recognized as errors, they can still provide some important reference information for related calligraphy Chinese characters, which provide important technical support for generating the content of calligraphy characters.

The method provided by the present invention is used to conduct font generation experiments and compared with other existing font generation technologies. The comparison results of the generated fonts are shown in Figure 3. The generation results of different generation methods are arranged from top to bottom. The Chinese characters used Divided into three groups from left to right, each group has four different Chinese characters. From left to right, the calligraphy font of Liu Gongquan is generated from the block font, the calligraphy font of Yan Zhenqing is generated from the block font, and the calligraphy font of Ouyang Xiu is generated from the block font; The circles marked in the figure indicate defective errors that occur when generating fonts. The Chinese characters selected with boxes in the figure indicate that the shape of the generated fonts is inaccurate and mode collapse occurs. The penultimate row adopts the method and model of the present invention (English abbreviation SCI-Font). This figure shows that this method has a better effect on the generation of calligraphy characters. Figure 4 is a comparison of the block script fonts and the calligraphy fonts of Yu Youren and Zhu Suiliang. It can be seen that the strokes and styles of the same Chinese characters vary greatly in different calligraphy fonts, and there are also changes in simplified and traditional fonts, reflecting the complexity of strokes and the variety of styles in calligraphy fonts. Change. Figure 5 is a rendering of applying this method to convert four different groups of Chinese characters in block script fonts into calligraphy fonts of Bada Shanren, Huang Tingjian, Zhu Suiliang and Master Hongyi. Among the four groups of fonts, the first group of "mourning", the second group of Chinese characters "Bing" in the third group, "Shu" in the third group and "Hao" in the fourth group all generate Chinese characters that are different from the input Chinese characters, while the calligraphy font style meets the requirements, thus reflecting the phenomenon of inaccurate matching in this method .

The present invention has been exemplarily described above in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited by the above-mentioned manner, as long as various non-substantive improvements are made using the inventive concepts and technical solutions of the present invention, or without improvement. Direct application of the concepts and technical solutions of the present invention to other situations shall fall within the protection scope of the present invention.

Claims (8)

1. A calligraphy character generation method based on skeleton and outline, including the following steps: Step 1: Establish a model; the model uses the CycleGAN model as the backbone network, and the CycleGAN model includes two sets of generative adversarial networks; Step 2: Train the model; the Chinese character font style input to the model is the source domain style, and the Chinese character image in the source domain style is the source domain image. The Chinese character images in the source domain style are collected as training samples. The calligraphy font image that needs to be generated is The font style is the target style, and the calligraphy font image of the target style is the target domain image. The target domain image is collected to form a calligraphy data set; the source domain image is used as the original image input model during training, and the original image is converted into For the target style image, the target style image output by the first group of generative adversarial networks is converted into a reconstructed image through the second group of generative adversarial networks. The font style of the target style image should be consistent with the target style, and the font style of the reconstructed image should be consistent with the source The domain style is consistent. During the training process, the model is optimized by calculating the loss of the entire model. The optimization goal is to minimize the loss of the entire model; Step 3: Obtain the optimized model for automatic generation of calligraphy fonts; It is characterized in that: both sets of generative adversarial networks include a contour extraction module Con, a skeleton extraction module Ske and a skeleton-contour fusion module SCF, and the model also includes an inexact pairing data module IPaD; In the second step, both groups of generative adversarial networks extract skeleton information and contour information respectively through the contour extraction module Con and the skeleton extraction module Ske, and fuse the skeleton information and contour information through the skeleton-contour fusion module SCF in the generator. The image is spliced with the input generator, and then processed by the corresponding generator to generate the image; The inexact pairing data module IPaD automatically recognizes the characters in the calligraphy data set and records them as identification tags, and then performs inexact pairing in the calligraphy data set based on the target style image. During pairing, it allows the use of wrong identification tags for the relevant calligraphy data sets, thus obtaining Inexact pairing data; The losses of the entire model include the first-generation adversarial loss L _advy , the second-generation adversarial loss L _advx , cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con and imprecise pairing loss L _inex . 2. A method _for generating calligraphy characters based on skeleton and outline according to claim 1, characterized in that: in the step one, the first group of generative adversarial networks includes a constructed generator- Gy and a discriminator- D _y , the second group of generative adversarial networks includes the built generator G _x and discriminator D _x ; the generator G _y is used to convert the original image into a target style image, and the discriminator D _y is used to identify the generated target Whether the font style is consistent between the style image and the target domain image; the second group _of generative adversarial networks uses the opposite process to reconstruct the results output by the first group of generative adversarial networks, that is, the target style image is converted into For the reconstructed image in the source domain style, the discriminator D _x is used to determine whether the font style between the generated reconstructed image and the source domain image is consistent. 3. A method for generating calligraphy characters based on skeleton and outline according to claim 2, characterized in that: in the second step, in the first group of generative adversarial networks, the source domain image x is used as the original image of the input respectively. Through the processing of the skeleton extraction module Ske and the contour extraction module Con, the skeleton information sx and the contour information cx are extracted correspondingly. The skeleton information sx and the contour information cx are fused through the skeleton-contour fusion module SCF; the original image x is input to the generator - G _y , during the processing process, the generator G _y splices the original image x with the skeleton feature E _asx and contour feature E _bcx obtained by the skeleton-contour fusion module SCF at the channel level, and generates the target style image after processing , collect the target domain image y to form the target domain data set Y , and convert the target style image/> and the target domain image y in the target domain data set Y are respectively input into the discriminator D _y to determine whether the results returned by the discriminator D _y are consistent, so as to evaluate the target style image/> authenticity. 4. A method for generating calligraphy characters based on skeleton and contour according to claim 3, characterized in that: after inputting the skeleton information and contour information of a given Chinese character to the skeleton-contour fusion module SCF, the skeleton-contour fusion module SCF first inputs them into the corresponding skeleton encoder and contour encoder to generate the corresponding skeleton feature E _sx and contour feature E _cx ; then the encoded skeleton feature E _sx and contour feature E _cx are added to obtain the feature E _scx and used The SoftMax function obtains the normalized feature c ^Z ; based on the normalized feature c ^Z , use the attention weight formula to calculate the weight a _c of the corresponding skeleton feature E _sx and the weight b _c of the contour feature E _cx ; finally, the calculated The weights a _c and b _c are multiplied by the corresponding skeleton feature E _sx and contour feature E _cx to obtain the skeleton feature E _asx of the fused weight and the contour feature E _bcx of the fused weight. The calculation formula is described as follows: ,/> Among them, a _c , b _c and c c in ^Z all represent the calculation on channel c , and A and B are matrices of two learnable parameters. 5. A method for generating calligraphy characters based on skeleton and outline according to claim 4, characterized in that: in the second group of generative adversarial networks, the target style image Then through the skeleton extraction module Ske and the contour extraction module Con, the corresponding skeleton information is extracted/> and contour information/> , skeleton information/> and contour information/> The two are fused through the skeleton-contour fusion module SCF; target style image/> Input generator two G _x . During the processing, generator two G _x converts the target style image/> The corresponding skeleton features and corresponding contour features obtained by fusion with the skeleton-contour fusion module SCF are spliced at the channel level, and reconstructed to generate a reconstructed image consistent with the style of the source domain/> ;Collect the source domain image x to form the source domain data set X , and reconstruct the image/> After inputting the source domain image x in _the source domain data set X to the discriminator D _x , it is judged whether the results returned by the two discriminators D authenticity. 6. A method for generating calligraphy characters based on skeleton and outline according to claim 5, characterized in that: in the second step, in the first group of generative adversarial networks in the CycleGAN model, the discriminator D _y is calculated. The difference in font style between the target style image and the target domain image, that is, the first generation adversarial loss L _advy , is used to optimize the generator one G _y ; the input of the second set of generative adversarial networks is based on the first set of generative adversarial The output of the generator G _y in the network, the discriminator D _x in the second group of generative adversarial networks calculates the difference in font style between the source domain image and the reconstructed image, that is, the second generation adversarial loss L _advx , using In optimizing generator two G _x ; Cycle consistency loss L _cyc , skeleton consistency loss L _ske , contour consistency loss L _con , and imprecise pairing loss L _inex all correspond to the optimized generator two G _x and generator one G _y ; the cycle consistency loss L _cyc is Original image x and reconstructed image in source domain style The loss between; the skeleton consistency loss L _ske is the skeleton information sx of the original image x and the reconstructed image/> Skeleton information extracted from The loss between; the contour consistency loss L _con is the contour information cx of the original image x and the reconstructed image/> Contour information extracted from The loss between, the imprecise pairing loss L _inex is the imprecise pairing data y _inex and the target style image corresponding to the imprecise pairing data/> between losses. 7. A calligraphy character generation method based on skeleton and outline according to claim 6, characterized in that: the second generation adversarial loss L _advx , the first generation adversarial loss L _advy , the cycle consistency loss L _cyc , The calculation formulas of skeleton consistency loss L _ske , contour consistency loss L _con , and inaccurate pairing loss L _inex are as follows: Among them, E _{x ~ X} [ ] represents the expected value of the data in [ ] under the given source domain image x distribution in the source domain data set X , Represented in a given set of reconstructed images/> Reconstructed image in/> The expected value of the data in [ ] under the distribution, log D _x ( x ) represents the probability that the discriminator D _x recognizes the source domain image x as the source domain image, log (1- log D _x (/> )) means that the discriminator D _x will reconstruct the image/> The probability of being recognized as not being a source domain image; E _{y ~ Y} [ ] represents the expected value of the data in [ ] under the given target domain image y distribution in the target domain data set Y , /> Represents a collection of images in a given target style/> Target style image in/> The expected value of the data in [ ] under the distribution, log D _y ( y ) represents the probability that the discriminator D _y recognizes the target domain image y as the target domain image, log (1- log D _y (/> )) means that the discriminator D _y converts the target style image/> The probability of being recognized as an image that is not the target domain;/> Represents a source domain image x in a given source domain data set X and a given set of reconstructed images /> Reconstructed image in/> The expected value of the norm of the data in || || ₁ under the distribution, Ske ( x ) and Ske (/> ) respectively represent the source domain image x and reconstructed image /> through the skeleton extraction module Ske Process the results obtained, Con ( x ) and Con (/> ) respectively represent the source domain image x and reconstructed image /> through the contour extraction module Con Processing results;/> Represents reconstructed image/> The set of Y _inex represents the set of inexact paired data y _inex , /> Represents the target style image corresponding to the imprecisely paired data, /> Represents target style images corresponding to imprecisely paired data/> collection of Represents the inexact paired data y _inex in the given set Y _inex and the given set /> Correspondence to target style image of imprecise paired data/> The expected value of the norm of the data in || || ₁ under the distribution. 8. A method for generating calligraphy characters based on skeleton and outline according to claim 7, characterized in that: Model loss for the entire model The calculation formula is as follows: In this formula, λ _cyc , λ _ske , λ _con , and λ _inex are four possible parameters corresponding to the cycle consistency loss L _cyc , the skeleton consistency loss L _ske , the contour consistency loss L _con , and the inaccurate pairing loss L _inex respectively. The tuned hyperparameter represents the weight of the corresponding loss in the entire model loss.