CN116091363A - Handwriting Chinese character image restoration method and system - Google Patents
Handwriting Chinese character image restoration method and system Download PDFInfo
- Publication number
- CN116091363A CN116091363A CN202310343214.8A CN202310343214A CN116091363A CN 116091363 A CN116091363 A CN 116091363A CN 202310343214 A CN202310343214 A CN 202310343214A CN 116091363 A CN116091363 A CN 116091363A
- Authority
- CN
- China
- Prior art keywords
- chinese character
- image
- network
- skeleton
- character image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000012545 processing Methods 0.000 claims abstract description 18
- 238000005070 sampling Methods 0.000 claims abstract description 7
- 238000011084 recovery Methods 0.000 claims description 31
- 238000012549 training Methods 0.000 claims description 21
- 230000007246 mechanism Effects 0.000 claims description 9
- 230000008485 antagonism Effects 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 4
- 230000001186 cumulative effect Effects 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 2
- 239000000284 extract Substances 0.000 abstract description 3
- 239000000463 material Substances 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 21
- 230000008569 process Effects 0.000 description 13
- 230000008439 repair process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 239000004973 liquid crystal related substance Substances 0.000 description 6
- 230000004913 activation Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000011176 pooling Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012850 discrimination method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Character Discrimination (AREA)
Abstract
The invention discloses a method and a system for restoring a calligraphy Chinese character image, wherein the method comprises the following steps: obtaining a damaged Chinese character image, inputting a pre-trained Chinese character image restoration network, and outputting a restored Chinese character image; the Chinese character image restoration network comprises a Chinese character restoration network, a Chinese character structure discrimination network and a skeleton matching network, wherein the Chinese character restoration network is used for acquiring depth characteristics of a damaged Chinese character image and carrying out up-sampling restoration to obtain an image with the same input size; the Chinese character structure judging network is used for judging whether the context consistency of the Chinese character restored by the Chinese character restoring network and the undamaged Chinese character image is maintained on the global content; the skeleton matching network performs skeleton feature comparison on the repaired Chinese character and undamaged Chinese character images, so that the structural consistency of the Chinese character repairing network is realized. The invention uses the calligraphy Chinese character restoration method combining the Chinese character skeleton characteristics to automatically extract the characteristics of Chinese character images, and human intervention processing is not needed, so that manpower and material resources are saved.
Description
Technical Field
The invention relates to a method and a system for restoring a handwriting Chinese character image, and belongs to the technical field of image processing.
Background
Chinese calligraphy works are valuable wealth of human civilization, but partial contents are damaged due to natural or artificial factors in inheritance, and the repairing work is always a difficult problem in the field of cultural heritage repairing due to different contents abstraction and styles. At present, repair is mainly finished manually by professionals, and the professionals can complete reasoning according to pixels around the Chinese character image of the handwriting and information provided by the semantics of the handwriting, and then finish repair by using image processing software and the like, so that the problems of low repair efficiency, large quality difference, high recognition difficulty and the like exist, and inheritance of precious handwriting is hindered. The task of restoring the image of the calligraphy Chinese characters can be regarded as a branch of restoring the image, in particular to filling and restoring unknown parts by utilizing the information of known positions, restoring the structure, the texture characteristics and the like of the damaged image as much as possible, ensuring that the restored calligraphy Chinese characters keep the consistent writing style with the original Chinese characters and can be identified.
With the development of deep learning based image restoration, some researchers have tried to complete a kanji restoration task using existing image restoration methods, for example, some researchers have tried to restore damaged characters using image restoration methods. For example, the prior art proposes a method for recovering blocked Chinese characters based on a deep generation countermeasure network, which can recover partially missing Chinese characters to a certain extent, and a method for recovering Yi-nationality handwriting Chinese characters based on a double discrimination method, which can recover Yi-nationality Chinese characters to a certain extent. However, although the method improves the Chinese character image restoration efficiency, the restoration result is not satisfactory, and the method has two problems in comprehensive analysis, namely, firstly, the image method is mainly applied to natural landscape images, mainly ensures visual perception, has little research applied to character images, the calligraphic Chinese characters are images combined by different strokes according to a certain layout mode, the image restoration method is directly applied to easily ignore the structural information of the Chinese characters, the restoration result is misled, and a plurality of character images which look like Chinese characters without any meaning in practice are generated; and then, along with the increase of the shielding area, effective information which can be utilized by the calligraphic Chinese character image is reduced in a large scale, so that the repair results of the methods can generate certain blurring.
Disclosure of Invention
The invention aims to provide a method and a system for restoring a calligraphic Chinese character image, which are used for extracting a skeleton from the restored calligraphic Chinese character image as a structural feature to guide the restoration process of the calligraphic Chinese character image, and solve the problems that the restoration effect is affected due to large shielding area and incapability of effectively utilizing Chinese character structural information in the existing restoration method.
A method for restoring a calligraphy chinese character image, the method comprising:
obtaining a damaged Chinese character image, inputting a pre-trained Chinese character image restoration network, and outputting a restored Chinese character image;
the Chinese character image restoration network comprises a Chinese character restoration network, a Chinese character structure discrimination network and a skeleton matching network, wherein the Chinese character restoration network is used for acquiring depth characteristics of a damaged Chinese character image, carrying out up-sampling restoration to obtain an image with the same size as the input damaged Chinese character image, and completing Chinese character restoration;
the Chinese character structure judging network is used for judging the context consistency of the Chinese character restored by the Chinese character restoring network and the undamaged Chinese character image on the global content;
the skeleton matching network is used for comparing skeleton characteristics of the repaired Chinese characters and undamaged Chinese character images and is used for realizing structural consistency of the Chinese character repairing network.
Further, the Chinese character recovery network comprises an encoding module and a decoding module;
the coding module comprises four convolution layers, wherein each layer uses 4×4 convolution to perform downsampling on an input image to obtain depth characteristics;
the decoding module comprises four transposed convolutional layers, and the 4×4 transposed convolutional layers are used for upsampling and restoring the obtained depth features into an image consistent with the input size.
Further, the encoding module also integrates a layer of attention mechanisms of the feature space for filling the damaged area with information of the undamaged area.
Further, the Chinese character structure distinguishing network consists of a 4×4 convolution layer, the input of the Chinese character structure distinguishing network is a repaired image and an undamaged Chinese character image, and a vector is output after convolution operation.
Further, the training method of the Chinese character image restoration network comprises the following steps:
collecting a handwriting Chinese character image as an original image, and performing shielding treatment to construct a shielding handwriting Chinese character image data set;
inputting the blocked calligraphic Chinese character image into a Chinese character recovery network, extracting the characteristics of the input image through an encoding module, focusing on a non-blocked area by using an attention mechanism layer, acquiring the information of the non-blocked area, repairing the blocked area, and restoring the image to the original image size by using a transposition convolution layer to finish repairing and outputting;
respectively calculating pixel reconstruction loss and antagonism loss aiming at the repaired calligraphic Chinese character image and the original image which is not subjected to shielding treatment, respectively extracting frameworks for the repaired calligraphic Chinese character image and the original image which is not subjected to shielding treatment, comparing the framework level characteristics, and calculating framework matching loss;
and jointly training a Chinese character image restoration network by using pixel reconstruction loss, antagonism loss and skeleton matching loss.
Further, the method for performing occlusion processing includes:
an area of 50×50 pixels in size on the original image is selected, and its gray value is set to 0.
Further, the method for extracting the skeleton comprises the following steps:
performing binarization processing on the image;
and judging the condition 1 of all foreground pixel points of the image subjected to binarization processing, deleting the pixel points meeting the condition 1, wherein the condition 1 is as follows:
wherein 0 represents a background pixel point, 1 represents a foreground pixel point, N (P1) represents the number of foreground pixel points in eight adjacent areas of the foreground pixel point P1, the method comprises the steps of
The eight neighborhood of (1) is expressed as P2-P9 clockwise, S (P1) represents the cumulative number of times that two adjacent pixels appearing from the pixels in the direction of P2-P9-P2 are sequentially 0 and 1,/>
Representing a multiplication operation;
then, judging a deleting condition 2 for all foreground pixel points in the image, and deleting the pixel points meeting the deleting condition 2, wherein the deleting condition 2 is as follows:
and extracting the skeleton of the Chinese character of the calligraphy until no pixel points need to be deleted.
Further, the training method of the skeleton matching network comprises the following steps:
performing skeleton extraction processing on the complete Chinese character data set to obtain a Chinese character skeleton data set;
inputting the Chinese character skeleton data set into a skeleton recognition network to be trained, optimizing network parameters by using a multi-classification cross entropy loss function, and training the skeleton recognition network;
and removing the full connection layer in the trained skeleton recognition network, fixing parameters of the network, and using the convolution layer as the trained skeleton matching network.
Further, the multi-class cross entropy loss function expression is as follows:
wherein:
for a multi-class cross entropy loss function, K is the number of classes, y is the label, i is the class, p is the output of the network, and represents the probability that the class is i.
A calligraphy chinese character image restoration system, the system comprising:
the acquisition module is used for acquiring the damaged Chinese character image;
the processing module is used for repairing the damaged Chinese character image and outputting the repaired Chinese character image;
the processing module comprises a Chinese character recovery network unit, a Chinese character structure distinguishing network unit and a skeleton matching network unit;
the Chinese character restoring network unit is used for acquiring depth characteristics of the damaged Chinese character image, carrying out up-sampling and restoring to an image with the same size as the input damaged Chinese character image, and carrying out Chinese character restoration;
the Chinese character structure judging network unit is used for judging the context consistency of the Chinese character restored by the Chinese character restoring network unit and the undamaged Chinese character image on the global content;
the skeleton matching network unit is used for comparing skeleton characteristics of the repaired Chinese characters and undamaged Chinese character images, and achieving structural consistency of a Chinese character repairing network.
Compared with the prior art, the invention has the beneficial effects that: the invention uses the Chinese character skeleton and the handwriting Chinese character restoration method combined with the generation countermeasure network technology, can automatically extract the characteristics of Chinese character images, does not need human intervention to process the handwriting Chinese character image restoration process, saves manpower and material resources, and compared with the image restoration method based on deep learning, the invention maximizes the use of the information of the missing area through the attention mechanism, extracts the Chinese character skeleton as the Chinese character structural characteristics, guides the Chinese character restoration process, ensures that the restored handwriting Chinese character images are consistent with the original images, and improves the handwriting Chinese character restoration effect.
Drawings
FIG. 1 is an overall flow chart of a method for restoring a calligraphy Chinese character image;
FIG. 2 is a diagram of a training process of a Chinese character image restoration network;
FIG. 3 is a main body network structure diagram of a Chinese character image restoration network;
FIG. 4 is a true undamaged Chinese character image;
FIG. 5 is a Chinese character image with added occlusion;
FIG. 6 is a skeleton diagram of extracted Chinese characters;
FIG. 7 is a restored Chinese character image;
fig. 8 is an eight-neighborhood example of the pixel point P1.
Detailed Description
The invention is further described in connection with the following detailed description, in order to make the technical means, the creation characteristics, the achievement of the purpose and the effect of the invention easy to understand.
The invention discloses a method for restoring a calligraphy Chinese character image, which has the overall flow chart shown in figure 1 and comprises the following steps:
obtaining a damaged Chinese character image, inputting a pre-trained Chinese character image restoration network, and outputting a restored Chinese character image;
the Chinese character image restoration network comprises a Chinese character restoration network, a Chinese character structure discrimination network and a skeleton matching network, wherein the Chinese character restoration network is used for acquiring depth characteristics of a damaged Chinese character image, carrying out up-sampling restoration to obtain an image with the same size as the input damaged Chinese character image, and completing Chinese character restoration;
the Chinese character structure judging network is used for judging the context consistency of the Chinese character restored by the Chinese character restoring network and the undamaged Chinese character image on the global content;
the skeleton matching network is used for comparing skeleton characteristics of the repaired Chinese characters and undamaged Chinese character images and is used for realizing structural consistency of the Chinese character repairing network.
Step 1: an image preprocessing step:
the repairing method needs to use a damaged Chinese character image dataset and a complete Chinese character skeleton image dataset, but the corresponding datasets are not well collected, so that a set of regular script works written by a calligrapher are collected by considering the synthesized datasets, but in the transmission process of the images, the image quality is reduced due to noise interference or data loss caused by various reasons, so that the images need to be subjected to certain enhancement treatment to reduce the influence, firstly, the Chinese character images are cut to be 256 multiplied by 256 in uniform size, and then the images are enhanced by using median filtering. Providing a better input image for the subsequent repair network.
Step 2: constructing a shielding handwriting Chinese character image data set as a damaged Chinese character image data set, and constructing a complete Chinese character skeleton image data set:
(1) And adding a random mask to the Chinese character image data set subjected to the preprocessing step, so as to construct an occlusion Chinese character image data set. The addition process of the random mask is as follows: randomly selecting an area with the size of 50 multiplied by 50 pixels on an original image, setting the gray value of the area to be 0, and obtaining an image data set of the blocked calligraphy Chinese characters, wherein fig. 5 is a diagram of adding blocked calligraphy Chinese characters;
(2) Considering that undamaged Chinese character skeletons are required to be used as structural features for guiding the Chinese character restoration process in the following steps, the method uses a traditional refinement algorithm to carry out skeleton extraction on a complete Chinese character image dataset which is not subjected to shielding treatment, thereby constructing a complete Chinese character skeleton image dataset, namely the undamaged Chinese character skeleton image dataset, and comprises the following steps:
2.1, carrying out binarization processing on a complete calligraphy Chinese character image, dividing the gray value of each pixel point of the original image into 0 and 1, then dividing the two types of pixel points, and dividing the gray value of 0 into a background pixel point of the calligraphy Chinese character image and a foreground pixel point of the calligraphy Chinese character image, wherein the gray value of 1 is the background pixel point of the calligraphy Chinese character image.
2.2, judging the condition 1 of all foreground pixels of the handwriting Chinese character image subjected to binarization, deleting the pixels meeting the condition 1, wherein the condition 1 is as follows:
wherein 0 represents a background pixel point, 1 represents a foreground pixel point, N (P1) represents the number of foreground pixel points in eight adjacent areas of the foreground pixel point P1, the method comprises the steps of
The eight neighborhoods are represented as P2 to P9 clockwise as shown in FIG. 8, S (P1) represents the cumulative number of times that two adjacent pixel points appearing from the pixels in the P2 to P9 to P2 directions are 0 and 1 in order,/-, respectively>
Representing a multiplication operation;
2.3, judging the deleting condition 2 of all foreground pixel points in the image, and deleting the pixel points meeting the deleting condition 2, wherein the deleting condition 2 is as follows:
until no pixel point is deleted, the output image is a complete Chinese character skeleton image, and as shown in fig. 6, the complete Chinese character skeleton image is used as structural information of corresponding Chinese characters to guide the subsequent Chinese character image restoration network training process, so that the restoration result is ensured to keep good structural consistency with the original image.
Step 3: building a Chinese character image restoration network:
the Chinese character image restoration network integrally adopts a structure for generating an countermeasure network, but a skeleton matching network for comparing skeleton characteristics is additionally added. As shown in fig. 3, the kanji recovery network adopts a coding-decoding module structure, which includes a coding module and a decoding module: and a coding module: there are four 4x4 convolutional layers in total, and downsampling is performed on the input image to obtain depth features. The convolution layer may improve the ability to obtain deep semantic information. The fourth layer of the coding module is integrated with an attention mechanism layer of the feature space, and the attention mechanism layer is used for filling the damaged area by maximally utilizing the information of the undamaged area; and the decoding module uses four 4 multiplied by 4 transposed convolution layers to up-sample and restore the obtained depth characteristics into an image with the same size as the input damaged Chinese character image.
Chinese character structure discrimination network: the Chinese character structure distinguishing network consists of 4x4 convolution layers, and the input is repaired image and undamaged Chinese character skeleton image, and a vector is output after a series of convolution operations; the Chinese character structure distinguishing network is used for ensuring the context consistency of the repaired image and the global content of the undamaged Chinese character skeleton image.
Skeleton matching network: and extracting the characteristics of the restored Chinese character skeleton and undamaged Chinese character skeleton images, and improving the capability of maintaining structural consistency of a Chinese character restoration network by comparing the characteristic losses of different layers.
Step 4: building and training a skeleton matching network:
(1) Firstly, a network for identifying Chinese character skeletons is built, the network model consists of a 3X 3 convolution layer, a 2X 2 pooling layer and three full-connection layers, and the training method of the skeleton identification network comprises the following steps:
performing skeleton extraction processing on the complete Chinese character data set to obtain a Chinese character skeleton data set; inputting the Chinese character skeleton data set into a skeleton recognition network to be trained, optimizing network parameters by using a multi-classification cross entropy loss function, and training the skeleton recognition network;
and removing the full connection layer in the trained skeleton recognition network, freezing parameters of the network, and using the convolution layer as the trained skeleton matching network.
Wherein the multi-class cross entropy loss function expression is as follows:
wherein:
for a multi-class cross entropy loss function, K is the number of classes, y is the label, i is the class, p is the output of the network, and represents the probability that the class is i.
Step 5: describing a modeling process:
(1) For completely describing the training process, the invention sets the input undamaged Chinese character image as
A binary mask image (defect position 0, rest 1) representing the defect position information by M, the defect image can be expressed as:
wherein the method comprises the steps of
Representing bitwise multiplication;
(2) The damaged image is input into a Chinese character recovery network, and an input characteristic diagram is obtained through the convolution layers of the first three layers, wherein the input characteristic diagram comprises the following formula:
wherein c=1, 2,3 … n, n is the number of layers of the convolution layer, C represents the number of channels of the feature map, H and W represent the size of the feature map, conv is vector convolution operation;
then entering a fourth layer of attention mechanism for adding feature space, wherein the attention layer is helpful for maximizing utilization of effective information of undamaged areas, and enabling a Chinese character recovery network to automatically focus on areas which are on feature graphs and are helpful for repairing tasks, and carrying out maximum pooling and average pooling operations on the feature graphs along channel dimensions to obtain
And->
(where H and W are as defined in equation 2 above) and then used for 4x4 standard convolution operations and sigmoid activation to get the output of the feature space attention mechanism:
wherein the method comprises the steps of
Representing the calculation of the maximum value on the feature map along the channel dimension,/->
Representing the calculation of the mean value in the feature map along the channel dimension,/-, for example>
Representing a standard 4x4 convolution operation, sigmoid represents an activation function, representing an operation of y mapping the degree of interest at each location to a value (0, 1), as represented by:
finally multiplying the original input feature image with a feature space attention layer to obtain an output feature image;
(3) Carrying out convolution operation on the feature map in the formula again, and restoring the image size by transpose convolution to obtain a final output:
step 6: definition of the loss function:
the functions of the Chinese character repair network are constrained by different loss functions, and parameters of the Chinese character repair network are optimized through the back propagation process of the loss functions, so that the repair effect can be optimized.
(1) Firstly, setting the input of Chinese character recovery network as
The undamaged Chinese character image is +.>
The output of the Chinese character recovery network is +.>
The function of the Chinese character recovery network is to extract and repair the complete Chinese character image from the damaged Chinese character image, so that the mean square error loss is used to ensure the pixel consistency of the repaired Chinese character image and the undamaged Chinese character image at each position, and the loss function is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
representing undamaged Chinese character image->
Representing the restored Chinese character image output by the Chinese character restoring network, n is the pixel number of the image, namely the product of the width and the height of the image,>
the sum of absolute value squares is carried out on all pixels in the image, and the difference value is calculated, namely the mean square error loss, so that the pixel consistency of the repaired Chinese character image and the undamaged Chinese character image is ensured.
(2) In addition, in order to ensure the consistency of the repaired Chinese character image and the undamaged Chinese character image in the font structure, the invention aims at the Chinese character image which passes through the Chinese character repairing network
Extracting a framework S by using a refinement algorithm in the step 2
) Then the corresponding undamaged Chinese character image in the step 1 is marked as S (++>
) The skeleton matching network is used for extracting the feature map and calculating the loss of skeleton level features, and the feature map is used for ensuring the high consistency of the repaired Chinese character image and the undamaged Chinese character image on the font structure, so that the Chinese character recovery network can synthesize more meaningful high-frequency details:
wherein, the liquid crystal display device comprises a liquid crystal display device,
representing skeleton match loss, < >>
Representing skeleton matching network input as +.>
Time->
Layer activation layer, the parameters of skeleton matching network in the whole process are kept unchanged +.>
Representing the size of the resulting feature map, S (+)>
) Representing the skeleton extracted from the repaired Chinese characters, S (/ -)>
) An image representing intact Chinese characters.
(3) Following the generation of countermeasuresTraining ideas, the invention needs undamaged Chinese character images
And Chinese character image repaired by Chinese character recovery network>
Respectively inputting the image feature information into a Chinese character structure discrimination network, extracting image feature information through a 3-layer convolution layer, and finally outputting the probability of true image through a full connection layer, wherein the label of the undamaged image is required to be set in advance>
Is true.
Wherein true means that the Chinese character structure distinguishing network scores the input image, the invention defines the highest score of 1 and the lowest score of 0, and the probability that the input image is close to the real image is represented.
The invention uses the Chinese character recovery network as G, the Chinese character structure discrimination network as D and the undamaged Chinese character image as
The restored Chinese character image is +.>
D () is the probability that the chinese character image is true.
Then the loss function of the kanji structure discrimination network D is:
the loss of the Chinese character structure distinguishing network consists of scoring the difference between the true undamaged image and the expected score and scoring the difference between the restored image and the expected score, and the Chinese character structure distinguishing network is used for distinguishing the restored image and the true undamaged image as far as possible, namely D #
) Approach to1,/>
Approaching 0;
loss function of kanji recovery network G:
the loss of the Chinese character recovery network is basically the difference between the probability distribution of the repaired Chinese character image and the undamaged Chinese character image, namely the difference between the expected Chinese character structure distinguishing network of the Chinese character recovery network and the actual Chinese character structure distinguishing network scoring of the recovered Chinese character image is to ensure that the repaired Chinese character image approaches the undamaged Chinese character image, namely the requirement is satisfied
Approach 1, at this time->
Trend towards 0;
the loss function of the Chinese character recovery network and the Chinese character structure discrimination network is used for obtaining the total counterloss function as follows:
wherein the method comprises the steps of
Indicate desire->
Is from->
Acquired in distribution->
Representing the distribution of real undamaged Chinese character image data, < >>
Representing the probability of outputting the image as true after inputting the Chinese character image into the discriminator (the undamaged Chinese character image needs to be set as the true label at the moment),/the method comprises the following steps>
Is a real undamaged Chinese character image;
indicate desire->
Is from->
G (x) represents the restored Chinese character image, ">
Is the data distribution of the repaired Chinese character image.
(4) In summary, the overall objective function of the kanji recovery network is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
representing different coefficients for each loss function, the corresponding coefficients may be set by a particular experiment.
Represents mean square error loss, < >>
Representing skeleton feature matching loss, < >>
Representing a loss of antagonism. />
The training strategy of step 6 is as follows:
(1) Firstly, randomly initializing network parameters of a Chinese character recovery network to obtain a data set with false labels as output of the Chinese character recovery network, and taking a real undamaged image as a data set with true labels;
(2) Fixing parameters of a Chinese character recovery network, training the Chinese character structure discrimination network, wherein the training targets are as follows: the Chinese character structure judging network can judge whether the Chinese character structure judging network belongs to a real sample or a false sample generated by a Chinese character recovering network based on a given sample;
(3) Fixing parameters of a Chinese character structure discrimination network, connecting the parameters with the Chinese character recovery network in series, judging true and false by the Chinese character structure discrimination network on one hand, extracting skeleton characteristics by a refinement algorithm on the other hand, calculating skeleton matching loss with the skeleton of the corresponding Chinese character in the real undamaged sample by a formula (8), calculating an integral objective function by a formula (12), and updating the parameters of the Chinese character recovery network;
(4) Then, generating a new sample by using the parameters of the new Chinese character recovery network in the step (3) to optimize the Chinese character structure discrimination network again;
(5) The above process is repeated continuously to reach the maximum training times or the convergence of the loss function, and the model of the Chinese character recovery network with the best effect can be obtained.
Step 7: test model:
initializing a model by using the model parameters of the optimal Chinese character repairing network obtained after training, extracting partial images from the complete Chinese character data set, adding random shielding, and then testing the repairing effect of the Chinese character repairing network model.
Based on the same inventive concept, the invention also discloses a calligraphy Chinese character image restoration system, which comprises:
the acquisition module is used for acquiring the damaged Chinese character image;
the processing module is used for repairing the damaged Chinese character image and outputting the repaired Chinese character image;
the processing module comprises a Chinese character recovery network unit, a Chinese character structure distinguishing network unit and a skeleton matching network unit;
the Chinese character recovery network unit is used for acquiring depth characteristics of the damaged Chinese character image, and performing up-sampling and recovery to obtain an image with the same input size;
the Chinese character structure judging network unit is used for judging the context consistency of the Chinese character restored by the Chinese character restoring network unit and the undamaged Chinese character image on the global content;
the skeleton matching network unit is used for comparing skeleton characteristics of the repaired Chinese characters with those of undamaged Chinese character images, so that structural consistency of the Chinese character repairing network is realized.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.
Claims (10)
1. A method for restoring a calligraphy Chinese character image is characterized by comprising the following steps:
obtaining a damaged Chinese character image, inputting a pre-trained Chinese character image restoration network, and outputting a restored Chinese character image;
the Chinese character image restoration network comprises a Chinese character restoration network, a Chinese character structure discrimination network and a skeleton matching network, wherein the Chinese character restoration network is used for acquiring depth characteristics of a damaged Chinese character image, carrying out up-sampling restoration to obtain an image with the same size as the input damaged Chinese character image, and completing Chinese character restoration;
the Chinese character structure judging network is used for judging the context consistency of the Chinese character restored by the Chinese character restoring network and the undamaged Chinese character image on the global content;
the skeleton matching network is used for comparing skeleton characteristics of the repaired Chinese characters and undamaged Chinese character images and is used for realizing structural consistency of the Chinese character repairing network.
2. The method for restoring a calligraphy kanji image according to claim 1, wherein said kanji restoring network comprises an encoding module and a decoding module;
the coding module comprises four convolution layers, wherein each layer uses 4×4 convolution to perform downsampling on an input image to obtain depth characteristics;
the decoding module comprises four transposed convolutional layers, and the 4×4 transposed convolutional layers are used for upsampling and restoring the obtained depth features into an image consistent with the input size.
3. The method of claim 2, wherein the coding module further integrates an attention mechanism layer of a feature space for filling a damaged area with information of an undamaged area.
4. The method for restoring a chinese character image according to claim 1, wherein said chinese character structure discrimination network is composed of a 4x4 convolutional layer, and the input is a restored image and an undamaged chinese character image, and a vector is outputted after a convolutional operation.
5. The method for restoring a calligraphy kanji image according to claim 3, wherein the training method for restoring a kanji image comprises:
collecting a handwriting Chinese character image as an original image, and performing shielding treatment to construct a shielding handwriting Chinese character image data set;
inputting the blocked calligraphic Chinese character image into a Chinese character recovery network, extracting the characteristics of the input image through an encoding module, focusing on a non-blocked area by using an attention mechanism layer, acquiring the information of the non-blocked area, repairing the blocked area, and restoring the image to the original image size by using a transposition convolution layer to finish repairing and outputting;
respectively calculating pixel reconstruction loss and antagonism loss aiming at the repaired calligraphic Chinese character image and the original image which is not subjected to shielding treatment, respectively extracting frameworks for the repaired calligraphic Chinese character image and the original image which is not subjected to shielding treatment, comparing the framework level characteristics, and calculating framework matching loss;
and jointly training a Chinese character image restoration network by using pixel reconstruction loss, antagonism loss and skeleton matching loss.
6. The method for restoring the calligraphy kanji image according to claim 5, wherein said method for shielding processing comprises:
an area of 50×50 pixels in size on the original image is selected, and its gray value is set to 0.
7. The method for restoring a calligraphy kanji image according to claim 5, wherein said method for extracting skeleton comprises:
performing binarization processing on the image;
and judging the condition 1 of all foreground pixel points of the image subjected to binarization processing, deleting the pixel points meeting the condition 1, wherein the condition 1 is as follows:
wherein 0 represents a background pixel point, 1 represents a foreground pixel point, N (P1) represents the number of foreground pixel points in eight adjacent areas of the foreground pixel point P1, the method comprises the steps of
The eight neighborhood of (1) is expressed as P2-P9 clockwise, S (P1) represents the cumulative number of times that two adjacent pixels appearing from the pixels in the direction of P2-P9-P2 are sequentially 0 and 1,/>
Representing a multiplication operation;
then, judging a deleting condition 2 for all foreground pixel points in the image, and deleting the pixel points meeting the deleting condition 2, wherein the deleting condition 2 is as follows:
and extracting the skeleton of the Chinese character of the calligraphy until no pixel points need to be deleted.
8. The method for restoring a calligraphy kanji image according to claim 1, wherein the training method of the skeleton matching network comprises:
performing skeleton extraction processing on the complete Chinese character data set to obtain a Chinese character skeleton data set;
inputting the Chinese character skeleton data set into a skeleton recognition network to be trained, optimizing network parameters by using a multi-classification cross entropy loss function, and training the skeleton recognition network;
and removing the full connection layer in the trained skeleton recognition network, fixing parameters of the network, and using the convolution layer as the trained skeleton matching network.
9. The method for restoring a calligraphy kanji image according to claim 8, wherein said multi-classification cross entropy loss function expression is as follows:
wherein:
for a multi-class cross entropy loss function, K is the number of classes, y is the label, i is the class, p is the output of the network, and represents the probability that the class is i.
10. A calligraphy kanji image restoration system, the system comprising:
the acquisition module is used for acquiring the damaged Chinese character image;
the processing module is used for repairing the damaged Chinese character image and outputting the repaired Chinese character image;
the processing module comprises a Chinese character recovery network unit, a Chinese character structure distinguishing network unit and a skeleton matching network unit;
the Chinese character restoring network unit is used for acquiring depth characteristics of the damaged Chinese character image, carrying out up-sampling and restoring to an image with the same size as the input damaged Chinese character image, and carrying out Chinese character restoration;
the Chinese character structure judging network unit is used for judging the context consistency of the Chinese character restored by the Chinese character restoring network unit and the undamaged Chinese character image on the global content;
the skeleton matching network unit is used for comparing skeleton characteristics of the repaired Chinese characters and undamaged Chinese character images, and achieving structural consistency of a Chinese character repairing network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310343214.8A CN116091363A (en) | 2023-04-03 | 2023-04-03 | Handwriting Chinese character image restoration method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310343214.8A CN116091363A (en) | 2023-04-03 | 2023-04-03 | Handwriting Chinese character image restoration method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116091363A true CN116091363A (en) | 2023-05-09 |
Family
ID=86187096
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310343214.8A Pending CN116091363A (en) | 2023-04-03 | 2023-04-03 | Handwriting Chinese character image restoration method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116091363A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108898639A (en) * | 2018-05-30 | 2018-11-27 | 湖北工业大学 | A kind of Image Description Methods and system |
| CN109389091A (en) * | 2018-10-22 | 2019-02-26 | 重庆邮电大学 | The character identification system and method combined based on neural network and attention mechanism |
| CN110210485A (en) * | 2019-05-13 | 2019-09-06 | 常熟理工学院 | The image, semantic dividing method of Fusion Features is instructed based on attention mechanism |
| CN111062290A (en) * | 2019-12-10 | 2020-04-24 | 西北大学 | Method and device for constructing Chinese calligraphy style conversion model based on generation confrontation network |
| CN113362255A (en) * | 2021-07-01 | 2021-09-07 | 清华大学深圳国际研究生院 | Character image restoration method and system based on improved DCGAN |
| CN113505772A (en) * | 2021-06-23 | 2021-10-15 | 北京华创智芯科技有限公司 | License plate image generation method and system based on generation countermeasure network |
| CN113920516A (en) * | 2021-12-08 | 2022-01-11 | 南京信息工程大学 | Calligraphy character skeleton matching method and system based on twin neural network |
| CN114742714A (en) * | 2021-10-29 | 2022-07-12 | 天津大学 | Chinese character image restoration algorithm based on skeleton extraction and antagonistic learning |
| CN115170403A (en) * | 2022-05-23 | 2022-10-11 | 广州大学 | Font repairing method and system based on deep meta learning and generation countermeasure network |
-
2023
- 2023-04-03 CN CN202310343214.8A patent/CN116091363A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108898639A (en) * | 2018-05-30 | 2018-11-27 | 湖北工业大学 | A kind of Image Description Methods and system |
| CN109389091A (en) * | 2018-10-22 | 2019-02-26 | 重庆邮电大学 | The character identification system and method combined based on neural network and attention mechanism |
| CN110210485A (en) * | 2019-05-13 | 2019-09-06 | 常熟理工学院 | The image, semantic dividing method of Fusion Features is instructed based on attention mechanism |
| CN111062290A (en) * | 2019-12-10 | 2020-04-24 | 西北大学 | Method and device for constructing Chinese calligraphy style conversion model based on generation confrontation network |
| CN113505772A (en) * | 2021-06-23 | 2021-10-15 | 北京华创智芯科技有限公司 | License plate image generation method and system based on generation countermeasure network |
| CN113362255A (en) * | 2021-07-01 | 2021-09-07 | 清华大学深圳国际研究生院 | Character image restoration method and system based on improved DCGAN |
| CN114742714A (en) * | 2021-10-29 | 2022-07-12 | 天津大学 | Chinese character image restoration algorithm based on skeleton extraction and antagonistic learning |
| CN113920516A (en) * | 2021-12-08 | 2022-01-11 | 南京信息工程大学 | Calligraphy character skeleton matching method and system based on twin neural network |
| CN115170403A (en) * | 2022-05-23 | 2022-10-11 | 广州大学 | Font repairing method and system based on deep meta learning and generation countermeasure network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Yeh et al. | Multi-scale deep residual learning-based single image haze removal via image decomposition | |
| Lei et al. | Coupled adversarial training for remote sensing image super-resolution | |
| CN100373399C (en) | Method and apparatus for establishing degradation dictionary | |
| CN114758383A (en) | Expression recognition method based on attention modulation context spatial information | |
| CN112966684A (en) | Cooperative learning character recognition method under attention mechanism | |
| CN111612008A (en) | Image segmentation method based on convolution network | |
| CN110059769B (en) | Semantic segmentation method and system based on pixel rearrangement reconstruction and used for street view understanding | |
| CN114187520B (en) | Building extraction model construction and application method | |
| CN112561876A (en) | Image-based pond and reservoir water quality detection method and system | |
| CN103150741B (en) | Method for rapidly skeletonizing graph of binary digital image | |
| CN111666813B (en) | Subcutaneous sweat gland extraction method of three-dimensional convolutional neural network based on non-local information | |
| CN113762265A (en) | Pneumonia classification and segmentation method and system | |
| CN115222750A (en) | Remote sensing image segmentation method and system based on multi-scale fusion attention | |
| CN113344933B (en) | Glandular cell segmentation method based on multi-level feature fusion network | |
| CN114581789A (en) | Hyperspectral image classification method and system | |
| CN117333881A (en) | Oracle auxiliary decoding method based on conditional diffusion model | |
| CN115660979A (en) | Attention mechanism-based double-discriminator image restoration method | |
| CN116091363A (en) | Handwriting Chinese character image restoration method and system | |
| CN113554655B (en) | Optical remote sensing image segmentation method and device based on multi-feature enhancement | |
| CN115147315A (en) | Neural network fluorescence microscopic image denoising method based on transformer module | |
| CN115131244A (en) | Single image rain removing method and system based on counterstudy | |
| CN114842339A (en) | Image extraction method for winter wheat planting area combining GF-6 and Sentinel-2 | |
| CN114331894A (en) | Face image restoration method based on potential feature reconstruction and mask perception | |
| CN113901913A (en) | Convolution network for ancient book document image binaryzation | |
| CN113192076A (en) | MRI brain tumor image segmentation method combining classification prediction and multi-scale feature extraction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230509 |