CN111862253A - Sketch coloring method and system for generating confrontation network based on deep convolution - Google Patents

Sketch coloring method and system for generating confrontation network based on deep convolution Download PDF

Info

Publication number
CN111862253A
CN111862253A CN202010672560.7A CN202010672560A CN111862253A CN 111862253 A CN111862253 A CN 111862253A CN 202010672560 A CN202010672560 A CN 202010672560A CN 111862253 A CN111862253 A CN 111862253A
Authority
CN
China
Prior art keywords
ggn
sketch
gray
cgn
scale
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010672560.7A
Other languages
Chinese (zh)
Other versions
CN111862253B (en
Inventor
张俊松
朱少强
刘坤祥
杨宗凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Central China Normal University
Original Assignee
Central China Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Central China Normal University filed Critical Central China Normal University
Priority to CN202010672560.7A priority Critical patent/CN111862253B/en
Publication of CN111862253A publication Critical patent/CN111862253A/en
Application granted granted Critical
Publication of CN111862253B publication Critical patent/CN111862253B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a sketch coloring method and a sketch coloring system for generating a confrontation network based on deep convolution, wherein the sketch coloring method comprises the following steps of: s100, extracting characteristics of the sketch by using a gray-scale map generator GGN, and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch; s110, utilizing a color map generator CGN to perform feature extraction on the gray-scale map and the graffiti information of the sketch by a user, and deconvoluting the features of the extracted gray-scale map and the graffiti information to generate a corresponding color map; in the CGN network in the coloring stage, the invention skillfully utilizes a training model to extract the high-level characteristics of the sketch and inputs the high-level characteristics into the CGN, so that the CGN network not only obtains the characteristics of the gray-scale image, but also obtains the characteristics of the sketch, and the coloring result is more accurate.

Description

Sketch coloring method and system for generating confrontation network based on deep convolution
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a sketch coloring method and system for generating a confrontation network based on deep convolution.
Background
With the advent of deep learning, the demand for automatic image processing has increased, wherein sketch coloring is an important field in image processing. The automatic sketch coloring technology can quickly finish coloring work of a large number of pictures, and can endow colors to some early comic books, movies and the like, so that the method has great application value.
In the traditional method, the sketch coloring is performed by utilizing some manually customized rules, most of the obtained results are color overflow and incomplete coloring, and most importantly, the method is difficult to perform corresponding coloring according to the wishes of users.
With the advent of the big data era, image processing using deep learning has become a trend. Image rendering also caters for another wave of research heat tide, mainly divided into two directions. Some researchers colored the gray level image, and an end-to-end network is constructed by using data driving, so that the network learns the direct mapping from the gray level image to the color image, and the coloring purpose is achieved. While others have colored the sketch because the sketch has less information than the grayscale, and because the sketch has no texture information and shading information, the coloring is more difficult. The paper "Outline colorization high Tandem adaptive Networks" extracts an Outline from a color map as input by using an Outline extraction algorithm, and then trains an end-to-end network model, but the Outline extracted by the Outline extraction algorithm is far from the actual draft data distribution, so that the mapping relation learned by the model is not suitable for the actual situation; then, in addition to the common countermeasure loss, the model only adds the difference of each pixel between the label and the model output as the loss, so that the model is difficult to learn the corresponding mapping relationship, and the result of the coloring is not ideal. The paper "User-Guided Deep animal Line Art arrangement coloring with Conditional additive networks" proposes another sketch coloring method, in which, compared with the previous sketch coloring method, the author tries to learn the mapping between the sketch and the color map by using a deeper network model, and in addition, the loss function of the model adds the sensory loss except for the resistance loss and the pixel loss, so that the model can well learn the coloring task, but the problems of coloring overflow, watercolor effect, consumption of computing resources and time and the like still exist.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a sketch coloring method and a sketch coloring system based on a deep convolution generation countermeasure network, and aims to solve the technical problems that the existing sketch coloring consumes huge computing resources and time and cannot accurately color.
In order to achieve the above object, in a first aspect, the present invention provides a sketch coloring method for generating a countermeasure network based on deep convolution, including the following steps:
s100, extracting characteristics of the sketch by using a gray-scale map generator GGN, and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch;
s110, utilizing a color map generator CGN to perform feature extraction on the gray-scale map and the graffiti information of the sketch by a user, and deconvoluting the features of the extracted gray-scale map and the graffiti information to generate a corresponding color map; and the GGN and the CGN both belong to a deep convolution generation countermeasure network.
Specifically, the scrawling information of the user on the sketch can also be user interaction information, which can be understood as information such as a preliminary coloring plan of the user on the sketch.
Optionally, the GGN comprises a generator, a discriminator, and a self-encoder;
the discriminator and the self-encoder in the GGN are used for assisting the generator in the GGN to generate a gray-scale map based on a sketch;
The discriminator in the GGN is used for discriminating the quality of the gray-scale map generated by the generator so as to generate the optimal quality of the gray-scale map;
the self-encoder is used for encoding and decoding the characteristics of the gray-scale image, after the self-encoder extracts the characteristic vector of the gray-scale image, the Euclidean distance corresponding to the characteristic vector of the rough sketch extracted by the GGN is calculated to be used as a loss function of the GGN, so that the characteristic vector of the rough sketch extracted by the GGN is similar to the characteristics of the gray-scale image as much as possible, and the gray-scale image with the optimal quality can be generated by the GGN through deconvolution.
Optionally, Loss of generator penalty function in the GGN_GGNThe calculation formula is as follows:
Loss_GGN＝Ladv_GGN_g+Lpix_GGN+Lfeat_GGN+Lpercep_GGN
Ladv_GGN＝-Es,g～Pdata(s,g)[logDiss(Gens(s))]
Lpix_GGN＝Es,g～Pdata(s,g)[||Gens(s)-g||1]
Lfeat_GGN＝Es,g～Pdata(s,g)[||Es(s)-EG(g)||1]
Figure BDA0002582866490000031
wherein L isadv_GGN_gTo combat losses in GGN, Lpix_GGNPixel loss, L, of gray scale map and label gray scale map generated for GGNfeat_GGNIs a loss of features in GGN, Lfeat_GGNThe method aims to ensure that the characteristic vector of the draft extracted by the GGN is similar to the characteristic vector of the label gray-scale image corresponding to the draft extracted by the self-encoder as much as possible, so that the gray-scale image generated by the GGN is vivid as much as possible, and L ispercep_GGNIs sensory loss in GGN, Lpercep_GGNThe aim is to promote the generated gray level image to be vivid as much as possible; s represents a sketch, g represents a label gray scale map corresponding to s,
Figure BDA0002582866490000032
representing a gray scale map generated by the GGN;
Figure BDA0002582866490000033
representing the data distribution P corresponding to the sketch s and the label gray-scale map g data(s, g) a set of sketches randomly extractedAnd a gray-scale map,
Figure BDA0002582866490000034
representing gray scale maps generated from GGNs
Figure BDA0002582866490000035
Data distribution corresponding to label gray-scale map g
Figure BDA0002582866490000036
A group of sketches and gray level images from the middle sample, NlAnd philRespectively representing the pixel number of the l-th layer feature map of the pre-training model VGG16 and the l-th layer feature map; dis (disease)sAnd GensDiscriminator and generator, respectively representing GGN, EsAnd EGRepresenting the encoding part of the self-encoder and the encoding part of the generator, respectively.
As can be appreciated, Diss、GensAnd EsThe subscript s in the middle and the left symbol are respectively used as the code number of the related network;
Figure BDA0002582866490000037
subscript s in (1) and s in parentheses and GensS in parentheses of(s) are each referred to as a sketch.
It should be noted that VGG16 is another pre-training model network for feature extraction, which is independent from GGNs, and it can be simply understood here that VGG is used to help GGNs generate better gray level maps. The using method is that the gray-scale image generated by the GGN is thrown into the VGG to obtain a characteristic vector L1, the label gray-scale image is also thrown into the VGG to obtain another characteristic vector L2, and then the difference between the two characteristic vectors is smaller, so that the quality of the gray-scale image generated by the GGN is improved.
Optionally, a discriminator loss function L in the GGN adv_GGN_dThe calculation formula of (2) is as follows:
Figure BDA0002582866490000041
optionally, the self-coder loss function in the GGNLAEThe calculation formula of (2) is as follows:
Figure BDA0002582866490000042
wherein g is a grey scale map of the label, EGBeing the coding part of an autoencoder, DGIs the decoding part of the self-encoder.
Optionally, the CGN comprises a generator and a discriminator;
the CGN generator is used for extracting features of the sketch according to the gray-scale image generated by the GGN and the doodle information of the user, and deconvoluting the extracted features of the gray-scale image and the doodle information to generate a corresponding color image;
the discriminator in the CGN is used for assisting in discriminating the color map generated by the generator in the CGN, so that the coloring result of the generator in the CGN is optimal.
Optionally, a generator Loss function Loss in the CGN_CGNThe calculation formula of (2) is as follows:
Loss_CGN＝Ladv_CGN_g+Lpix_CGN+Lpercep_CGN
Figure BDA0002582866490000043
Figure BDA0002582866490000044
Figure BDA0002582866490000051
wherein L isadv_CGN_gTo combat losses in CGN, Lpix_CGNPixel loss, L, of color map and label color map generated for CGNpercep_CGNIs sensory loss in CGN, Lpercep_CGNAims to promote the generated color image to be vivid as much as possible; y represents the color drawing of the label,
Figure BDA0002582866490000052
representative GGN network generationS represents a sketch, c represents user graffiti information for the sketch,
Figure BDA0002582866490000053
representing a color map generated by the CGN generator;
Figure BDA0002582866490000054
representing slave data distribution
Figure BDA0002582866490000055
In the random drawing of a set of data,
Figure BDA0002582866490000056
Representing slave data distribution
Figure BDA0002582866490000057
A set of data, N, drawn at randomlAnd philRespectively representing the pixel number of the l-th layer feature map of the pre-training model VGG16 and the l-th layer feature map; dis (disease)cAnd GencRespectively representing the discriminator and generator of the CGN.
Optionally, a discriminator loss function L in the CGNadv_CGN_dThe calculation formula of (2) is as follows:
Figure BDA0002582866490000058
optionally, the GGN and CGN are trained by:
taking a sketch as the input of the GGN, calculating the current loss function, and then performing reverse propagation by a gradient descent algorithm until the loss function is reduced to the minimum to enable the GGN model to be converged;
taking a gray scale image output by the GGN and corresponding user doodling information as the input of the CGN, calculating a current loss function, and then performing a gradient descent algorithm to perform back propagation until the loss function is minimized to enable the CGN model to be converged;
and performing joint training on the GGN and the CGN to ensure that the effect of the model is optimal.
In a second aspect, the present invention provides a sketch coloring system for generating a countermeasure network based on deep convolution, comprising:
the gray-scale map generation unit is used for extracting the characteristics of the sketch by using the gray-scale map generator GGN and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch;
and the color map generating unit is used for extracting the characteristics of the gray-scale image and the doodle information of the sketch by the user by using the color map generator CGN and deconvolving the extracted characteristics of the gray-scale image and the doodle information to generate a corresponding color map.
Generally, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:
(1) the sketch coloring method and the sketch coloring system based on the deep convolution generation confrontation network skillfully avoid the complex problem of directly learning from the sketch to the color map, but divide the problem into two relatively simple stages, an imitation stage and a coloring stage, and enable the two stages to respectively play their own roles, so that the final coloring result is greatly improved, and compared with the existing commercial coloring software PaintsChainer, the method and the system can obtain a smaller FID distance.
(2) According to the sketch coloring method and system based on the deep convolution generation countermeasure network, the self-encoder module is added in the GGN network in the simulation stage, so that the GGN can well learn the mapping from the sketch to the gray scale image, and the final coloring effect is indirectly improved.
(3) According to the sketch coloring method and system based on the deep convolution generation confrontation network, the advanced features of the sketch are skillfully extracted by utilizing the training model in the CGN network at the coloring stage and are input into the CGN, so that the CGN network not only obtains the features of the gray level image but also obtains the features of the sketch, and the coloring result is more accurate.
(4) According to the sketch coloring method and system for generating the confrontation network based on deep convolution, provided by the invention, the GGN network and the CGN network in the simulation stage are trained step by step and then are connected in series for training, and the training mode can optimize the model and greatly reduce the training process.
Drawings
FIG. 1 is a flow chart of a sketch coloring method for generating a countermeasure network based on deep convolution according to the invention;
FIG. 2 is a framework and flow diagram of a user-directed sketch coloring system provided by an embodiment of the present invention;
FIG. 3 is a block diagram of a user-guided sketch coloring system provided by an embodiment of the present invention;
FIG. 4 is a schematic diagram of a sketch rendering system for generating a countermeasure network based on deep convolution according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention discloses a system and a method for coloring a user-guided sketch for generating a confrontation network based on deep convolution. Extracting a characteristic vector of the sketch in the simulation stage, and converting the characteristic vector into a gray level graph by generating a multi-reactance network; the coloring stage takes the gray-scale image generated in the first stage and the interaction of the user as input, and correspondingly maps the colors to corresponding areas according to the intention of the user to generate a vivid color image. The invention reasonably segments the sketch coloring into two simple sub-modules, and compared with the prior method, the invention greatly reduces the FID distance through experimental verification, wherein the FID is an important index for measuring the quality and diversity of image generation, and the smaller the value is, the better the quality of the image generated by the generator is.
In response to the above-identified deficiencies in the art or needs in the art, the present invention provides a system and method for shading a user-interactive sketch for generating a confrontational network based on deep convolution. The method skillfully divides the complicated problem of sketch coloring into two stages, an imitation stage and a coloring stage, wherein the imitation stage utilizes deep convolution to generate a mapping relation from a confrontation network (GGN) learning sketch to a single-channel grey-scale map; the coloring stage of the method utilizes deep convolution to generate a confrontation network (CGN) to learn the mapping relation between the gray-scale image generated by the simulation stage and the color image. The method aims to divide the complex problem of sketch coloring into two simple sub-problems and solve the two simple sub-problems in sequence, thereby solving the problems that the existing sketch coloring consumes huge computing resources and time and cannot accurately color.
To achieve the above object, according to one aspect of the present invention, there is provided a method for generating user interaction sketch coloring of a confrontation network based on deep convolution, which specifically includes two steps of an emulation stage and a coloring stage:
and the simulation stage is used for extracting the characteristics of the sketch by using the GGN and then outputting a gray-scale image corresponding to the sketch through deconvolution.
And the coloring stage is used for extracting the characteristics of the gray-scale image generated by the simulation stage and the scrawling of the corresponding picture of the user by using CGN (common color network), and then generating a corresponding color image through deconvolution. In order to better help the CGN coloring, the system inputs the characteristics of the sketch into the CGN by extracting the characteristics of the sketch through a pre-training model for extracting the characteristics of the sketch, thereby helping the network to achieve a better training effect.
The CGN network and the CGN network are respectively used for generating a gray scale image and a color image, and are composed of a generator and a discriminator.
The generator and the discriminator are basic frameworks for generating the countermeasure network, the generator is mainly used for generating a specific problem, and the discriminator is mainly used for assisting the generation of the generator.
Preferably, the user-interactive sketch coloring system comprises a self-encoder in a simulation stage (GGN), wherein the self-encoder is used for encoding and decoding the characteristics of the real gray-scale map, and the main function of the self-encoder is to help the GGN to improve the quality of the generated gray-scale map. The self-encoder loss function calculation formula is as follows:
Figure BDA0002582866490000081
wherein g is the gray of the labelDegree diagram, EGBeing the coding part of an autoencoder, DGIs the decoding part of the self-encoder.
Preferably, in the user-interactive sketch coloring system, after extracting the feature vector of the grayscale from the encoder, the euclidean distance corresponding to the feature vector of the GGN-extracted sketch is calculated as a loss function of the GGN, so as to force the feature vector of the GGN-extracted sketch to be similar to the feature of the grayscale as much as possible, and the GGN can generate the grayscale with good quality by deconvolution.
FIG. 1 is a flow chart of a sketch coloring method for generating a countermeasure network based on deep convolution according to the invention; as shown in fig. 1, the method comprises the following steps:
s100, extracting characteristics of the sketch by using a gray-scale map generator GGN, and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch;
s110, utilizing a color map generator CGN to perform feature extraction on the gray-scale map and the graffiti information of the sketch by a user, and deconvoluting the features of the extracted gray-scale map and the graffiti information to generate a corresponding color map; and the GGN and the CGN both belong to a deep convolution generation countermeasure network.
FIG. 2 is a general diagram of the shading of a user interaction sketch to generate a countermeasure network based on deep convolution according to an embodiment of the present invention. As shown in fig. 2, the method comprises two major stages: a mimic phase and a render phase. For the simulation stage and the GGN network, a sketch is used as the input of the GGN, and a gray-scale image is the output of the GGN; for the coloring stage and the CGN network, the output of the first stage and the user interaction are used as the input of the CGN, and the output of the CGN is a colored color map.
Specifically, in fig. 2, s represents a sketch, c represents scribble information of the user on the sketch,
Figure BDA0002582866490000091
representing a gray scale map generated by the GGN,
Figure BDA0002582866490000092
representing a color map generated by the CGN. Wherein the color map
Figure BDA0002582866490000093
Is to gray scale map
Figure BDA0002582866490000094
The colored picture does not show the coloring result in the present invention because the drawing cannot have a color image, but the coloring of the color image can be understood by those skilled in the art.
FIG. 3 is a detailed schematic diagram of the shading of the user interaction sketch of the countermeasure network based on deep convolution generation according to the embodiment of the present invention, which includes the following specific steps:
s1, preparing a data set for training a network, downloading 30000 images of color characters (256 × 256 in size) from an open cartoon website, extracting the color image outline by an outline extraction algorithm XDoG, screening to obtain a better outline image as the input of GGN, and converting the color image into a gray image which is the label of the GGN.
And S2, training the GGN network in the simulation phase. Specifically, firstly, the discriminator of the GGN is trained once, the label gray-scale image and the gray-scale image generated by the GGN are respectively input into the discriminator, the loss function is calculated, and the gradient descent algorithm is used for back propagation to update the network parameters of the discriminator. And then training the GGN generator five times, taking a sketch as the input of the generator, obtaining a loss function corresponding to the label data after output, and performing back propagation to update network parameters through a gradient descent algorithm. This process is repeated until the loss function of the GGN converges.
The loss function calculation mode of the GGN network generator is as follows:
LossGGN＝Ladv_GGN_g+Lpix_GGN+Lfeat_GGN+Lpercep_GGN
Figure BDA0002582866490000095
Figure BDA0002582866490000101
Figure BDA0002582866490000102
Figure BDA0002582866490000103
wherein L isadv_GGN_gTo combat losses, Lpix_GGNPixel loss, L, of gray scale map and label gray scale map generated for GGNfeat_GGNThe loss of the characteristic vector aims to make the characteristic vector of the draft extracted by the GGN similar to the characteristic vector of the label gray-scale image corresponding to the draft extracted by the self-encoder as much as possible, so that the gray-scale image generated by the GGN is vivid as much as possible, and L is a loss of the characteristic vectorpercep_GGNThe aim is also to make the generated grey scale images as realistic as possible for the sake of sensory loss. In addition, s represents a sketch, g represents a label gray scale image corresponding to s,
Figure BDA0002582866490000105
representing grey-scale maps generated by GGN, NlAnd philRespectively representing the pixel number of the l-th layer feature map of the pre-training model VGG16 and the l-th layer feature map; dis (disease)sAnd GensDiscriminator and generator, respectively representing GGN, EsAnd EGRepresenting the encoding part of the self-encoder and the encoding part of the generator, respectively.
The loss function calculation mode of the GGN network discriminator is as follows:
Figure BDA0002582866490000104
wherein s represents sketch, g represents gray scale map of label corresponding to s, DissAnd GensRespectively representing the discriminator and the generator of the GGN.
S3, training the CGN network of the coloring stage. Specifically, firstly, training a discriminator of the CGN network once, inputting a color image and a label color image generated by the CGN into the discriminator, calculating a loss function, and performing back propagation through a gradient descent algorithm to update network parameters; and then training a generator of the CGN for five times, taking the interaction diagram generated in the simulation stage and connected with a user and the feature diagram of the pre-training model extraction sketch as the input of the CGN generator, obtaining the output and the label calculation loss function, and performing back propagation by using a gradient descent algorithm to update network parameters. This process is repeated until the loss function of the GGN converges.
The construction process of the user interaction is as follows: and randomly digging 60 pixel blocks with the size of 10 × 10 in the label color image, then carrying out Gaussian blur, and taking the obtained color plate as a user interaction image in the training process.
The loss function calculation mode of the CGN generator is as follows:
LossCGN＝Ladv_CGN_g+Lpix_CGN+Lpercep_CGN
Figure BDA0002582866490000111
Figure BDA0002582866490000112
Figure BDA0002582866490000113
wherein L isadv_CGN_gTo combat losses, Lpix_CGNPixel loss, L, of color map and label color map generated for CGNpercep_CGNThe aim is also to make the colour image generated as realistic as possible for organoleptic losses. In addition, where y represents the label color map,
Figure BDA0002582866490000114
representing a grey scale map generated by the GGN network during the simulation phase, s representing a sketch, c representing graffiti information of a user, NlAnd philRespectively representing the pixel number of the l-th layer feature map of the pre-training model VGG16 and the l-th layer feature map; dis (disease)cAnd GencRespectively representing the discriminator and generator of the CGN.
The loss function calculation mode of the CGN network discriminator is as follows:
Figure BDA0002582866490000115
wherein, y represents the color image of the label,
Figure BDA0002582866490000116
representing a grey scale map generated by the GGN network during the simulation phase, s representing a sketch, c representing user interaction, DiscAnd GencRespectively representing the discriminator and generator of the CGN.
S4, a combined training phase. The GGN network and the CGN network are connected in series, input as a sketch, and a characteristic diagram of the sketch is extracted through a training model, and output is a color diagram. And calculating a corresponding loss function, and performing back propagation by using a gradient descent algorithm to update the network parameters until the loss function is converged.
The user interaction sketch coloring algorithm for generating the confrontation network based on deep clipping can realize coloring according to the interaction of the user. After the model training is completed, 1000 pictures which are not seen in the model are used as a test set to be compared with the currently common coloring model, and the results are shown in table 1:
TABLE 1 comparison of coloring quality
FID
PaintsChainer(canna) 164.25
PaintsChainer(tanpopo) 168.13
PaintsChainer(satsuki) 148.12
The coloring method of the present invention 108.85
Table 1 shows FID distance comparisons between the rendering method of the present invention and three versions of the commercial software paintsscainer, where FID is an important index for measuring the quality and diversity of pictures, and a smaller FID value indicates a better quality and diversity of pictures. It can be seen that the shading method of the present invention can greatly reduce the FID distance compared to the currently used shading methods, i.e. the shading method of the present invention can achieve higher shading quality.
Fig. 4 is a schematic diagram of a sketch coloring system for generating a countermeasure network based on deep convolution according to the present invention, as shown in fig. 4, including:
the gray-scale map generation unit 410 is used for extracting the characteristics of the sketch by using the gray-scale map generator GGN and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch;
and the color map generating unit 420 is configured to perform feature extraction on the grayscale map and the graffiti information of the sketch by the user by using the color map generator CGN, and perform deconvolution on the features of the extracted grayscale map and the graffiti information to generate a corresponding color map.
The functions of each unit in fig. 4 are described in the foregoing method embodiments, and are not described herein again.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (10)

1. A sketch coloring method for generating a confrontation network based on deep convolution is characterized by comprising the following steps:
s100, extracting characteristics of the sketch by using a gray-scale map generator GGN, and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch;
s110, utilizing a color map generator CGN to perform feature extraction on the gray-scale map and the graffiti information of the sketch by a user, and deconvoluting the features of the extracted gray-scale map and the graffiti information to generate a corresponding color map; and the GGN and the CGN both belong to a deep convolution generation countermeasure network.
2. The sketch coloring method according to claim 1, wherein the GGN comprises a generator, a discriminator and a self-encoder;
the discriminator and the self-encoder in the GGN are used for assisting the generator in the GGN to generate a gray-scale map based on a sketch;
The discriminator in the GGN is used for discriminating the quality of the gray-scale map generated by the generator so as to generate the optimal quality of the gray-scale map;
the self-encoder is used for encoding and decoding the characteristics of the gray-scale image, after the self-encoder extracts the characteristic vector of the gray-scale image, the Euclidean distance corresponding to the characteristic vector of the rough sketch extracted by the GGN is calculated to be used as a loss function of the GGN, so that the characteristic vector of the rough sketch extracted by the GGN is similar to the characteristics of the gray-scale image as much as possible, and the gray-scale image with the optimal quality can be generated by the GGN through deconvolution.
3. The sketch coloring method as claimed in claim 2, wherein the Loss of generator Loss function in the GGN is less than_GGNThe calculation formula is as follows:
Loss_GGN＝Ladv_GGN_g+Lpix_GGN+Lfeat_GGN+Lpercep_GGN
Figure FDA0002582866480000011
Figure FDA0002582866480000012
Figure FDA0002582866480000013
Figure FDA0002582866480000021
wherein L isadv_GGN_gTo combat losses in GGN, Lpix_GGNPixel loss, L, of gray scale map and label gray scale map generated for GGNfeat_GGNIs a loss of features in GGN, Lfeat_GGNThe method aims to ensure that the characteristic vector of the draft extracted by the GGN is similar to the characteristic vector of the label gray-scale image corresponding to the draft extracted by the self-encoder as much as possible, so that the gray-scale image generated by the GGN is vivid as much as possible, and L ispercep_GGNIs sensory loss in GGN, Lpercep_GGNThe aim is to promote the generated gray level image to be vivid as much as possible; s represents a sketch, g represents a label gray scale map corresponding to s,
Figure FDA0002582866480000022
Representing a gray scale map generated by the GGN;
Figure FDA0002582866480000023
representing the data distribution P corresponding to the sketch s and the label gray-scale map gdata(s, g) a group of sketches and gray-scale maps which are randomly extracted,
Figure FDA0002582866480000024
representing gray scale maps generated from GGNs
Figure FDA0002582866480000025
Data distribution corresponding to label gray-scale map g
Figure FDA0002582866480000026
A group of sketches and gray-scale images, N, randomly extracted fromlAnd philRespectively representing the pixel number of the l-th layer feature map of the pre-training model VGG16 and the l-th layer feature map; dis (disease)sAnd GensDiscriminator and generator, respectively representing GGN, EsAnd EGRepresenting the encoding part of the self-encoder and the encoding part of the generator, respectively.
4. The sketch coloring method as claimed in claim 3, wherein said discriminator loss function L in GGNadv_GGN_dThe calculation formula of (2) is as follows:
Figure FDA0002582866480000027
5. the sketch coloring method as claimed in claim 3, wherein said self-coder loss function L in GGNAEThe calculation formula of (2) is as follows:
Figure FDA0002582866480000028
wherein g is a grey scale map of the label, EGBeing the coding part of an autoencoder, DGIs the decoding part of the self-encoder.
6. The sketch coloring method of claim 1, wherein the CGN comprises a generator and a discriminator;
the CGN generator is used for extracting features of the sketch according to the gray-scale image generated by the GGN and the doodle information of the user, and deconvoluting the extracted features of the gray-scale image and the doodle information to generate a corresponding color image;
The discriminator in the CGN is used for assisting in discriminating the color map generated by the generator in the CGN, so that the coloring result of the generator in the CGN is optimal.
7. The sketch coloring method as claimed in claim 6, wherein said generator Loss function Loss in CGN_CGNThe calculation formula of (2) is as follows:
Loss_CGN＝Ladv_CGN_g+Lpix_CGN+Lpercep_CGN
Figure FDA0002582866480000031
Figure FDA0002582866480000032
Figure FDA0002582866480000033
wherein L isadv_CGN_gTo combat losses in CGN, Lpix_CGNPixel loss, L, of color map and label color map generated for CGNpercep_CGNIs sensory loss in CGN, Lpercep_CGNAims to promote the generated color image to be vivid as much as possible; y represents the color drawing of the label,
Figure FDA0002582866480000034
representing a gray scale map generated by the GGN network, s representing a sketch, c representing graffiti information of a user on the sketch,
Figure FDA0002582866480000035
represents the color map generated by the CGN generator,
Figure FDA0002582866480000036
representing slave data distribution
Figure FDA0002582866480000037
In the random drawing of a set of data,
Figure FDA0002582866480000038
representing slave data distribution
Figure FDA0002582866480000039
A set of data, N, drawn at randomlAnd philRespectively representing the pixel number of the l-th layer feature map of the pre-training model VGG16 and the l-th layer feature map; dis (disease)cAnd GencRespectively representing the discriminator and generator of the CGN.
8. The sketch coloring method as claimed in claim 7, wherein a discriminator loss function L in said CGNadv_CGN_dThe calculation formula of (2) is as follows:
Figure FDA0002582866480000041
9. the sketch coloring method of any one of claims 1 to 8, wherein the GGNs and CGNs are trained by:
Taking a sketch as the input of the GGN, calculating the current loss function, and then performing reverse propagation by a gradient descent algorithm until the loss function is reduced to the minimum to enable the GGN model to be converged;
taking a gray scale image output by the GGN and corresponding user doodling information as the input of the CGN, calculating a current loss function, and then performing a gradient descent algorithm to perform back propagation until the loss function is minimized to enable the CGN model to be converged;
and performing joint training on the GGN and the CGN to ensure that the effect of the model is optimal.
10. A sketch coloring system for generating a confrontation network based on deep convolution, comprising:
the gray-scale map generation unit is used for extracting the characteristics of the sketch by using the gray-scale map generator GGN and performing deconvolution on the characteristics of the sketch to generate a gray-scale map corresponding to the sketch;
and the color map generating unit is used for extracting the characteristics of the gray-scale image and the doodle information of the sketch by the user by using the color map generator CGN and deconvolving the extracted characteristics of the gray-scale image and the doodle information to generate a corresponding color map.
CN202010672560.7A 2020-07-14 2020-07-14 Sketch coloring method and system for generating countermeasure network based on deep convolution Active CN111862253B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010672560.7A CN111862253B (en) 2020-07-14 2020-07-14 Sketch coloring method and system for generating countermeasure network based on deep convolution

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010672560.7A CN111862253B (en) 2020-07-14 2020-07-14 Sketch coloring method and system for generating countermeasure network based on deep convolution

Publications (2)

Publication Number Publication Date
CN111862253A true CN111862253A (en) 2020-10-30
CN111862253B CN111862253B (en) 2023-09-15

Family

ID=72984733

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010672560.7A Active CN111862253B (en) 2020-07-14 2020-07-14 Sketch coloring method and system for generating countermeasure network based on deep convolution

Country Status (1)

Country Link
CN (1) CN111862253B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112991371A (en) * 2021-04-20 2021-06-18 云南大学 Automatic image coloring method and system based on coloring overflow constraint

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001043361A (en) * 1999-08-03 2001-02-16 Yokogawa Electric Corp Image processing method and image processing using the same
AU2018100325A4 (en) * 2018-03-15 2018-04-26 Nian, Xilai MR A New Method For Fast Images And Videos Coloring By Using Conditional Generative Adversarial Networks
CN107992572A (en) * 2017-11-30 2018-05-04 天津大学 A kind of distributed graph coloring algorithm based on Pregel
CN108717698A (en) * 2018-05-28 2018-10-30 深圳市唯特视科技有限公司 A kind of high quality graphic generation method generating confrontation network based on depth convolution
CN109712203A (en) * 2018-12-29 2019-05-03 福建帝视信息科技有限公司 A kind of image rendering methods based on from attention generation confrontation network
CN110084751A (en) * 2019-04-24 2019-08-02 复旦大学 Image re-construction system and method
CN110111236A (en) * 2019-04-19 2019-08-09 大连理工大学 The method for generating image based on the multiple target sketch that gradual confrontation generates network
CN110223359A (en) * 2019-05-27 2019-09-10 浙江大学 It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation
CN110569761A (en) * 2019-08-27 2019-12-13 武汉大学 Method for retrieving remote sensing image by hand-drawn sketch based on counterstudy
US20200013205A1 (en) * 2018-07-05 2020-01-09 Adobe Inc. Colorizing Vector Graphic Objects
CN111080522A (en) * 2019-12-13 2020-04-28 福州大学 Image super-resolution reconstruction method based on bidirectional countermeasure network
CN111402179A (en) * 2020-03-12 2020-07-10 南昌航空大学 Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001043361A (en) * 1999-08-03 2001-02-16 Yokogawa Electric Corp Image processing method and image processing using the same
CN107992572A (en) * 2017-11-30 2018-05-04 天津大学 A kind of distributed graph coloring algorithm based on Pregel
AU2018100325A4 (en) * 2018-03-15 2018-04-26 Nian, Xilai MR A New Method For Fast Images And Videos Coloring By Using Conditional Generative Adversarial Networks
CN108717698A (en) * 2018-05-28 2018-10-30 深圳市唯特视科技有限公司 A kind of high quality graphic generation method generating confrontation network based on depth convolution
US20200013205A1 (en) * 2018-07-05 2020-01-09 Adobe Inc. Colorizing Vector Graphic Objects
CN109712203A (en) * 2018-12-29 2019-05-03 福建帝视信息科技有限公司 A kind of image rendering methods based on from attention generation confrontation network
CN110111236A (en) * 2019-04-19 2019-08-09 大连理工大学 The method for generating image based on the multiple target sketch that gradual confrontation generates network
CN110084751A (en) * 2019-04-24 2019-08-02 复旦大学 Image re-construction system and method
CN110223359A (en) * 2019-05-27 2019-09-10 浙江大学 It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation
CN110569761A (en) * 2019-08-27 2019-12-13 武汉大学 Method for retrieving remote sensing image by hand-drawn sketch based on counterstudy
CN111080522A (en) * 2019-12-13 2020-04-28 福州大学 Image super-resolution reconstruction method based on bidirectional countermeasure network
CN111402179A (en) * 2020-03-12 2020-07-10 南昌航空大学 Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
K. LATA, M. DAVE AND K. N. NISHANTH: "Image-to-Image Translation Using Generative Adversarial Network", COMMUNICATION AND AEROSPACE TECHNOLOGY, pages 186 - 189 *
谢敏珊: "《基于感知损失的深度卷积网络的草图简化研究》", 中国优秀硕士学位沦为全文数据库（电子期刊）, pages 11 - 24 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112991371A (en) * 2021-04-20 2021-06-18 云南大学 Automatic image coloring method and system based on coloring overflow constraint
CN112991371B (en) * 2021-04-20 2024-01-05 云南大学 Automatic image coloring method and system based on coloring overflow constraint

Also Published As

Publication number Publication date
CN111862253B (en) 2023-09-15

Similar Documents

Publication Publication Date Title
Golts et al. Unsupervised single image dehazing using dark channel prior loss
CN110378985B (en) Animation drawing auxiliary creation method based on GAN
CN109859288B (en) Image coloring method and device based on generation countermeasure network
CN109447906B (en) Picture synthesis method based on generation countermeasure network
CN111145116B (en) Sea surface rainy day image sample augmentation method based on generation of countermeasure network
CN108875935B (en) Natural image target material visual characteristic mapping method based on generation countermeasure network
CN111161137B (en) Multi-style Chinese painting flower generation method based on neural network
Zhang et al. Bionic face sketch generator
CN113807265B (en) Diversified human face image synthesis method and system
CN113724354B (en) Gray image coloring method based on reference picture color style
Zhou et al. An interactive and generative approach for chinese shanshui painting document
CN115424088A (en) Image processing model training method and device
CN107730568B (en) Coloring method and device based on weight learning
CN116935043A (en) Typical object remote sensing image generation method based on multitasking countermeasure network
CN111862253A (en) Sketch coloring method and system for generating confrontation network based on deep convolution
He Application of local color simulation method of landscape painting based on deep learning generative adversarial networks
CN115018729B (en) Content-oriented white box image enhancement method
CN111382871A (en) Domain generalization and domain self-adaptive learning method based on data expansion consistency
CN110516202A (en) Acquisition methods, document structure tree method, apparatus and the electronic equipment of document generator
CN114332286B (en) Artificial intelligent drawing method and device and computer storage medium
CN114917583A (en) Animation style game background generation method and platform based on generation confrontation network
CN113129409B (en) Cartoon line manuscript coloring method based on deep learning
CN114549288A (en) Sketch landscape map generation method based on deep learning
CN115496843A (en) Local realistic-writing cartoon style migration system and method based on GAN
CN114419195A (en) Image synthesis method and device based on relation embedding and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant