CN111862253B - Sketch coloring method and system for generating countermeasure network based on deep convolution - Google Patents
Sketch coloring method and system for generating countermeasure network based on deep convolution Download PDFInfo
- Publication number
- CN111862253B CN111862253B CN202010672560.7A CN202010672560A CN111862253B CN 111862253 B CN111862253 B CN 111862253B CN 202010672560 A CN202010672560 A CN 202010672560A CN 111862253 B CN111862253 B CN 111862253B
- Authority
- CN
- China
- Prior art keywords
- ggn
- sketch
- cgn
- gray
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention provides a sketch coloring method and a sketch coloring system for generating an countermeasure network based on deep convolution, wherein the sketch coloring method comprises the following steps: s100, extracting features of the sketch by using a gray scale generator GGN, and deconvoluting the features of the sketch to generate a gray scale corresponding to the sketch; s110, feature extraction is carried out on the gray level map and the graffiti information of the sketch by using a color map generator CGN, and corresponding color maps are generated by feature deconvolution of the extracted gray level map and the graffiti information; in the coloring stage CGN network, the invention skillfully utilizes a training model to extract the advanced features of the sketch and inputs the advanced features into the CGN, so that the CGN network not only obtains the features of the gray level map, but also obtains the features of the sketch, and the coloring result is more accurate.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a sketch coloring method and system for generating an countermeasure network based on deep convolution.
Background
With the advent of deep learning, the need for automatic image processing has increased, with sketch coloring being a relatively important area in image processing. The sketch automatic coloring technology can quickly complete the coloring work of a large number of pictures, and can endow color to early-stage cartoons, movies and the like, so that the sketch automatic coloring technology has great application value.
In the traditional method, aiming at sketch coloring, a plurality of manually customized rules are utilized to color the sketch, and most importantly, the method is difficult to color correspondingly according to the wish of a user when color overflow and incomplete coloring occur in the obtained result.
With the advent of the big data age, image processing using deep learning has become a trend. Image coloring also comes from another wave of research hot flashes, which are mainly divided into two directions. Some researchers color the gray-scale image by using data driving to build an end-to-end network, so that the network learns the direct mapping from the gray-scale image to the color image, thereby achieving the purpose of coloring. While other researchers color sketches because they have a smaller amount of information than gray-scale images, because they have no texture information and no shading information, and therefore are more difficult to color. The paper Outline Colorization through Tandem Adversarial Networks uses a contour extraction algorithm to extract a contour from a color map as input, and trains an end-to-end network model, but the mapping relation learned by the model is not applicable to the actual situation because the contour extracted by the contour extraction algorithm is far away from the actual sketch data distribution; the model loss function is then designed to only add the difference of each pixel between the label and the model output as a loss in addition to the usual counterloss, so that the model has difficulty learning the corresponding mapping relationship, so that the resulting effect of coloring is less than ideal. Paper User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks proposes another sketch coloring method, and authors try to learn the mapping between sketches and color maps by using deeper network models than the previous sketch coloring method, and besides, the loss function of the model adds sensory loss except for anti-loss and pixel loss, so that the model can learn the task of coloring well, but the problems of coloring overflow, watercolor effect, and consumption of computing resources and time still exist.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a sketch coloring method and a sketch coloring system for generating an countermeasure network based on deep convolution, which aim to solve the technical problems that the existing sketch coloring consumes huge computing resources and time and cannot be used for precise coloring.
To achieve the above object, in a first aspect, the present invention provides a sketch coloring method for generating an countermeasure network based on deep convolution, including the steps of:
s100, extracting features of the sketch by using a gray scale generator GGN, and deconvoluting the features of the sketch to generate a gray scale corresponding to the sketch;
s110, feature extraction is carried out on the gray level map and the graffiti information of the sketch by using a color map generator CGN, and corresponding color maps are generated by feature deconvolution of the extracted gray level map and the graffiti information; the GGN and the CGN both belong to a deep convolution generating countermeasure network.
Specifically, the graffiti information of the sketch by the user can also be user interaction information, and the graffiti information can be understood as information such as a preliminary coloring plan of the sketch by the user.
Optionally, the GGN comprises a generator, a discriminator, and a self-encoder;
the identifier and the self-encoder in the GGN are used for assisting the generator in the GGN to generate a gray map based on the sketch;
the identifier in the GGN is used for identifying the quality of the gray scale map generated by the generator so as to generate the optimal quality of the gray scale map;
the self-encoder is used for encoding and decoding the characteristics of the gray map, after the characteristic vector of the gray map is extracted by the self-encoder, the Euclidean distance corresponding to the characteristic vector of the GGN extracted sketch is calculated as a loss function of the GGN, so that the characteristic vector of the GGN extracted sketch is similar to the characteristics of the gray map as much as possible, and the GGN can generate the gray map with optimal quality through deconvolution.
Optionally, the Loss of generator function in the GGN _GGN The calculation formula is as follows:
Loss _GGN ＝L adv_GGN_g +L pix_GGN +L feat_GGN +L percep_GGN
L adv_GGN ＝-E s,g～Pdata(s,g) [logDis s (Gen s (s))]
L pix_GGN ＝E s,g～Pdata(s,g) [||Gen s (s)-g|| 1 ]
L feat_GGN ＝E s,g～Pdata(s,g) [||E s (s)-E G (g)|| 1 ]
wherein L is adv_GGN_g To combat loss in GGN, L pix_GGN Pixel loss, L, of gray-scale image and label gray-scale image generated for GGN feat_GGN L is a characteristic loss in GGN feat_GGN The purpose is to make the feature vector of GGN extracted sketch and the feature vector of label gray-scale image corresponding to the extracted sketch from encoder as similar as possible, thereby making the gray-scale image generated by GGN as lifelike as possible, L percep_GGN For sensory loss in GGN, L percep_GGN The aim is to promote the generated gray level image to be as lifelike as possible; s represents sketch, g represents label gray scale corresponding to s,representing a gray scale map generated by the GGN; />Representing the data distribution P corresponding to the sketch s and the label gray-scale g data A set of sketches and grey-scale patterns randomly extracted from (s, g), and +.>Represents gray-scale patterns generated from GGN +.>Data distribution corresponding to the label gray map g +.>A group of sketches and gray-scale images, N, from sample l And phi is l The number of pixels and the first layer characteristic diagram of the pretraining model VGG16 are represented respectively; dis (Dis) s And Gen s Discriminator and generator, E, respectively representing GGNs s And E is G Representing the coding part of the self-encoder and the coding part of the generator, respectively.
It will be appreciated that Dis s 、Gen s E and E s The subscript s of the corresponding code is respectively integrated with the left symbol as the code number of the related network;subscript s in (a) and s in brackets and Gen s S in brackets of(s) refer to sketches.
It should be noted that VGG16 is another pre-trained model network that extracts features and is independent of GGNs, and it is also understood that VGG is used herein simply to help GGNs generate better gray maps. The using method is that the gray level image generated by the GGN is lost into the VGG to obtain a characteristic vector L1, the label gray level image is also lost into the VGG to obtain another characteristic vector L2, and then the smaller the difference between the two characteristic vectors is, the better the difference is, so that the quality of the gray level image generated by the GGN is improved.
Optionally, the discriminator loss function L in the GGN adv_GGN_d The calculation formula of (2) is as follows:
optionally, the self-encoder loss function L in the GGN AE The calculation formula of (2) is as follows:
where g is the gray scale of the label, E G Is the coding part of the self-encoder, D G Is the decoding part of the self-encoder.
Optionally, the CGN includes a generator and a discriminator;
the CGN generator is used for extracting features of the graffiti information of the sketch according to the gray level graph generated by the GGN and the user, and deconvoluting the extracted features of the gray level graph and the graffiti information to generate a corresponding color graph;
the discriminator in the CGN is used for assisting in discriminating the color map generated by the generator in the CGN, so that the coloring result of the generator in the CGN is optimal.
Optionally, a generator Loss function Loss in the CGN _CGN The calculation formula of (2) is as follows:
Loss _CGN ＝L adv_CGN_g +L pix_CGN +L percep_CGN
wherein L is adv_CGN_g To combat losses in CGN, L pix_CGN Pixel loss, L, of color map and label color map generated for CGN percep_CGN For sensory loss in CGN, L percep_CGN The purpose is to promote the generated color map to be as lifelike as possible; y represents the color map of the label,representing a gray scale map generated by the GGN network, s representing a sketch, c representing graffiti information of a user on the sketch,representing a color map generated by the CGN generator; />Representing the slave data distribution->A set of data randomly extracted in +.>Representing the slave data distribution->A set of data, N, randomly extracted l And phi is l The number of pixels and the first layer characteristic diagram of the pretraining model VGG16 are represented respectively; dis (Dis) c And Gen c Respectively representing a discriminator and a generator of the CGN.
Optionally, a discriminator loss function L in the CGN adv_CGN_d The calculation formula of (2) is as follows:
optionally, the GGN and CGN are trained by:
taking the sketch as the input of the GGN, calculating the current loss function, and then carrying out back propagation by a gradient descent algorithm until the loss function is minimized to enable the GGN model to be converged;
taking a gray level image output by the GGN network and corresponding user graffiti information as the input of the CGN, calculating a current loss function, and then carrying out back propagation by a gradient descent algorithm until the loss function is minimized to enable the CGN model to be converged;
and performing joint training on the GGN and the CGN to ensure that the effect of the model is optimal.
In a second aspect, the present invention provides a sketch-coloring system for generating an countermeasure network based on deep convolution, comprising:
the gray level image generating unit is used for extracting the characteristics of the sketch by utilizing the gray level image generator GGN and deconvoluting the characteristics of the sketch to generate a gray level image corresponding to the sketch;
and the color map generating unit is used for carrying out feature extraction on the gray map and the graffiti information of the sketch by utilizing a color map generator CGN, and carrying out feature deconvolution on the extracted gray map and the graffiti information to generate a corresponding color map.
In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:
(1) The sketch coloring method and the sketch coloring system based on the deep convolution generating countermeasure network skillfully solve the complex problem of directly learning from sketch to color drawing, divide the problem into two relatively simple stages and an imitation stage and a coloring stage, and make the two stages have their own roles, thereby greatly improving the final coloring result and being capable of obtaining smaller FID distance compared with the existing commercial coloring software PaintsChainer.
(2) According to the sketch coloring method and system based on the depth convolution generation countermeasure network, the self-encoder module is added into the GGN network in the simulation stage, so that the GGN can learn the mapping from the sketch to the gray level map well, and the final coloring effect is improved indirectly.
(3) According to the sketch coloring method and system based on the deep convolution generation countermeasure network, in the coloring stage CGN network, advanced features of the sketch are extracted by skillfully utilizing the training model and are input into the CGN, so that the CGN network not only obtains the features of the gray level image but also obtains the features of the sketch, and a coloring result is more accurate.
(4) According to the sketch coloring method and system for generating the countermeasure network based on the deep convolution, the GGN network and the CGN network are firstly trained gradually in the simulation stage, and then the GGN network and the CGN network are trained together in series, and the training mode can optimize the model and greatly reduce the training process.
Drawings
FIG. 1 is a flow chart of a sketch coloring method for generating an countermeasure network based on deep convolution, which is provided by the invention;
FIG. 2 is a framework and flow chart of a user-guided sketch coloring system provided by an embodiment of the present invention;
FIG. 3 is a block diagram of a user-guided sketch coloring system provided by an embodiment of the present invention;
fig. 4 is a schematic diagram of a system architecture for generating a countermeasure network based on deep convolution according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
A user-directed sketch coloring system and method for generating an countermeasure network based on deep convolution includes an imitation phase and a coloring phase. Extracting a characteristic vector of a sketch in an imitation stage, and converting the characteristic vector into a gray level diagram through generating a multi-antibody network; the coloring stage takes the gray-scale image generated in the first stage and the interaction of the user as input, and maps the color correspondence to the corresponding area according to the intention of the user to generate a realistic color image. According to the invention, the sketch coloring is reasonably divided into two simple sub-modules, and compared with the existing method, the FID distance is greatly reduced through experimental verification, wherein the FID is an important index for measuring the quality and diversity of image generation, and the smaller the FID value is, the better the image quality generated by the generator is.
In response to the above-identified deficiencies or improvements in the art, the present invention provides a system and method for user-interactive sketch coloring of a generating countermeasure network based on deep convolution. The method skillfully divides the complex problem of sketch coloring into two stages and a simulation stage and a coloring stage, wherein the simulation stage utilizes deep convolution to generate a mapping relation from a countermeasure network (GGN) learning sketch to a single-channel gray level image; the coloring stage uses depth convolution to generate a mapping relation between gray level images and color images generated by the antagonism network (CGN) learning simulation stage. The method aims to divide the complex problem of sketch coloring into two simple sub-problems and sequentially solve the problems of huge consumption of computing resources and time and incapability of accurate coloring in the conventional sketch coloring.
To achieve the above object, according to one aspect of the present invention, there is provided a method for generating user interaction sketch coloring of an countermeasure network based on deep convolution, which specifically includes two steps of an emulation phase and a coloring phase:
the simulation stage is used for extracting the characteristics of the sketch according to the sketch by utilizing the GGN, and then outputting a gray level diagram corresponding to the sketch through deconvolution.
The coloring stage is used for extracting the characteristics of the gray level image and the graffiti drawn by the user according to the gray level image generated in the imitation stage by using the CGN, and then generating a corresponding color image through deconvolution. In order to better help the CGN to color, the system inputs the features of the sketch to the CGN through a pre-training model for extracting the features of the sketch, so that the network is helped to achieve better training effect.
The CGN network and the CGN network are respectively used for generating the gray level image and the color image, and each of the CGN network and the CGN network consists of a generator and a discriminator.
The generator and the discriminator are basic frameworks for generating the countermeasure network, the generator is mainly used for generating specific problems, and the discriminator is mainly used for assisting the generation of the generator.
Preferably, the user interaction sketch coloring system comprises a self-encoder for encoding and decoding features of the gray-scale image in reality, wherein the self-encoder is mainly used for helping the GGN to improve the quality of generating the gray-scale image. Wherein the self-encoder loss function calculation formula is:
where g is the gray scale of the label, E G Is the coding part of the self-encoder, D G Is the decoding part of the self-encoder.
Preferably, in the user interaction sketch coloring system, after the feature vector of the gray-scale image is extracted from the encoder, the euclidean distance corresponding to the feature vector of the GGN extraction sketch is calculated as a loss function of the GGN, so that the feature vector of the GGN extraction sketch is forced to be similar to the feature of the gray-scale image as much as possible, and the GGN can generate the gray-scale image with good quality through deconvolution.
FIG. 1 is a flow chart of a sketch coloring method for generating an countermeasure network based on deep convolution, which is provided by the invention; as shown in fig. 1, the method comprises the following steps:
s100, extracting features of the sketch by using a gray scale generator GGN, and deconvoluting the features of the sketch to generate a gray scale corresponding to the sketch;
s110, feature extraction is carried out on the gray level map and the graffiti information of the sketch by using a color map generator CGN, and corresponding color maps are generated by feature deconvolution of the extracted gray level map and the graffiti information; the GGN and the CGN both belong to a deep convolution generating countermeasure network.
FIG. 2 is a general schematic of user interaction sketch coloring of a deep convolution-based countermeasure network in accordance with an embodiment of the present invention. As shown in fig. 2, the method includes two major stages: an emulation phase and a coloring phase. For the simulation stage and GGN network, the sketch is used as the input of GGN, and the gray-scale image is the output of GGN; for the coloring stage and the CGN network, the output of the first stage and the user interaction are used as the input of the CGN, and the output of the CGN is a colored color chart.
In particular, in fig. 2, s represents a sketch, c represents graffiti information of the sketch by the user,representing GGN generated gray-scale map, +.>Color representing CGN generationColor map. Wherein, the color picture->Is->The colored picture does not show coloring results because the figure cannot have a color chart, but as a person skilled in the art can understand the coloring condition of the color chart.
FIG. 3 is a detailed schematic diagram of user interaction sketch coloring of a deep convolution-based countermeasure network according to an embodiment of the present invention, and the specific steps are as follows:
s1, preparing a data set of a training network, downloading 30000 images of color characters (256 x 256 in size) from a public cartoon website, extracting outlines of the color images through an outline extraction algorithm XDoG, screening to obtain a better outline image as input of GGN, and converting the color image into a label of the GGN with the highest gray level image.
S2, training the GGN network in the simulation stage. Specifically, the discriminator of the GGN is first trained once, the label gray scale map and the GGN generated gray scale map are respectively input to the discriminator, a loss function is calculated, and the network parameters of the discriminator are updated by back propagation using a gradient descent algorithm. And training the GGN generator for five times, taking a sketch as an input of the generator, obtaining a loss function corresponding to the label data calculation after output, and carrying out back propagation to update network parameters through a gradient descent algorithm. This process is repeated until the loss function of the GGN converges.
The loss function calculation mode of the GGN network generator is as follows:
Loss GGN ＝L adv_GGN_g +L pix_GGN +L feat_GGN +L percep_GGN
wherein L is adv_GGN_g To combat losses, L pix_GGN Pixel loss, L, of gray-scale image and label gray-scale image generated for GGN feat_GGN The feature vector loss is aimed at enabling the feature vector of the GGN extracted sketch to be similar to the feature vector of the label gray-scale image corresponding to the extracted sketch of the self-encoder as much as possible, thereby achieving the purpose of enabling the gray-scale image generated by the GGN to be vivid as much as possible, L percep_GGN For sensory loss, the aim is also to make the generated grey-scale image as realistic as possible. In addition, s represents sketch, g represents label gray scale corresponding to s,represents a GGN generated gray-scale image, N l And phi is l The number of pixels and the first layer characteristic diagram of the pretraining model VGG16 are represented respectively; dis (Dis) s And Gen s Discriminator and generator, E, respectively representing GGNs s And E is G Representing the coding part of the self-encoder and the coding part of the generator, respectively.
The loss function calculation mode of the GGN network discriminator is as follows:
wherein s represents sketch, g represents label gray level diagram corresponding to s, dis s And Gen s Respectively represent a discriminator and a generator of the GGN.
S3, training the CGN network of the coloring stage. Specifically, firstly training a discriminator of a CGN network once, inputting a color chart and a label color chart generated by the CGN into the discriminator, calculating a loss function, and carrying out back propagation to update network parameters through a gradient descent algorithm; and training the CGN network generator for five times, taking the interactive graph of the user and the pre-training model which are connected with the gray graph generated in the simulation stage as the input of the CGN network generator, obtaining the output, calculating a loss function with the label, and carrying out counter propagation by utilizing a gradient descent algorithm to update network parameters. This process is repeated until the loss function of the GGN converges.
The construction process of the user interaction is as follows: and randomly digging 60 pixel blocks with the size of 10 x 10 in the label color image, and then carrying out Gaussian blur to obtain a color plate which is used as a user interaction image in the training process.
The loss function calculation mode of the CGN network generator is as follows:
Loss CGN ＝L adv_CGN_g +L pix_CGN +L percep_CGN
wherein L is adv_CGN_g To combat losses, L pix_CGN Pixel loss, L, of color map and label color map generated for CGN percep_CGN For sensory loss, the aim is also to promote the realism of the generated colour map as much as possible. In addition, wherein y represents a label color map,representing gray level diagram generated by GGN network in simulation stage, s representing sketch, c representing graffiti information of user, N l And phi is l The number of pixels and the first layer characteristic diagram of the pretraining model VGG16 are represented respectively; dis (Dis) c And Gen c Respectively representing a discriminator and a generator of the CGN.
The loss function calculation mode of the CGN network discriminator is as follows:
wherein y represents a label color chart,representing gray-scale patterns generated by GGN network in simulation stage, s representing sketch, c representing user interaction, dis c And Gen c Respectively representing a discriminator and a generator of the CGN.
S4, a joint training stage. And connecting the GGN network and the CGN network in series, inputting the GGN network and the CGN network into a sketch, extracting a characteristic diagram of the sketch through a training model, and outputting the characteristic diagram as a color diagram. And calculating a corresponding loss function, and carrying out back propagation updating on network parameters by using a gradient descent algorithm until the loss function converges.
The user interaction sketch coloring algorithm for generating the countermeasure network based on the depth clipping can realize coloring according to user interaction. After model training, 1000 pictures which are not seen by the model are adopted as a test set to be compared with a coloring model which is commonly used at present, and the results are shown in table 1:
TABLE 1 color quality comparison results
| FID | |
| PaintsChainer(canna) | 164.25 |
| PaintsChainer(tanpopo) | 168.13 |
| PaintsChainer(satsuki) | 148.12 |
| The coloring method of the invention | 108.85 |
Table 1 shows FID distance comparison between the coloring method of the invention and three versions of commercial software PaintsChainer, wherein FID is an important index for measuring the quality and diversity of pictures, and the smaller the FID value, the better the quality and diversity of the pictures are represented. It can be seen that the coloring method of the present invention can greatly reduce the FID distance compared with the conventional coloring method, that is, the coloring method of the present invention can achieve higher coloring quality.
FIG. 4 is a schematic diagram of a system architecture for generating a countermeasure network based on deep convolution, as shown in FIG. 4, including:
a gray map generating unit 410, configured to extract features of the sketch by using the gray map generator GGN, and deconvolve the features of the sketch to generate a gray map corresponding to the sketch;
and the color map generating unit 420 is configured to perform feature extraction on the gray map and the graffiti information of the sketch by using the color map generator CGN, and deconvolve features of the extracted gray map and the graffiti information to generate a corresponding color map.
The functions of the specific units in fig. 4 are referred to in the foregoing method embodiment, and are not described herein.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (8)
1. A method for generating sketch coloring of an countermeasure network based on deep convolution, comprising the steps of:
s100, extracting features of the sketch by using a gray scale generator GGN, and deconvoluting the features of the sketch to generate a gray scale corresponding to the sketch;
s110, feature extraction is carried out on the gray level map and the graffiti information of the sketch by using a color map generator CGN, and corresponding color maps are generated by feature deconvolution of the extracted gray level map and the graffiti information; the GGN and the CGN both belong to a deep convolution generating countermeasure network;
the GGN includes a generator, a discriminator, and a self-encoder; the identifier and the self-encoder in the GGN are used for assisting the generator in the GGN to generate a gray map based on the sketch; the identifier in the GGN is used for identifying the quality of the gray scale map generated by the generator so as to generate the optimal quality of the gray scale map; the self-encoder is used for encoding and decoding the characteristics of the gray level map, after the characteristic vector of the gray level map is extracted by the self-encoder, the Euclidean distance corresponding to the characteristic vector of the GGN extracted sketch is calculated and used as a loss function of the GGN, so that the characteristic vector of the GGN extracted sketch is similar to the characteristics of the gray level map as much as possible, and the GGN can generate the gray level map with optimal quality through deconvolution;
the CGN comprises a generator and a discriminator; the CGN generator is used for extracting features of the graffiti information of the sketch according to the gray level graph generated by the GGN and the user, and deconvoluting the extracted features of the gray level graph and the graffiti information to generate a corresponding color graph; the discriminator in the CGN is used for assisting in discriminating the color map generated by the generator in the CGN, so that the coloring result of the generator in the CGN is optimal.
2. The sketch coloring method according to claim 1, wherein Loss of generator Loss function in the GGN _GGN The calculation formula is as follows:
Loss _GGN ＝L adv_GGN_g +L pix_GGN +L feat_GGN +L percep_GGN
wherein L is adv_GGN_g To combat loss in GGN, L pix_GGN Pixel loss, L, of gray-scale image and label gray-scale image generated for GGN feat_GGN L is a characteristic loss in GGN feat_GGN The purpose is to make the feature vector of GGN extracted sketch and the feature vector of label gray-scale image corresponding to the extracted sketch from encoder as similar as possible, thereby making the gray-scale image generated by GGN as lifelike as possible, L percep_GGN For sensory loss in GGN, L percep_GGN The aim is to promote the generated gray level image to be as lifelike as possible; s represents sketch, g represents label gray scale corresponding to s,representing a gray scale map generated by the GGN; />Representing the data distribution P corresponding to the sketch s and the label gray-scale g data A set of sketches and grey-scale patterns randomly extracted from (s, g), and +.>Represents gray-scale patterns generated from GGN +.>Data distribution corresponding to the label gray map g +.>A group of sketches and gray-scale images randomly extracted from the matrix, N l And phi is l The number of pixels and the first layer characteristic diagram of the pretraining model VGG16 are represented respectively; dis (Dis) s And Gen s Discriminator and generator, E, respectively representing GGNs s And E is G Representing the coding part of the self-encoder and the coding part of the generator, respectively.
3. The sketch coloring method according to claim 2, wherein a discriminator loss function L in the GGN adv_GGN_d The calculation formula of (2) is as follows:
4. the sketch coloring method according to claim 2, wherein the self-encoder loss function L in the GGN AE The calculation formula of (2) is as follows:
where g is the gray scale of the label, E G Is the coding part of the self-encoder, D G Is the decoding part of the self-encoder.
5. The sketch coloring method according to claim 1, wherein a generator Loss function Loss in the CGN _CGN The calculation formula of (2) is as follows:
Loss _CGN ＝L adv_CGN_g +L pix_CGN +L percep_CGN
wherein L is adv_CGN_g To combat losses in CGN, L pix_CGN Pixel loss, L, of color map and label color map generated for CGN percep_CGN For sensory loss in CGN, L percep_CGN The purpose is to promote the generated color map to be as lifelike as possible; y represents the color map of the label,representing a gray scale map generated by a GGN network, s representing a sketch, c representing graffiti information of a user on the sketch,/for the sketch>Representing the color map generated by the CGN generator, < >>Representing the slave data distribution->A set of data randomly extracted in +.>Representing the slave data distribution->A set of data, N, randomly extracted l And phi is l The number of pixels and the first layer characteristic diagram of the pretraining model VGG16 are represented respectively; dis (Dis) c And Gen c Respectively representing a discriminator and a generator of the CGN.
6. The sketch coloring method according to claim 5, wherein a discriminator loss function L in the CGN adv_CGN_d The calculation formula of (2) is as follows:
7. the sketch coloring method according to any one of claims 1 to 6, wherein the GGN and the CGN are trained by:
taking the sketch as the input of the GGN, calculating the current loss function, and then carrying out back propagation by a gradient descent algorithm until the loss function is minimized to enable the GGN model to be converged;
taking a gray level image output by the GGN network and corresponding user graffiti information as the input of the CGN, calculating a current loss function, and then carrying out back propagation by a gradient descent algorithm until the loss function is minimized to enable the CGN model to be converged;
and performing joint training on the GGN and the CGN to ensure that the effect of the model is optimal.
8. A sketch-coloring system for generating an countermeasure network based on deep convolution, comprising:
the gray level image generating unit is used for extracting the characteristics of the sketch by utilizing the gray level image generator GGN and deconvoluting the characteristics of the sketch to generate a gray level image corresponding to the sketch; the GGN includes a generator, a discriminator, and a self-encoder; the identifier and the self-encoder in the GGN are used for assisting the generator in the GGN to generate a gray map based on the sketch; the identifier in the GGN is used for identifying the quality of the gray scale map generated by the generator so as to generate the optimal quality of the gray scale map; the self-encoder is used for encoding and decoding the characteristics of the gray level map, after the characteristic vector of the gray level map is extracted by the self-encoder, the Euclidean distance corresponding to the characteristic vector of the GGN extracted sketch is calculated and used as a loss function of the GGN, so that the characteristic vector of the GGN extracted sketch is similar to the characteristics of the gray level map as much as possible, and the GGN can generate the gray level map with optimal quality through deconvolution;
the color map generating unit is used for carrying out feature extraction on the gray map and the graffiti information of the sketch by utilizing a color map generator CGN, and deconvoluting the extracted features of the gray map and the graffiti information to generate a corresponding color map; the CGN comprises a generator and a discriminator; the CGN generator is used for extracting features of the graffiti information of the sketch according to the gray level graph generated by the GGN and the user, and deconvoluting the extracted features of the gray level graph and the graffiti information to generate a corresponding color graph; the discriminator in the CGN is used for assisting in discriminating the color map generated by the generator in the CGN, so that the coloring result of the generator in the CGN is optimal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010672560.7A CN111862253B (en) | 2020-07-14 | 2020-07-14 | Sketch coloring method and system for generating countermeasure network based on deep convolution |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010672560.7A CN111862253B (en) | 2020-07-14 | 2020-07-14 | Sketch coloring method and system for generating countermeasure network based on deep convolution |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111862253A CN111862253A (en) | 2020-10-30 |
| CN111862253B true CN111862253B (en) | 2023-09-15 |
Family
ID=72984733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010672560.7A Active CN111862253B (en) | 2020-07-14 | 2020-07-14 | Sketch coloring method and system for generating countermeasure network based on deep convolution |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111862253B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112991371B (en) * | 2021-04-20 | 2024-01-05 | 云南大学 | Automatic image coloring method and system based on coloring overflow constraint |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001043361A (en) * | 1999-08-03 | 2001-02-16 | Yokogawa Electric Corp | Image processing method and image processing using the same |
| AU2018100325A4 (en) * | 2018-03-15 | 2018-04-26 | Nian, Xilai MR | A New Method For Fast Images And Videos Coloring By Using Conditional Generative Adversarial Networks |
| CN107992572A (en) * | 2017-11-30 | 2018-05-04 | 天津大学 | A kind of distributed graph coloring algorithm based on Pregel |
| CN108717698A (en) * | 2018-05-28 | 2018-10-30 | 深圳市唯特视科技有限公司 | A kind of high quality graphic generation method generating confrontation network based on depth convolution |
| CN109712203A (en) * | 2018-12-29 | 2019-05-03 | 福建帝视信息科技有限公司 | A kind of image rendering methods based on from attention generation confrontation network |
| CN110084751A (en) * | 2019-04-24 | 2019-08-02 | 复旦大学 | Image re-construction system and method |
| CN110111236A (en) * | 2019-04-19 | 2019-08-09 | 大连理工大学 | The method for generating image based on the multiple target sketch that gradual confrontation generates network |
| CN110223359A (en) * | 2019-05-27 | 2019-09-10 | 浙江大学 | It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation |
| CN110569761A (en) * | 2019-08-27 | 2019-12-13 | 武汉大学 | Method for retrieving remote sensing image by hand-drawn sketch based on counterstudy |
| CN111080522A (en) * | 2019-12-13 | 2020-04-28 | 福州大学 | Image super-resolution reconstruction method based on bidirectional countermeasure network |
| CN111402179A (en) * | 2020-03-12 | 2020-07-10 | 南昌航空大学 | Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10621764B2 (en) * | 2018-07-05 | 2020-04-14 | Adobe Inc. | Colorizing vector graphic objects |
-
2020
- 2020-07-14 CN CN202010672560.7A patent/CN111862253B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001043361A (en) * | 1999-08-03 | 2001-02-16 | Yokogawa Electric Corp | Image processing method and image processing using the same |
| CN107992572A (en) * | 2017-11-30 | 2018-05-04 | 天津大学 | A kind of distributed graph coloring algorithm based on Pregel |
| AU2018100325A4 (en) * | 2018-03-15 | 2018-04-26 | Nian, Xilai MR | A New Method For Fast Images And Videos Coloring By Using Conditional Generative Adversarial Networks |
| CN108717698A (en) * | 2018-05-28 | 2018-10-30 | 深圳市唯特视科技有限公司 | A kind of high quality graphic generation method generating confrontation network based on depth convolution |
| CN109712203A (en) * | 2018-12-29 | 2019-05-03 | 福建帝视信息科技有限公司 | A kind of image rendering methods based on from attention generation confrontation network |
| CN110111236A (en) * | 2019-04-19 | 2019-08-09 | 大连理工大学 | The method for generating image based on the multiple target sketch that gradual confrontation generates network |
| CN110084751A (en) * | 2019-04-24 | 2019-08-02 | 复旦大学 | Image re-construction system and method |
| CN110223359A (en) * | 2019-05-27 | 2019-09-10 | 浙江大学 | It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation |
| CN110569761A (en) * | 2019-08-27 | 2019-12-13 | 武汉大学 | Method for retrieving remote sensing image by hand-drawn sketch based on counterstudy |
| CN111080522A (en) * | 2019-12-13 | 2020-04-28 | 福州大学 | Image super-resolution reconstruction method based on bidirectional countermeasure network |
| CN111402179A (en) * | 2020-03-12 | 2020-07-10 | 南昌航空大学 | Image synthesis method and system combining countermeasure autoencoder and generation countermeasure network |
Non-Patent Citations (2)
| Title |
|---|
| 《基于感知损失的深度卷积网络的草图简化研究》;谢敏珊;中国优秀硕士学位沦为全文数据库（电子期刊）;信息科技辑卷;第11-24页 * |
| Image-to-Image Translation Using Generative Adversarial Network;K. Lata, M. Dave and K. N. Nishanth;Communication and Aerospace Technology;186-189 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111862253A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111145116B (en) | Sea surface rainy day image sample augmentation method based on generation of countermeasure network | |
| CN110717977B (en) | Method, device, computer equipment and storage medium for processing game character face | |
| CN108875935B (en) | Natural image target material visual characteristic mapping method based on generation countermeasure network | |
| CN109859288B (en) | Image coloring method and device based on generation countermeasure network | |
| CN108830913B (en) | Semantic level line draft coloring method based on user color guidance | |
| CN110222722A (en) | Interactive image stylization processing method, calculates equipment and storage medium at system | |
| CN109829868B (en) | Lightweight deep learning model image defogging method, electronic equipment and medium | |
| CN113807265B (en) | Diversified human face image synthesis method and system | |
| CN110717953A (en) | Black-white picture coloring method and system based on CNN-LSTM combined model | |
| CN112489164B (en) | Image coloring method based on improved depth separable convolutional neural network | |
| CN113255813B (en) | Multi-style image generation method based on feature fusion | |
| CN112995534B (en) | Video generation method, device, equipment and readable storage medium | |
| CN113724354B (en) | Gray image coloring method based on reference picture color style | |
| CN107730568B (en) | Coloring method and device based on weight learning | |
| CN112529774A (en) | Remote sensing simulation image generation method based on cycleGAN | |
| CN111862253B (en) | Sketch coloring method and system for generating countermeasure network based on deep convolution | |
| CN110097615B (en) | Stylized and de-stylized artistic word editing method and system | |
| CN115018729B (en) | Content-oriented white box image enhancement method | |
| CN115082800B (en) | Image segmentation method | |
| CN112464924A (en) | Method and device for constructing training set | |
| CN114917583A (en) | Animation style game background generation method and platform based on generation confrontation network | |
| CN113129409B (en) | Cartoon line manuscript coloring method based on deep learning | |
| Zhang et al. | Animation Costume Style Migration Based on CycleGAN | |
| CN111062862A (en) | Color-based data enhancement method and system, computer device and storage medium | |
| CN111260570A (en) | Binarization background noise simulation method for posts based on cycle consistency confrontation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |