CN116580121A - Method and system for generating 2D model by single drawing based on deep learning - Google Patents

Method and system for generating 2D model by single drawing based on deep learning Download PDF

Info

Publication number
CN116580121A
CN116580121A CN202310561085.XA CN202310561085A CN116580121A CN 116580121 A CN116580121 A CN 116580121A CN 202310561085 A CN202310561085 A CN 202310561085A CN 116580121 A CN116580121 A CN 116580121A
Authority
CN
China
Prior art keywords
image
model
generator
input
generating
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310561085.XA
Other languages
Chinese (zh)
Other versions
CN116580121B (en
Inventor
甘凌
顾大桐
王步国
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Yuanyue Technology Co ltd
Original Assignee
Beijing Yuanyue Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Yuanyue Technology Co ltd filed Critical Beijing Yuanyue Technology Co ltd
Priority to CN202310561085.XA priority Critical patent/CN116580121B/en
Publication of CN116580121A publication Critical patent/CN116580121A/en
Application granted granted Critical
Publication of CN116580121B publication Critical patent/CN116580121B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/20—Drawing from basic elements, e.g. lines or circles
    • G06T11/206—Drawing of charts or graphs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

BaseThe invention relates to a method and a system for generating a 2D model by a single drawing based on deep learning, which comprises the following steps: acquiring a plurality of pictures, classifying the pictures into a plurality of data sets, and preprocessing the data sets; image cutting is carried out on the images in the preprocessed data set, and an input image is obtainedxAnd a target imageyThe method comprises the steps of carrying out a first treatment on the surface of the Establishing a 2D generation model, and training the 2D generation model by taking the data set after image cutting as a training sample; optimizing and testing the trained 2D generation model; and inputting the single drawing as an input image into the tested 2D generation model, and performing post-processing operation on the 2D model output by the 2D generation model. The invention can generate a specific 2D model based on the single Zhang Huihua image and can be applied to a plurality of fields such as movie animation special effects, game scenes, virtual fitting and the like.

Description

Method and system for generating 2D model by single drawing based on deep learning
Technical Field
The invention relates to the technical field of drawing modeling, in particular to a method and a system for generating a 2D model by a single drawing based on deep learning.
Background
Deep learning is a machine learning method, training is performed by constructing a multi-layer neural network, and tasks such as classification, regression and the like are automatically extracted from data.
At present, a drawing generation algorithm has become one of research hotspots in the field of artificial intelligence. By utilizing the deep learning technology, high-quality and diversified drawing generation can be realized through learning and analysis of a large number of artwork samples, and the method has a wide application prospect.
In the existing painting generation methods, although models based on a variational self-encoder (VAE), a countermeasure generation network (GAN) and the like are studied, the methods still have a certain limitation in terms of generating sense of reality and diversity due to the structural complexity and more detail textures of a single painting image.
Disclosure of Invention
In order to overcome the technical defects in the prior art, the invention provides a method and a system for generating a 2D model by using a single drawing based on deep learning, which can effectively solve the problems in the background art.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
in a first aspect, an embodiment of the present invention discloses a method for generating a 2D model from a single drawing based on deep learning, including the following steps:
acquiring a plurality of pictures, classifying the pictures into a plurality of data sets, and preprocessing the data sets;
image cutting is carried out on the images in the preprocessed data set, and an input image x and a target image y are obtained;
establishing a 2D generation model, and training the 2D generation model by taking the data set after image cutting as a training sample;
optimizing and testing the trained 2D generation model;
and inputting the single drawing as an input image into the tested 2D generation model, and performing post-processing operation on the 2D model output by the 2D generation model.
In any of the foregoing aspects, preferably, the step of acquiring a plurality of drawings, classifying the plurality of drawings into a plurality of data sets, and preprocessing the data sets includes the steps of:
performing picture scaling, image enhancement and data cleaning on the data set;
performing edge detection on the image in the data set, and extracting the outline of the painting work;
and extracting color features of the pictorial representation according to the outline of the pictorial representation.
In any of the above aspects, preferably, the edge detection of the image in the dataset, and the extraction of the outline of the pictorial representation, include:
smoothing the image by a gaussian filter using the formulaSmoothing the image by a two-dimensional gaussian function, wherein the parameters of the distribution by the gaussian functionσThe degree of smoothing the image is controlled,σthe smaller the filter, the higher the positioning accuracy of the filter, the lower the signal-to-noise ratio, and vice versa;
by the formula:
；
；
；
computing gradient magnitude for each point in pictorial representation IGAnd gradientθDirection of (a) whereinG x (i,j) AndG y (i,j) Respectively is a point%i,j) At the position ofx,yPartial derivative of direction;
in the way ofi,j) As the center point of the field, the field isθ(i,j) Gradient value of each point in the directionG (i,j) Comparing, and taking the point where the gradient maximum value is locatedi,j) As candidate edge points, or else, non-edge points, obtaining a candidate edge image K;
setting a high thresholdT h And a low thresholdT l Any point of the obtained candidate edge pointsi,j) Detecting if it is%i,j) Gradient value of the outputG (i,j)＞T h If the store is determined to be an edge point, ifG (i,j)＜T l The point is not an edge point; if it isT l ＜G (i,j)＜T h Judging whether an edge point exists in the field of the point, if so, judging that the point is the edge point, otherwise, judging that the point is not the edge point.
In any of the above aspects, preferably, the extracting the color features of the pictorial representation according to the outline of the pictorial representation includes:
by the formula:
；
；
the method comprises the steps of carrying out a first treatment on the surface of the Calculating first moments of colors in pictorial representationsμ ci Second moment ofσ ci Moment of third orderξ ci The method comprises the steps of carrying out a first treatment on the surface of the Wherein, Nfor the number of pixels in the image,P ci for a color value ofCAnd the color component isiIs used to determine the probability of the occurrence of a pixel,Ca number of colors included in the pictorial representation; each color has three components, each component having a third order moment;
by the formula:
P C =(μ cr , σ cr , ξ cr , μ cg , σ cg , ξ cg , μ cb , σ cb , ξ cb )，generating a set of color features from the color features of each pictorial representationP C 。
In any of the above aspects, preferably, the image in the preprocessed dataset is subjected to image segmentation to obtain an input imagexAnd a target imageyComprising the following steps:
from [0.5,1]Randomly selecting a scaling factor within a range of (a)sAnd zoom the image tosX original resolution;
randomly selecting a window with the width and the height being the original resolution from the zoomed image as a cutting area;
cutting the image into a plurality of small blocks according to the cutting area, wherein the image is inputxFor small block image after random cutting, target imageyIs a small block image at the same position as the input image x in the corresponding original image.
In any of the above schemes, preferably, the establishing a 2D generating model and training the 2D generating model by using the data set after image cutting as a training sample includes the following steps:
building a generator G and a discriminator D, and building a 2D generation model;
defining a loss function comprising a generator loss and a arbiter loss;
based on the data set after image cutting, the loss function is lowered through an alternate training generator and a discriminator, and training of the 2D generation model is completed.
In any of the above schemes, preferably, the generator is defined as:
w=MLP(z)ϵR dlatent ，(w,y 1 )=G proj (w,y 1 )ϵR dlatent ，x=G (w,y 1 )ϵR C*H*W where z represents the input low-dimensional noise vector,MLPrepresenting a multi-layer perceptron for vector of noise inputzMapping to style vectors in potential spacew，dlatentRepresenting the dimensions of the potential space,G proj representing a learnable projection layer for vector of patternswAnd condition information y 1 Merging and generating a new dimension vectorw, y 1 ) G represents a Generator model composed of a plurality of Generator Blocks for inputting pattern vectors #w, y 1 ) Conversion to high resolution imagesx 1 ，C、HAndWthe number, height and width of channels representing the image respectively;
the discriminant is defined as:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein, xrepresenting the input image, obtaining feature images with different resolutions after convolution and downsampling processing,S i (x) Representing an input imagexCarry out the first stepiThe feature map obtained after the layer convolution and downsampling processes,D i represent the firstiLayer discriminant model forFor the firstiThe feature map of the layer is classified and scored,a i representing the corresponding discriminant D i And the weight coefficient of the classification module is used for controlling the contribution degree of each layer to the final classification result.
In any of the above schemes, preferably, the generator loss is:
L G =-E w y pdat(,1)~ a[logD(G (w,y 1 ))]wherein, the method comprises the steps of, wherein,pdatarepresenting the distribution of potential vectors and condition information in the input training set,Drepresenting a discriminant model for evaluating whether an input image is a true image, logD(G(w,y 1 ) A) represents the generatorGThe image produced is taken as input and calculated to pass through the discriminatorDThe logarithm of the probability of the post-output,Erepresenting the operation of summing and averaging the potential vectors and condition information in all input training sets,L G representation generatorGA loss function of (2);
the loss of the discriminator is as follows:
wherein, the method comprises the steps of, wherein,pdata(x) The distribution of the real image is represented,pG(z,y) Representation generatorGDistribution of the generated image, whereinzThe noise vector representing the input is represented by a vector,y 1 the condition information is represented by a set of conditions,D(x) Representing a real image as an input imagexAfter input, the probability of the output of the discriminator;D(G(z,y) A) represents the generatorGThe probability of the output of the arbiter after the generated image is taken as input,aandbrepresenting the threshold at which the arbiter attempts to distinguish between the real image and the generated image,Erepresenting the desire, representing the operation of summing and averaging all the input real images and the generated images,L D representation discriminatorDIs a function of the loss of (2).
In any of the above schemes, preferably, the training of the 2D generative model is completed by alternately training the generator and the discriminator to lower the loss function based on the data set after image cutting, and the method includes the following steps:
initialization generator and arbiter: initializing a generator G and a discriminator D as random functions;
preparing training data: randomly sampling mini-batch data from a real data set, wherein half of the mini-batch data are real images, and the other half of the mini-batch data are false images generated by a generator G;
forward propagation and backward propagation: the sampled mini-batch data are input into a generator G to obtain false images, and the true and false images are respectively input into a discriminator D to calculate discrimination probabilities. Then back-propagating and updating parameters of the generator and the discriminator according to the loss function;
calculating a loss function: during the forward and backward propagation, the value of the loss function needs to be calculated and recorded;
repeating the initializing generator and the arbiter to the calculating the loss function to drop the loss function by alternately training the generator and the arbiter.
In a second aspect, a system for generating a 2D model based on deep learning of a single drawing, the system comprising:
the acquisition module is used for acquiring a plurality of pictures classified into a plurality of data sets and preprocessing the data sets;
the processing module is used for cutting the image in the preprocessed data set to obtain an input image x and a target image y;
the generating module is used for establishing a 2D generating model, and training the 2D generating model by taking the data set after image cutting as a training sample;
the optimization module is used for optimizing and testing the trained 2D generation model;
the input module is used for inputting the single drawing as an input image into the tested 2D generation model and performing post-processing operation on the 2D model output by the 2D generation model.
Compared with the prior art, the invention has the beneficial effects that:
the method for generating the 2D model by using the single drawing based on the deep learning can generate a specific 2D model based on the single Zhang Huihua image, and can be applied to a plurality of fields such as movie animation special effects, game scenes, virtual fitting and the like;
by introducing the pattern noise layer and the projection layer, the diversity of the generator and the image detail are increased, and the sense of reality and the artistry of the generated image are improved;
regularization technology and super-parameter adjustment optimization are also applied in model training, so that stability and image quality of the generator can be further improved.
Drawings
The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification.
FIG. 1 is a flow chart of a method of generating a 2D model from a single drawing based on deep learning in accordance with the present invention;
FIG. 2 is a block diagram of a system for generating a 2D model based on a single drawing for deep learning in accordance with the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It will be understood that when an element is referred to as being "mounted" or "disposed" on another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element.
In the description of the present invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate describing the present invention and simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
In order to better understand the above technical scheme, the following detailed description of the technical scheme of the present invention will be given with reference to the accompanying drawings of the specification and the specific embodiments.
As shown in fig. 1, the invention provides a method for generating a 2D model by a single drawing based on deep learning, which comprises the following steps:
step 1, acquiring a plurality of paintings classified into a plurality of data sets, and preprocessing the data sets;
step 2, image cutting is carried out on the images in the preprocessed data set, and an input image x and a target image y are obtained;
step 3, a 2D generation model is established, and the data set after image cutting is used as a training sample to train the 2D generation model;
step 4, optimizing and testing the trained 2D generation model;
and 5, inputting the single drawing as an input image into the tested 2D generation model, and performing post-processing operation on the 2D model output by the 2D generation model.
Specifically, step 1 obtains a plurality of drawings classified into a plurality of data sets, and preprocesses the data sets, including the steps of:
step 11, performing picture scaling, image enhancement and data cleaning on the data set;
step 12, carrying out edge detection on the image in the data set, and extracting the outline of the painting work;
and 13, extracting color features of the painting according to the outline of the painting.
Specifically, step 12, performing edge detection on the image in the dataset, and extracting the outline of the pictorial representation, including:
step 121, smoothing the image by a Gaussian filter using the formulaSmoothing the image by a two-dimensional gaussian function, wherein the parameters of the distribution by the gaussian functionσThe degree of smoothing the image is controlled,σthe smaller the filter, the higher the positioning accuracy of the filter, the lower the signal-to-noise ratio, and vice versa;
step 122, by the formula:
；
；
；
computing gradient magnitude for each point in pictorial representation IGAnd gradientθDirection of (a) whereinG x (i,j) AndG y (i,j) Respectively is a point%i,j) At the position ofx,yPartial derivative of direction;
step 123, using points @ to makei,j) As the center point of the field, the field isθ(i,j) Gradient value of each point in the directionG(i,j) Comparing, and taking the point where the gradient maximum value is locatedi,j) As candidate edge points, or else, non-edge points, obtaining a candidate edge image K;
step 124, setting a high thresholdT h And a low thresholdT l Any point of the obtained candidate edge pointsi,j) Detecting if it is%i,j) Gradient value of the outputG (i,j)＞T h If the store is determined to be an edge point, ifG (i,j)＜T l The point is not an edge point; if it isT l ＜G (i,j)＜T h Judging whether an edge point exists in the field of the point, if so, judging that the point is the edge point, otherwise, judging that the point is not the edge point.
Specifically, step 13 extracts color features of the pictorial representation according to the outline of the pictorial representation, including:
step 131, by the formula:
；
；
the method comprises the steps of carrying out a first treatment on the surface of the Calculating first moments of colors in pictorial representationsμ ci Second moment ofσ ci Moment of third orderξ ci The method comprises the steps of carrying out a first treatment on the surface of the Wherein, Nfor the number of pixels in the image,P ci for a color value ofCAnd the color component isiIs used to determine the probability of the occurrence of a pixel,Ca number of colors included in the pictorial representation; each color has three components, each component having a third order moment;
by the formula:
P C =(μ cr , σ cr , ξ cr , μ cg , σ cg , ξ cg , μ cb , σ cb , ξ cb )，generating a set of color features from the color features of each pictorial representationP C 。
Specifically, in step 2, the image in the preprocessed dataset is subjected to image cutting to obtain an input imagexAnd a target imageyComprising the following steps:
step 21, from [0.5,1]Randomly selecting a scaling factor within a range of (a)sAnd zoom the image tosX original resolution;
step 22, randomly selecting a window with the width and the height being the original resolution from the zoomed image as a cutting area;
step 23, cutting the image into a plurality of small blocks according to the cutting area, wherein the image is inputxFor small block image after random cutting, target imageyIs the corresponding original image and input imagexSmall images at the same positions.
Specifically, in the step 3, a 2D generation model is built, and the 2D generation model is trained by taking the data set after image cutting as a training sample, including the following steps:
step 31, building a 2D generation model by building a generator G and a discriminator D;
step 32, defining a loss function, including generator loss and arbiter loss;
and step 33, based on the data set after image cutting, the loss function is lowered by alternating the training generator and the discriminator, so that the training of the 2D generation model is completed.
Further, the generator is defined as:
w=MLP(z)ϵR dlatent ，(w,y 1 )=G proj (w,y 1 )ϵR dlatent ，x=G (w,y 1 )ϵR C*H*W where z represents the input low-dimensional noise vector,MLPrepresenting a multi-layer perceptron forTo be input noise vectorzMapping to style vectors in potential spacew，dlatentRepresenting the dimensions of the potential space,G proj representing a learnable projection layer for vector of patternswAnd condition information y 1 Merging and generating a new dimension vectorw, y 1 ) G represents a Generator model composed of a plurality of Generator Blocks for inputting pattern vectors #w, y 1 ) Conversion to high resolution imagesx 1 ，C、HAndWthe number, height and width of channels representing the image respectively;
further, the discriminant is defined as:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein, xrepresenting the input image, obtaining feature images with different resolutions after convolution and downsampling processing,S i (x) Representing an input imagexCarry out the first stepiThe feature map obtained after the layer convolution and downsampling processes,D i represent the firstiA layer discriminant model for the firstiThe feature map of the layer is classified and scored,a i representing the corresponding discriminant D i And the weight coefficient of the classification module is used for controlling the contribution degree of each layer to the final classification result.
Further, the generator penalty is:
L G =-E w y pdat(,1)~ a[logD(G (w,y 1 ))]wherein, the method comprises the steps of, wherein,pdatarepresenting the distribution of potential vectors and condition information in the input training set,Drepresenting a discriminant model for evaluating whether an input image is a true image, logD(G(w,y 1 ) A) represents the generatorGThe image produced is taken as input and calculated to pass through the discriminatorDThe logarithm of the probability of the post-output,Erepresenting potential vectors and condition information in all input training setsThe operation of summing and averaging is performed,L G representation generatorGA loss function of (2);
further, the discriminator loss is:
wherein, the method comprises the steps of, wherein,pdata(x) The distribution of the real image is represented,pG(z,y) Representation generatorGDistribution of the generated image, whereinzThe noise vector representing the input is represented by a vector,y 1 the condition information is represented by a set of conditions,D(x) Representing a real image as an input imagexAfter input, the probability of the output of the discriminator;D(G(z,y) A) represents the generatorGThe probability of the output of the arbiter after the generated image is taken as input,aandbrepresenting the threshold at which the arbiter attempts to distinguish between the real image and the generated image,Erepresenting the desire, representing the operation of summing and averaging all the input real images and the generated images,L D representation discriminatorDIs a function of the loss of (2).
Specifically, step 33, based on the data set after image cutting, through an alternate training generator and a discriminator, makes the loss function descend, and completes the training of the 2D generation model, and includes the following steps:
step 331, initializing a generator and a discriminator: initializing a generator G and a discriminator D as random functions;
step 332, preparing training data: randomly sampling mini-batch data from a real data set, wherein half of the mini-batch data are real images, and the other half of the mini-batch data are false images generated by a generator G;
step 333, forward propagation and backward propagation: the sampled mini-batch data are input into a generator G to obtain false images, and the true and false images are respectively input into a discriminator D to calculate discrimination probabilities. Then back-propagating and updating parameters of the generator and the discriminator according to the loss function;
step 334, calculate the loss function: during the forward and backward propagation, the value of the loss function needs to be calculated and recorded;
step 335, repeating the initializing generator and the arbiter to the calculating the loss function to drop the loss function by alternately training the generator and the arbiter.
Specifically, step 4 optimizes and tests the trained 2D generation model, including:
step 41, adjusting and optimizing the super parameters;
step 42, using regularization technique to avoid overfitting;
step 43, introducing a pattern noise layer and a projection layer into the generator, and increasing the generation diversity and image details;
step 44, testing the model by using the test set, and evaluating the effect of generating a vivid 2D model;
the generated image is evaluated 45 using various quality metrics such as diversity, realism, sharpness, etc.
Specifically, step 5 inputs the sheet Zhang Huihua as an input image into the tested 2D generation model, and performs post-processing operation on the 2D model output by the 2D generation model, including:
the vector z of the single drawing is input into the generator G, the generated realistic 2D image is output, and post-processing such as denoising, brightness adjustment, etc. is performed to improve the realism and artistry thereof.
As shown in fig. 2, the present invention further provides a system for generating a 2D model from a single drawing based on deep learning, the system comprising:
the acquisition module is used for acquiring a plurality of pictures classified into a plurality of data sets and preprocessing the data sets;
the processing module is used for cutting the image in the preprocessed data set to obtain an input image x and a target image y;
the generating module is used for establishing a 2D generating model, and training the 2D generating model by taking the data set after image cutting as a training sample;
the optimization module is used for optimizing and testing the trained 2D generation model;
the input module is used for inputting the single drawing as an input image into the tested 2D generation model and performing post-processing operation on the 2D model output by the 2D generation model.
Compared with the prior art, the invention has the beneficial effects that:
the method for generating the 2D model by using the single drawing based on the deep learning can generate a specific 2D model based on the single Zhang Huihua image, and can be applied to a plurality of fields such as movie animation special effects, game scenes, virtual fitting and the like;
by introducing the pattern noise layer and the projection layer, the diversity of the generator and the image detail are increased, and the sense of reality and the artistry of the generated image are improved;
regularization technology and super-parameter adjustment optimization are also applied in model training, so that stability and image quality of the generator can be further improved.
The above is only a preferred embodiment of the present invention, and the present invention is not limited thereto, but it is to be understood that the present invention is described in detail with reference to the foregoing embodiments, and modifications and equivalents of some of the technical features described in the foregoing embodiments may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method for generating a 2D model by a single drawing based on deep learning is characterized by comprising the following steps: the method comprises the following steps:
acquiring a plurality of pictures, classifying the pictures into a plurality of data sets, and preprocessing the data sets;
image cutting is carried out on the images in the preprocessed data set, and an input image is obtainedxAnd a target imagey；
Establishing a 2D generation model, and training the 2D generation model by taking the data set after image cutting as a training sample;
optimizing and testing the trained 2D generation model;
and inputting the single drawing as an input image into the tested 2D generation model, and performing post-processing operation on the 2D model output by the 2D generation model.
2. The method for generating a 2D model based on deep learning individual drawings of claim 1, wherein: the method comprises the steps of obtaining a plurality of paintings, classifying the paintings into a plurality of data sets, preprocessing the data sets, and comprising the following steps of:
performing picture scaling, image enhancement and data cleaning on the data set;
performing edge detection on the image in the data set, and extracting the outline of the painting work;
and extracting color features of the pictorial representation according to the outline of the pictorial representation.
3. The method for generating a 2D model based on deep learning individual drawings of claim 2, wherein: the edge detection is carried out on the image in the data set, and the outline of the painting work is extracted, which comprises the following steps:
smoothing the image by a gaussian filter using the formulaSmoothing the image by a two-dimensional gaussian function, wherein the parameters of the distribution by the gaussian functionσThe degree of smoothing the image is controlled,σthe smaller the filter, the higher the positioning accuracy of the filter, the lower the signal-to-noise ratio, and vice versa;
by the formula:
；
；
；
computing gradient magnitude for each point in pictorial representation IGAnd gradientθDirection of (a) whereinG x (i,j) AndG y (i,j) Respectively is a point%i,j) At the position ofx,yPartial derivative of direction;
in the way ofi,j) As the center point of the field, the field isθ(i,j) Gradient value of each point in the directionG (i,j) Comparing, and taking the point where the gradient maximum value is locatedi,j) As candidate edge points, or else, non-edge points, obtaining a candidate edge image K;
setting a high thresholdT h And a low thresholdT l Any point of the obtained candidate edge pointsi,j) Detecting if it is%i,j) Gradient value of the outputG (i,j)＞T h If the store is determined to be an edge point, ifG (i,j)＜T l The point is not an edge point; if it isT l ＜G(i,j)＜T h Judging whether an edge point exists in the field of the point, if so, judging that the point is the edge point, otherwise, judging that the point is not the edge point.
4. A method of generating a 2D model based on deep learning individual drawings as claimed in claim 3, wherein: the extracting the color features of the pictorial representation according to the outline of the pictorial representation includes:
by the formula:
；
；
the method comprises the steps of carrying out a first treatment on the surface of the Calculating first moments of colors in pictorial representationsμ ci Second moment ofσ ci Moment of third orderξ ci The method comprises the steps of carrying out a first treatment on the surface of the Wherein, Nfor the number of pixels in the image,P ci for a color value ofCAnd the color component isiIs used to determine the probability of the occurrence of a pixel,Ca number of colors included in the pictorial representation; each color has three components, each component having a third order moment;
by the formula:
P C =(μ cr , σ cr , ξ cr , μ cg , σ cg , ξ cg , μ cb , σ cb , ξ cb )，generating a set of color features from the color features of each pictorial representationP C 。
5. The method for generating a 2D model based on deep learning individual drawings of claim 4, wherein: the image in the preprocessed data set is subjected to image cutting to obtain an input imagexAnd a target imageyComprising the following steps:
from [0.5,1]Randomly selecting a scaling factor within a range of (a)sAnd zoom the image tosX original resolution;
randomly selecting a window with the width and the height being the original resolution from the zoomed image as a cutting area;
cutting the image into a plurality of small blocks according to the cutting area, wherein the image is inputxFor small block image after random cutting, target imageyIs at the same position as the input image x in the corresponding original imageIs a small block image of (a).
6. The method for generating a 2D model based on deep learning individual drawings of claim 5, wherein: the 2D generation model is built, the data set after image cutting is used as a training sample, and the 2D generation model is trained, and the method comprises the following steps:
by building up generatorsGDistinguishing deviceDEstablishing a 2D generation model;
defining a loss function comprising a generator loss and a arbiter loss;
based on the data set after image cutting, the loss function is lowered through an alternate training generator and a discriminator, and training of the 2D generation model is completed.
7. The method for generating a 2D model based on deep learning individual drawings of claim 6, wherein: the generator is defined as:
w=MLP(z)ϵR dlatent ，(w,y 1 )=G proj (w,y 1 )ϵR dlatent ，x=G (w,y 1 )ϵR C*H*W where z represents the input low-dimensional noise vector,MLPrepresenting a multi-layer perceptron for vector of noise inputzMapping to style vectors in potential spacew，dlatentRepresenting the dimensions of the potential space,G proj representing a learnable projection layer for vector of patternswAnd condition informationy 1 Merging and generating a new dimension vectorw, y 1 ) G represents a Generator model composed of a plurality of Generator Blocks for inputting pattern vectors #w, y 1 ) Conversion to high resolution imagesx 1 ，C、HAndWthe number, height and width of channels representing the image respectively;
the discriminant is defined as:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein, xrepresenting the input image, obtaining feature images with different resolutions after convolution and downsampling processing,S i (x) Representing an input imagexCarry out the first stepiThe feature map obtained after the layer convolution and downsampling processes,D i represent the firstiA layer discriminant model for the firstiThe feature map of the layer is classified and scored,a i representing the corresponding discriminant D i And the weight coefficient of the classification module is used for controlling the contribution degree of each layer to the final classification result.
8. The method for generating a 2D model based on deep learning individual drawings of claim 7, wherein:
the generator penalty is:
L G =-E w y pdat(,1)~ a[logD(G (w,y 1 ))]wherein, the method comprises the steps of, wherein,pdatarepresenting the distribution of potential vectors and condition information in the input training set,Drepresenting a discriminant model for evaluating whether an input image is a true image, logD(G(w,y 1 ) A) represents the generatorGThe image produced is taken as input and calculated to pass through the discriminatorDThe logarithm of the probability of the post-output,Erepresenting the operation of summing and averaging the potential vectors and condition information in all input training sets,L G representation generatorGA loss function of (2);
the loss of the discriminator is as follows:
wherein, the method comprises the steps of, wherein,pdata(x) The distribution of the real image is represented,pG(z,y) Representation generatorGDistribution of the generated image, whereinzThe noise vector representing the input is represented by a vector,y 1 the condition information is represented by a set of conditions,D(x) Representing a real image as an input imagexAfter input, the probability of the output of the discriminator;D(G(z,y) A) represents the generatorGThe probability of the output of the arbiter after the generated image is taken as input,aandbrepresenting the threshold at which the arbiter attempts to distinguish between the real image and the generated image,Erepresenting the desire, representing the operation of summing and averaging all the input real images and the generated images,L D representation discriminatorDIs a function of the loss of (2).
9. The method for generating a 2D model based on deep learning individual drawings of claim 8, wherein: the method for training the 2D generation model based on the image cut data set comprises the following steps of:
initialization generator and arbiter: will generatorGDistinguishing deviceDInitializing to a random function;
preparing training data: randomly sampling mini-batch data from a real data set, wherein half of the mini-batch data are real images, and the other half are generators G A generated false image;
forward propagation and backward propagation: inputting sampled mini-batch data to a generatorG Obtaining false images and respectively inputting the true and false images into a discriminatorD Calculating the discrimination probability, and carrying out back propagation and parameter updating on the generator and the discriminator according to the loss function;
calculating a loss function: during the forward and backward propagation, the value of the loss function needs to be calculated and recorded;
repeating the initializing generator and the arbiter to the calculating the loss function to drop the loss function by alternately training the generator and the arbiter.
10. A system for generating a 2D model from a single drawing based on deep learning, characterized in that: the system comprises:
the acquisition module is used for acquiring a plurality of pictures classified into a plurality of data sets and preprocessing the data sets;
the processing module is used for cutting the image in the preprocessed data set to obtain an input imagexAnd a target imagey；
The generating module is used for establishing a 2D generating model, and training the 2D generating model by taking the data set after image cutting as a training sample;
the optimization module is used for optimizing and testing the trained 2D generation model;
the input module is used for inputting the single drawing as an input image into the tested 2D generation model and performing post-processing operation on the 2D model output by the 2D generation model.
CN202310561085.XA 2023-05-18 2023-05-18 Method and system for generating 2D model by single drawing based on deep learning Active CN116580121B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310561085.XA CN116580121B (en) 2023-05-18 2023-05-18 Method and system for generating 2D model by single drawing based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310561085.XA CN116580121B (en) 2023-05-18 2023-05-18 Method and system for generating 2D model by single drawing based on deep learning

Publications (2)

Publication Number Publication Date
CN116580121A true CN116580121A (en) 2023-08-11
CN116580121B CN116580121B (en) 2024-04-09

Family

ID=87535488

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310561085.XA Active CN116580121B (en) 2023-05-18 2023-05-18 Method and system for generating 2D model by single drawing based on deep learning

Country Status (1)

Country Link
CN (1) CN116580121B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132687A (en) * 2023-08-14 2023-11-28 北京元跃科技有限公司 Animation generation method and device and electronic equipment

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109472838A (en) * 2018-10-25 2019-03-15 广东智媒云图科技股份有限公司 A kind of sketch generation method and device
CN110211192A (en) * 2019-05-13 2019-09-06 南京邮电大学 A kind of rendering method based on the threedimensional model of deep learning to two dimensional image
CN111724299A (en) * 2020-05-21 2020-09-29 同济大学 Super-realistic painting image style migration method based on deep learning
US20210343063A1 (en) * 2020-05-04 2021-11-04 Microsoft Technology Licensing, Llc Computing photorealistic versions of synthetic images
US20220020167A1 (en) * 2020-07-16 2022-01-20 Hyundai Motor Company Method and apparatus for detecting anchor-free object based on deep learning
US20220067451A1 (en) * 2020-08-26 2022-03-03 X Development Llc Generating quasi-realistic synthetic training data for use with machine learning models
US20220217321A1 (en) * 2021-01-06 2022-07-07 Tetavi Ltd. Method of training a neural network configured for converting 2d images into 3d models
CN115068951A (en) * 2022-06-29 2022-09-20 北京元跃科技有限公司 Method for generating intelligence-developing game through pictures
WO2023005186A1 (en) * 2021-07-29 2023-02-02 广州柏视医疗科技有限公司 Modal transformation method based on deep learning
US20230133103A1 (en) * 2021-10-28 2023-05-04 Terumo Kabushiki Kaisha Learning model generation method, image processing apparatus, program, and training data generation method

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109472838A (en) * 2018-10-25 2019-03-15 广东智媒云图科技股份有限公司 A kind of sketch generation method and device
CN110211192A (en) * 2019-05-13 2019-09-06 南京邮电大学 A kind of rendering method based on the threedimensional model of deep learning to two dimensional image
US20210343063A1 (en) * 2020-05-04 2021-11-04 Microsoft Technology Licensing, Llc Computing photorealistic versions of synthetic images
CN111724299A (en) * 2020-05-21 2020-09-29 同济大学 Super-realistic painting image style migration method based on deep learning
US20220020167A1 (en) * 2020-07-16 2022-01-20 Hyundai Motor Company Method and apparatus for detecting anchor-free object based on deep learning
US20220067451A1 (en) * 2020-08-26 2022-03-03 X Development Llc Generating quasi-realistic synthetic training data for use with machine learning models
US20220217321A1 (en) * 2021-01-06 2022-07-07 Tetavi Ltd. Method of training a neural network configured for converting 2d images into 3d models
WO2023005186A1 (en) * 2021-07-29 2023-02-02 广州柏视医疗科技有限公司 Modal transformation method based on deep learning
US20230133103A1 (en) * 2021-10-28 2023-05-04 Terumo Kabushiki Kaisha Learning model generation method, image processing apparatus, program, and training data generation method
CN115068951A (en) * 2022-06-29 2022-09-20 北京元跃科技有限公司 Method for generating intelligence-developing game through pictures

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132687A (en) * 2023-08-14 2023-11-28 北京元跃科技有限公司 Animation generation method and device and electronic equipment
CN117132687B (en) * 2023-08-14 2024-06-11 北京元跃科技有限公司 Animation generation method and device and electronic equipment

Also Published As

Publication number Publication date
CN116580121B (en) 2024-04-09

Similar Documents

Publication Publication Date Title
CN107657279B (en) Remote sensing target detection method based on small amount of samples
CN107818554B (en) Information processing apparatus and information processing method
CN109684925B (en) Depth image-based human face living body detection method and device
CN103632132B (en) Face detection and recognition method based on skin color segmentation and template matching
CN109360232B (en) Indoor scene layout estimation method and device based on condition generation countermeasure network
CN108647588A (en) Goods categories recognition methods, device, computer equipment and storage medium
CN111462206B (en) Monocular structure light depth imaging method based on convolutional neural network
CN107330371A (en) Acquisition methods, device and the storage device of the countenance of 3D facial models
CN112784736B (en) Character interaction behavior recognition method based on multi-modal feature fusion
CN111242864B (en) Finger vein image restoration method based on Gabor texture constraint
CN103177446A (en) Image foreground matting method based on neighbourhood and non-neighbourhood smoothness prior
WO2019071976A1 (en) Panoramic image saliency detection method based on regional growth and eye movement model
Trouvé et al. Single image local blur identification
CN107944437B (en) A kind of Face detection method based on neural network and integral image
CN108564120A (en) Feature Points Extraction based on deep neural network
Zhang et al. A swarm intelligence based searching strategy for articulated 3D human body tracking
CN116580121B (en) Method and system for generating 2D model by single drawing based on deep learning
Venkatesan et al. Face recognition system with genetic algorithm and ANT colony optimization
CN111009005A (en) Scene classification point cloud rough registration method combining geometric information and photometric information
CN105321188A (en) Foreground probability based target tracking method
CN106529441A (en) Fuzzy boundary fragmentation-based depth motion map human body action recognition method
CN114463492A (en) Adaptive channel attention three-dimensional reconstruction method based on deep learning
CN103593639A (en) Lip detection and tracking method and device
CN112766145B (en) Method and device for identifying dynamic facial expressions of artificial neural network
CN105913084A (en) Intensive track and DHOG-based ultrasonic heartbeat video image classifying method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant