CN110766786A - Sketch-to-bas-relief model generation method based on generation of confrontation network - Google Patents
Sketch-to-bas-relief model generation method based on generation of confrontation network Download PDFInfo
- Publication number
- CN110766786A CN110766786A CN201910878715.XA CN201910878715A CN110766786A CN 110766786 A CN110766786 A CN 110766786A CN 201910878715 A CN201910878715 A CN 201910878715A CN 110766786 A CN110766786 A CN 110766786A
- Authority
- CN
- China
- Prior art keywords
- network
- bas
- relief
- height field
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses a method for generating a sketch-to-bas-relief model based on a generation countermeasure network, which comprises the following steps: model design: and designing a reasonable network model based on the generation of the confrontation network structure. The method comprises the steps of obtaining corresponding sketches and bas-relief models from a three-dimensional model at different viewing angles, extracting a height field of the obtained bas-relief model, storing the height field as a picture, and establishing a sketches-relief data set. Model training: and inputting the established data set into a designed model for training, optimizing and generating a network and judging parameters of the network. And (3) testing by a user: and extracting the characteristics of the sketch input by the user to generate a height field of the bas-relief model corresponding to the network output, and then restoring the generated height field into the bas-relief model. The method uses the generation confrontation network structure, can generate the corresponding bas-relief model from the user hand-drawn sketch, and the obtained bas-relief model has good visual effect.
Description
Technical Field
The invention relates to the field of picture generation, in particular to a method for generating a sketch-to-shallow relief model based on generation of a confrontation network.
Background
The relief is an ancient and mature artistic form, and the engraver engraves the figure to be shaped on a flat plate to make it separate from the plane of the original material. The modeling of relief has also been widely introduced in the field of computer graphics. It mainly includes three types, high relief, concave relief, and shallow relief (i.e., bas relief), depending on the thickness. Manually creating a bas-relief is a very cumbersome and inefficient process because it relies entirely on the artist's stereographic imagination and skill. In the last decade of development, great progress has been made by converting three-dimensional models into digital relief. Although the digital relief can be rapidly obtained by a computer, a corresponding three-dimensional model needs to be input in advance when the relief model is manufactured, so that the imagination of a creator is greatly limited, the corresponding three-dimensional model needs to be selected when the relief model is created, and if the required three-dimensional model is lacked, the relief model cannot be obtained. Therefore, the meaning that the generation of the current embossment depends on the corresponding three-dimensional model seriously can be solved.
Disclosure of Invention
Technical problem to be solved
The invention aims to provide a method for generating a sketch to bas-relief based on a generation countermeasure network, which solves the technical problem.
(II) technical scheme
The invention provides a sketch-to-money embossment generating method based on a generation countermeasure network, which comprises the following steps:
designing a model, namely designing a network of a network architecture based on an image generation task by combining with the current popular generation;
data acquisition, this step has made four kinds of data sets including: chairs, airplanes, animals, and humans; by rotating the three-dimensional models in the X axis and the Y axis, each three-dimensional model obtains 2500 sketches and corresponding bas-relief models under different viewing angles.
Training the model, inputting the collected data set into the network model, setting corresponding hyper-parameters to train the network model, optimizing and generating the network and judging the parameters of the network;
and (4) user testing, namely extracting the characteristics of the sketch input by the user to generate a height field of the bas-relief model corresponding to the network output, and then reducing the generated height field into the bas-relief model. The method and the device can generate the corresponding bas-relief model from the user hand-drawn sketch by using the generation countermeasure network structure, and the obtained bas-relief model has good visual effect.
In some embodiments of the invention, the design of the model comprises:
and generating a network G by adopting the processes from encoding to decoding. The input sketch is encoded, and then the encoded feature vector is decoded; this step subdivides the generating network into encoders and decoders. The encoder consists of eight downsampling modules, each containing an activation function, a convolution operation, and a normalization operation. In encoding, a sketch image is input to an encoder, and the encoder extracts features, whereby the image is represented by a low-dimensional feature vector. The decoder is also composed of eight upsampling modules, each of which contains the deconvolution operation and the activation function, and the feature maps in the encoder and those in the decoder are fused. When decoding, the extracted feature vector is input into a decoder, and then the decoder outputs a corresponding bas-relief height field according to the input feature.
And judging the network D, namely identifying the truth of the input bas-relief height field, wherein the network D is theoretically judged to measure the distance between the real bas-relief height field and the distribution corresponding to the generated bas-relief height field data set. The discrimination network comprises four down-sampling modules, and the first module is subjected to convolution operation and activation function. And the second module, the third module and the fourth module all comprise convolution, normalization and activation functions, and the last module is used as the output of the discrimination network after being fully connected. Judging whether the output high field of the bas-relief model and the real bas-relief model are true or false by using a judging network, and then feeding back the judging result to the generating network for updating parameters of the generating network; meanwhile, the judgment network can also learn the generated bas-relief height field and the real bas-relief height field, and update the parameters of the judgment network to achieve accurate judgment rate.
A pre-trained classification network. In order to speed up and stabilize the training of the generating network G, a pre-training classification network is used to calculate the perceptual loss between the bas-relief height field generated by the generating network G and the real bas-relief height field. Because the network can well extract the features of the image, although the pre-trained classification network is not trained with the bas-relief height field, the good classification network trained on other larger data sets can well extract the features of the image, and calculating the error between the features of the real and false bas-relief height fields can accelerate and stabilize the training of the network. The pre-trained classification network is used to extract feature maps of the input images and calculate the L1 loss values between the feature maps. The network at the lower layer of the network extracts the characteristics of the image such as the edge and the texture, and the network gradually extracts the characteristics of higher layers and more whole along with the improvement of the layers. Calculating the L1 error between different levels can identify the error between the generated bas-relief model and the true bas-relief model at the feature level, thus enabling the generation network G to robustly reach the convergence state.
In some embodiments of the invention, the constructing the data set comprises:
a large number of paired sketches and bas-relief models at different viewing angles are obtained by using conventional methods. Wherein, the step makes four kinds of data sets including: chairs, airplane models, animals, and humans. The method comprises the steps that each model rotates uniformly on an X axis and a Y axis to obtain different visual angles, the Y axis rotates 7.2 degrees each time, when a ring is formed by rotation, the X axis rotates for one time, the rotation angle is 7.2 degrees, then the rotation is carried out along the Y axis, and the rotation of the visual angles is finished until the X axis rotates to form the ring. Each model used 2500 perspectives, so that each three-dimensional model yielded 2500 pairs of sketch and bas-relief models. And then extracting the height field of the obtained bas-relief model, mapping the height field of the bas-relief model between 0 and 255, and storing the bas-relief model as a picture, wherein the picture is a single-channel gray-scale image.
HiRepresents the height field value of the bas-relief, HminRepresenting the minimum height in the height field of the bas-relief, HmaxRepresenting the maximum value of the corresponding bas-relief height field, PiIs the calculated corresponding pixel value between 0 and 255. The generating network outputs a corresponding height field according to the input sketch, and then converts the height field into a bas-relief model.
For the data set partitioning, this step partitions the collected paired sketch and bas-relief height fields into the training set by 80% and the remaining 20% into the test set.
In some embodiments of the invention, the training of the model comprises:
in the data input stage, inputting a sketch and a corresponding bas-relief height field together, then inputting the sketch picture into a generation network for learning, simultaneously using the output result of the generation network and the input bas-relief height field as input into a discrimination network, and judging whether the two pictures are true or false by the discrimination network, wherein theoretically, the difference between the distribution of the pictures generated by the generation network and the distribution formed by the real pictures is measured.
During training, the parameters of the discrimination network D are fixed, and then the parameters of the generation network are provided with gradient updating according to the difference between the distribution of the bas-relief height field generated by the generation network and the distribution corresponding to the real bas-relief height field calculated by the discrimination network. When the parameters of the generation network are updated in one round, the distance between the distribution formed by the picture generated by the generation network G and the distribution of the real bas-relief height field is reduced, and the judgment network can not judge the genuineness of the generated bas-relief height field and the real bas-relief height field well. Therefore, the parameters of the generating network G are fixed, then the sketch is input to the generating network G to output a false bas-relief height field, the false bas-relief height field and the real bas-relief height field are identified through the identifying network D, and the parameters of the gradient updating identifying network D are provided through the loss of the bas-relief height field and the real bas-relief height field generated by the generating network G identified by the identifying network, so that the identifying network D can identify the true and false of the picture generated by the generating network G and the real bas-relief height field after updating again. By this time, the training of the first round of network is finished, and the generation network G and the discrimination network D progress each other under a similar situation of mutual confrontation. The generated image generated by the generating network G is more and more real, and correspondingly, the generated bas-relief height field and a real bas-relief height field data set are fitted together in distribution; the discrimination network D can always well discriminate the truth of the input bas-relief height field.
In some embodiments of the invention, the training of the model comprises:
the WGAN-GP is selected to be used for measuring the distance between the image distribution domains, because the WGAN-GP can more accurately measure the distance between the distribution domains so as to provide more stable gradient to update the generation network G;
the parameters of the generated network G are fixed firstly, the parameters of the discrimination network D are updated five times, and then the parameters of the discrimination network are fixed to update the generated network G, so that the stable training effect can be achieved;
in some embodiments of the present invention, the testing of the model comprises the steps of:
the generation network G inputs according to a sketch of a user, an encoder encodes firstly, and then a decoder decodes the encoded characteristic vector; in the decoding process, the characteristic diagram in the encoding process and the decoded characteristic diagram are subjected to cross fusion, so that the pressure of generating the network G can be relieved. Although the pixel values of the input sketch and the output bas-relief height field are greatly different, the sketch and the bas-relief height field have the same outline and boundary, so that the sharing of the features can relieve the pressure of generating the network to a certain extent and accelerate the convergence of the model.
After having obtained the bas-relief height field generated by the generating network G, this step requires its conversion into a corresponding three-dimensional bas-relief model. The pixel values are converted back to the bas-relief height field using the pixel point values multiplied by the difference between the maximum height value and the minimum height value divided by 255, plus the minimum height value. Before the transition, the smoothing may be performed by a 3 × 3 low-pass filter, for example: and (4) Gaussian filtering. Thus, the surface of the three-dimensional low-relief model after reduction can be smoother.
(III) advantageous effects
Compared with the prior art, the method for generating the sketch-to-bas-relief model based on the generation countermeasure network has at least the following advantages:
1. the generation of the bas-relief model eliminates the need to input the existing 3D model, greatly freeing up the artist's creation space.
2. The generation of the corresponding bas-relief model can be performed by sketching the user's hand.
3. An end-to-end user editable interface is designed, drawing and modifying operations can be provided for a user, and a sketch created by the user can be converted into a low-relief model in real time.
Drawings
Fig. 1 is a schematic diagram of the generation of a sketch-to-bas-relief model based on the generation of a countermeasure network according to an embodiment of the present invention.
Fig. 2 is a network structure diagram according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating downsampling according to an embodiment of the present invention.
FIG. 4 is a three-dimensional model used to acquire a data set according to an embodiment of the present invention.
FIG. 5 is a schematic diagram of a rotation strategy used to collect a data set according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of a user using system according to an embodiment of the present invention.
Detailed Description
The invention provides a method for generating a sketch-to-bas-relief model based on a generated confrontation network, which adopts a generated confrontation network structure. The method comprises the steps of designing a convolutional neural network model based on a height field from a sketch to a bas-relief model of a generated confrontation network, collecting a large number of sketches and corresponding bas-relief models to make into a data set, inputting the collected data set into the designed neural network model to train and optimize model parameters, and finally testing to convert a hand-drawn sketch input by a user into the corresponding bas-relief model in real time, so that the creation of the user is greatly facilitated without the need of the corresponding 3D model as input. The generated bas-relief model has good visual effect.
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
In one aspect of the present invention, there is provided a method for generating a sketch-to-bas-relief model based on generation of a countermeasure network, fig. 1 is a schematic structural diagram of a sketch-to-bas-relief model based on generation of a countermeasure network according to an embodiment of the present invention, as shown in fig. 1, the apparatus includes: design of model 1, building of data set 2, training of model 3 and testing of model 4. Designing a model 1, namely designing a network of a network architecture based on an image generation task by combining with the current popular generation; constructing a data set 2, wherein the step of making four types of data sets comprises the following steps: four-legged animals, teddy bears, chairs, and human heads; by rotating the three-dimensional models in the X axis and the Y axis, each three-dimensional model obtains 2500 sketch maps and corresponding bas-relief models at different viewing angles; training 3 of the model, inputting the collected data set into a network model, setting corresponding hyper-parameters to train the network model, optimizing and generating the network and judging the parameters of the network; and 4, testing the model, namely extracting characteristics of the sketch input by the user to generate a height field of the bas-relief model corresponding to the network output, and then reducing the generated height field into the bas-relief model.
Next, each module will be described in detail with reference to fig. 2 to 6.
Design of model 1, as shown in fig. 2, the model includes a generation network G, a discrimination network D, and a pre-trained classification network.
The generation network G employs a process from encoding to decoding. The input sketch is encoded, and then the encoded feature vector is decoded; this step subdivides the generating network into encoders and decoders. The encoder consists of eight downsampling modules (as shown in fig. 3), each containing an activation function, a convolution operation, and a normalization operation. In encoding, a sketch image is input to an encoder, and the encoder extracts features, whereby the image is represented by a low-dimensional feature vector. The decoder is also composed of eight upsampling modules, each of which contains a deconvolution operation and an activation function, and the feature maps in the encoder and the decoder are fused. When decoding, the extracted feature vector is input into a decoder, and then the decoder outputs a corresponding bas-relief height field according to the input feature.
And judging the network D, namely identifying the truth of the input bas-relief height field, wherein the network D is theoretically judged to measure the distance between the real bas-relief height field and the distribution corresponding to the generated bas-relief height field data set. The discrimination network comprises four down-sampling modules, and the first module is subjected to convolution operation and activation function. And the second module, the third module and the fourth module all comprise convolution, normalization and activation functions, and the last module is used as the output of the discrimination network after being fully connected. Judging whether the output high field of the bas-relief model and the real bas-relief model are true or false by using a judging network, and then feeding back the judging result to the generating network for updating parameters of the generating network; meanwhile, the judgment network can also learn the generated bas-relief height field and the real bas-relief height field, and update the parameters of the judgment network to achieve accurate judgment rate.
A pre-trained classification network. In order to speed up and stabilize the training of the generating network G, a pre-training classification network is used to calculate the perceptual loss between the bas-relief height field generated by the generating network G and the real bas-relief height field. Because the network can well extract the features of the image, although the pre-trained classification network is not trained with the bas-relief height field, the good classification network trained on other larger data sets can well extract the features of the image, and calculating the error between the features of the real and false bas-relief height fields can accelerate and stabilize the training of the network. The pre-trained classification network is used to extract feature maps of the input images and calculate the L1 loss values between the feature maps. The network at the lower layer of the network extracts the characteristics of the image such as the edge and the texture, and the network gradually extracts the characteristics of higher layers and more whole along with the improvement of the layers. Calculating the L1 error between different levels can identify the error between the generated bas-relief model and the true bas-relief model at the feature level, thus enabling the generation network G to robustly reach the convergence state.
The data set 2 is constructed to obtain a large number of paired sketches and bas-relief models at different perspectives using conventional methods. Wherein, the step makes four kinds of data sets including: chairs, airplanes, animals, and humans (as shown in fig. 4). The method comprises the following steps that each model rotates uniformly on an X axis and a Y axis to obtain different visual angles, the Y axis rotates 7.2 degrees each time, when a ring is formed by rotation, the X axis rotates 7.2 degrees, and then the rotation on the Y axis is continued until the Y axis rotates to form a new ring. The rotation of the viewing angle ends by the time the X-axis rotates to form a circle (as shown in fig. 5). Each model used 2500 perspectives, so that each three-dimensional model yielded 2500 pairs of sketch and bas-relief models. And then extracting the height field of the obtained bas-relief model, mapping the height field of the bas-relief model between 0 and 255 and storing the bas-relief model as a picture, wherein the picture is a single-channel gray-scale image.
HiRepresents the height field value of the bas-relief, HminRepresenting the minimum height in the height field of the bas-relief, HmaxRepresenting the maximum value of the corresponding bas-relief height field, PiIs the calculated corresponding pixel value between 0 and 255. The generating network outputs a corresponding height field according to the input sketch, and then converts the height field into a bas-relief model.
For the data set partitioning, this step partitions the collected paired sketch and bas-relief height fields into the training set by 80% and the remaining 20% into the test set.
And 3, training the model, namely inputting the draft and the corresponding bas-relief height field together in the data input stage, inputting the draft picture into the generation network for learning, simultaneously inputting the output result of the generation network and the input bas-relief height field into a judgment network, and judging whether the two pictures are true or false by the judgment network, wherein theoretically, the difference between the distribution of the pictures generated by the generation network and the distribution formed by the real pictures is measured.
During training, the parameters of the discrimination network D are fixed, and then the parameters of the generation network are updated in a gradient manner according to the difference between the distribution of the bas-relief height field generated by the generation network and the distribution corresponding to the real bas-relief height field calculated by the discrimination network. When the parameters of the generation network are updated in one round, the distance between the distribution formed by the picture generated by the generation network G and the distribution of the real bas-relief height field is reduced, and at this time, the judgment network can not judge the genuineness of the generated bas-relief height field and the real bas-relief height field well. Therefore, the parameters of the generating network G are fixed, then the sketch is input to the generating network G to output a false bas-relief height field, the false bas-relief height field and the real bas-relief height field are identified through the identifying network D, the parameters of the gradient updating identifying network D are provided through the loss of the bas-relief height field and the real bas-relief height field generated by the generating network G identified by the identifying network, and the identifying network D can re-identify the truth of the picture generated by the new generating network G and the real bas-relief height field. By this time, the training of the first round of network is finished, and the generation network G and the discrimination network D progress each other under a similar situation of mutual confrontation. The generated image generated by the generating network G is more and more real, and correspondingly, the generated bas-relief height field and a real bas-relief height field data set are fitted together in distribution; the discrimination network D always keeps a high field of the input bas-relief that can be well discriminated.
The WGAN-GP is selected to be used for measuring the distance between the image distribution domains, because the WGAN-GP can more accurately measure the distance between the distribution domains so as to provide more stable gradient to update the generation network G; this step uses the following loss function:
the parameters of the generated network G are fixed firstly, the parameters of the discrimination network D are updated five times, and then the parameters of the discrimination network are fixed to update the generated network G, so that the stable training effect can be achieved;
testing of the model 4, as shown in fig. 6, this step establishes an end-to-end sketch to bas-relief model generation system, the user can draw the sketch, modify and store the sketch, and the sketched sketch can generate the bas-relief model in real time by generating a network G model.
The specific process generation network G inputs according to a sketch of a user, an encoder encodes firstly, and then a decoder decodes the encoded feature vector; in the decoding process, the characteristic diagram in the encoding process and the decoded characteristic diagram are subjected to cross fusion, so that the pressure of generating the network G can be relieved. Although there is a great difference in pixel value between the input sketch and the output bas-relief height field, the sketch and the bas-relief height field have the same contour and boundary, so that the sharing of features can relieve the pressure of generating the network to some extent and accelerate the convergence of the model.
After having obtained the bas-relief height field generated by the generating network G, this step requires its conversion into a corresponding three-dimensional bas-relief model. The pixel values are converted back to the bas-relief height field using the pixel point values multiplied by the difference between the maximum height value and the minimum height value divided by 255, plus the minimum height value. Before the transition is made, smoothing is performed by a 3 x 3 low pass filter, and the method uses gaussian filtering. Thus, the surface of the three-dimensional low-relief model after reduction can be smoother.
In summary, the method for generating the sketch-to-bas-relief model based on the generation countermeasure network can overcome the defect that the conventional bas-relief model generation depends on a corresponding 3D model through the collection of a data set, the determination of the network model and the training and testing of the model, can generate the bas-relief model of the user hand-drawn sketch in real time, and the generated bas-relief model has a good visual effect.
So far, the embodiments of the present disclosure have been described in detail with reference to the accompanying drawings. It is to be noted that, in the attached drawings or in the description, the implementation modes which are not shown or described are all the modes which are known by the ordinary skilled person in the technical field and are not described in detail. In addition, the above definitions of the various elements and methods are not limited to the specific structures, shapes or modes mentioned in the embodiments, and those skilled in the art can easily modify or replace them.
The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.
In addition, unless steps are specifically described or must occur in sequence, the order of the steps is not limited to that listed above and may be changed or rearranged as desired by the desired design. The embodiments described above may be mixed and matched with each other or with other embodiments based on design and reliability considerations, i.e., technical features in different embodiments may be freely combined to form further embodiments.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. The step of bas-relief generation based on generation of a countermeasure network includes:
designing a model, namely designing a network of a network architecture based on an image generation task by combining with the current popular generation;
data acquisition, this step has made four kinds of data sets including: chairs, airplanes, animals, and humans; by rotating the three-dimensional models in the X axis and the Y axis, each three-dimensional model obtains 2500 sketches and corresponding bas-relief models at different viewing angles.
Training a model, inputting the collected data set into a network model, setting corresponding hyper-parameters to train the network model, optimizing and generating the network and judging the parameters of the network;
and (4) user testing, namely extracting the characteristics of the sketch input by the user to generate a height field of the bas-relief model corresponding to the network output, and then restoring the generated height field into the bas-relief model. The method uses the generation confrontation network structure, can generate the corresponding bas-relief model from the user hand-drawn sketch, and the obtained bas-relief model has good visual effect.
2. The bas-relief generation based on generating a counterpoise network of claim 1, wherein the model design further comprises: and generating a network G by adopting the processes from encoding to decoding. The input sketch is encoded, and then the encoded feature vector is decoded; and judging the network D, namely identifying the truth of the input bas-relief height field, wherein the network D is theoretically judged to measure the distance between the real bas-relief height field and the distribution corresponding to the generated bas-relief height field data set. A pre-trained classification network. In order to accelerate and stabilize the training of the generation network G, the device uses a pre-trained classification network to calculate the perception loss between the bas-relief height field generated by the generation network G and the real bas-relief height field. Because the network can well extract the characteristics of the image, although the classification network pre-trained by the device is not trained by the bas-relief height field, the classification network trained on other larger data sets can also well extract the characteristics of the image, and the error between the characteristics of the real and false bas-relief height fields can be calculated, so that the training of the network can be accelerated and stabilized.
3. The bas-relief generation based on generating a counterpoise network of claim 3, further comprising: an encoder and a decoder. The apparatus subdivides the generation network into an encoder and a decoder. The encoder consists of eight downsampling modules, each containing an activation function, a convolution operation and a normalization operation. In encoding, a sketch image is input to an encoder, and the encoder extracts features, whereby the image is represented by a low-dimensional feature vector. The decoder is also composed of eight upsampling modules, each of which contains the deconvolution operation and the activation function, and the feature maps in the encoder and those in the decoder are fused. When decoding, the extracted feature vector is input into a decoder, and the decoder outputs a corresponding bas-relief height field according to the input feature.
4. The bas-relief generation based on generating a countermeasure network of claim 2, further comprising a discriminant network. The discrimination network comprises four down-sampling modules, the first of which is subjected to convolution operation and an activation function. And the second module, the third module and the fourth module all comprise convolution, normalization and activation functions, and the last module is used as the output of the discrimination network after being fully connected. Judging whether the output height field of the bas-relief model and the real bas-relief model are true or false by using a judging network, and then feeding back the judging result to the generating network for updating parameters of the generating network; meanwhile, the judgment network can also learn the generated bas-relief height field and the real bas-relief height field, and update the parameters of the judgment network to achieve accurate judgment rate.
5. The bas-relief generation based on generating a countering network of claim 2, further comprising a pre-trained classification network. The pre-trained classification network is used to extract feature maps of the input images and calculate the L1 loss values between the feature maps.
y representsA characteristic diagram extracted from the real bas-relief height field data,representing the extracted feature map of the generated bas-relief height field. Because the trained neural network can well extract the characteristics of the image. The network at the lower layer of the network extracts the features such as the edge and texture of the image, and the network gradually extracts the features of higher layers and more overall as the layers are improved. Calculating the L1 error between different levels can identify the error between the generated bas-relief model and the true bas-relief model at the feature level, thus enabling the generated network G to reach convergence robustly.
6. The bas-relief generation based on generating a counterpoise network of claim 1, wherein the data acquisition comprises: a large number of paired sketches and bas-relief models at different viewing angles are obtained by using conventional methods. Wherein, the step makes four kinds of data sets including: chairs, airplane models, animals, and humans. The method comprises the steps that each model rotates uniformly on an X axis and a Y axis to obtain different visual angles, the Y axis rotates 7.2 degrees each time, when a ring is formed by rotation, the X axis rotates for one time, the rotation angle is 7.2 degrees, then the rotation is carried out along the Y axis, and the rotation of the visual angles is finished until the X axis rotates to form the ring. Each model used 2500 perspectives, so that each three-dimensional model yielded 2500 pairs of sketch and bas-relief models. And then extracting the height field of the obtained bas-relief model, mapping the height field of the bas-relief model between 0 and 255, and storing the bas-relief model as a picture, wherein the picture is a single-channel gray-scale image.
HiRepresents the height field value of the bas-relief, HminRepresenting the minimum height in the height field of the bas-relief, HmaxRepresenting the maximum value of the corresponding bas-relief height field, PiFor calculated correspondences between 0 and 255The pixel value. The generating network outputs a corresponding height field according to the input sketch, and then converts the height field into a bas-relief model.
For the data set partitioning, this step partitions the collected paired sketch and bas-relief height fields into the training set by 80%, and the remaining 20% into the test set.
7. The bas-relief generation based on generating a countering network of claim 1, wherein the training of the model comprises: in the data input stage, inputting a sketch and a corresponding bas-relief height field together, then inputting a sketch picture into a generation network for learning, simultaneously using a result output by the generation network and the input bas-relief height field as input into a discrimination network, and judging whether the two pictures are true or false by the discrimination network, wherein theoretically, the difference between the distribution of the pictures generated by the generation network and the distribution formed by the real pictures is measured.
During training, the parameters of the discrimination network D are fixed, and then the parameters of the generation network are provided with gradient updating according to the difference between the distribution of the bas-relief height field generated by the generation network and the distribution corresponding to the real bas-relief height field calculated by the discrimination network. When the parameters of the generation network are updated in one round, the distance between the distribution formed by the picture generated by the generation network G and the distribution of the real bas-relief height field is reduced, and at this time, the judgment network can not judge the genuineness of the generated bas-relief height field and the real bas-relief height field well. Therefore, the parameters of the generating network G are fixed, then the sketch is input to the generating network G to output a false bas-relief height field, then the false bas-relief height field and the real bas-relief height field are identified through the identifying network D, the parameters of the gradient updating identifying network D are provided through the loss of the bas-relief height field and the real bas-relief height field generated by the generating network G identified by the identifying network, and the identifying network D can re-identify the truth of the picture generated by the generating network G and the real bas-relief height field. By this time, the training of the first round of network is finished, and the generation network G and the discrimination network D progress each other under a similar situation of mutual confrontation. The generated image generated by the generating network G is more and more real, and correspondingly, the generated bas-relief height field and a real bas-relief height field data set are fitted together in distribution; the discrimination network D can always well discriminate the truth of the input bas-relief height field.
8. The bas-relief generation based on generating a countering network of claim 7, wherein the training of the model comprises:
the WGAN-GP is selected to be used for measuring the distance between the image distribution domains, and because the WGAN-GP can more accurately measure the distance between the distribution domains so as to provide more stable gradient to update the generated network G, the method uses the following loss function;
in the step, the parameters of the generation network G are fixed firstly, the parameters of the discrimination network D are updated five times, and then the parameters of the discrimination network are fixed to update the generation network G, so that the stable training effect can be achieved.
9. The bas-relief generation based on generation of a antagonistic network according to claim 1, wherein the testing of the model comprises the steps of:
the generation network G carries out encoding according to the sketch input of a user, and then carries out decoding on the encoded feature vector through a decoder; in the decoding process, the characteristic diagram in the encoding process and the decoded characteristic diagram are subjected to cross fusion, so that the pressure of generating the network G can be relieved. Although there is a large difference in pixel value between the input sketch and the output bas-relief height field, the sketch and the bas-relief height field have the same outline and boundary, so sharing between features can relieve the pressure of generating the network to some extent and accelerate model convergence.
After having obtained the bas-relief height field generated by the generating network G, this step requires its conversion into a corresponding three-dimensional bas-relief model. The pixel values are converted back to the bas-relief height field using the pixel point values multiplied by the difference between the maximum height value and the minimum height value divided by 255, plus the minimum height value. Before the transition, smoothing is performed by a 3 x 3 low pass filter, which uses gaussian filtering in this step. Thus, the surface of the three-dimensional low-relief model after reduction can be smoother.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910878715.XA CN110766786A (en) | 2019-09-18 | 2019-09-18 | Sketch-to-bas-relief model generation method based on generation of confrontation network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910878715.XA CN110766786A (en) | 2019-09-18 | 2019-09-18 | Sketch-to-bas-relief model generation method based on generation of confrontation network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110766786A true CN110766786A (en) | 2020-02-07 |
Family
ID=69330337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910878715.XA Pending CN110766786A (en) | 2019-09-18 | 2019-09-18 | Sketch-to-bas-relief model generation method based on generation of confrontation network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110766786A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583412A (en) * | 2020-04-29 | 2020-08-25 | 齐鲁工业大学 | Method for constructing calligraphy relief deep learning network and method for constructing calligraphy relief |
| CN112561797A (en) * | 2020-12-09 | 2021-03-26 | 齐鲁工业大学 | Flower relief model construction method and flower relief reconstruction method based on line drawing |
| WO2021238113A1 (en) * | 2020-05-25 | 2021-12-02 | 清华大学 | Shear wall structure arrangement method and apparatus based on generative adversarial network |
| CN114297176A (en) * | 2021-12-15 | 2022-04-08 | 东南大学 | Artificial intelligence-based automatic generation method and system for Chinese classical garden rockery |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050251275A1 (en) * | 2004-05-06 | 2005-11-10 | Carlson Keith R | Apparatus and method for creating three dimensional objects |
| CN109377448A (en) * | 2018-05-20 | 2019-02-22 | 北京工业大学 | A kind of facial image restorative procedure based on generation confrontation network |
| RU2698402C1 (en) * | 2018-08-30 | 2019-08-26 | Самсунг Электроникс Ко., Лтд. | Method of training a convolutional neural network for image reconstruction and a system for forming an image depth map (versions) |
-
2019
- 2019-09-18 CN CN201910878715.XA patent/CN110766786A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050251275A1 (en) * | 2004-05-06 | 2005-11-10 | Carlson Keith R | Apparatus and method for creating three dimensional objects |
| CN109377448A (en) * | 2018-05-20 | 2019-02-22 | 北京工业大学 | A kind of facial image restorative procedure based on generation confrontation network |
| RU2698402C1 (en) * | 2018-08-30 | 2019-08-26 | Самсунг Электроникс Ко., Лтд. | Method of training a convolutional neural network for image reconstruction and a system for forming an image depth map (versions) |
Non-Patent Citations (4)
| Title |
|---|
| J.W，ETC: "Making bas-reliefs from photographs of human faces", 《COMPUTER-AIDED DESIGN》 * |
| TIM WEYRICH，ETC: "Digital bas-relief from 3D scenes", 《ACM TRANSACTIONS ON GRAPHICS》 * |
| ZHAOLIANG LUN，ETC: "3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks", 《2017 INTERNATIONAL CONFERENCE ON 3D VISION (3DV)》 * |
| 张兴治: "数字凹浮雕生成算法", 《现代计算机（专业版）》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583412A (en) * | 2020-04-29 | 2020-08-25 | 齐鲁工业大学 | Method for constructing calligraphy relief deep learning network and method for constructing calligraphy relief |
| CN111583412B (en) * | 2020-04-29 | 2021-06-01 | 齐鲁工业大学 | Method for constructing calligraphy relief deep learning network and method for constructing calligraphy relief |
| WO2021238113A1 (en) * | 2020-05-25 | 2021-12-02 | 清华大学 | Shear wall structure arrangement method and apparatus based on generative adversarial network |
| CN112561797A (en) * | 2020-12-09 | 2021-03-26 | 齐鲁工业大学 | Flower relief model construction method and flower relief reconstruction method based on line drawing |
| CN114297176A (en) * | 2021-12-15 | 2022-04-08 | 东南大学 | Artificial intelligence-based automatic generation method and system for Chinese classical garden rockery |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110458939B (en) | Indoor scene modeling method based on visual angle generation | |
| CN110766786A (en) | Sketch-to-bas-relief model generation method based on generation of confrontation network | |
| CN108921926B (en) | End-to-end three-dimensional face reconstruction method based on single image | |
| CN108038906B (en) | Three-dimensional quadrilateral mesh model reconstruction method based on image | |
| CN112085836A (en) | Three-dimensional face reconstruction method based on graph convolution neural network | |
| CN111832655A (en) | Multi-scale three-dimensional target detection method based on characteristic pyramid network | |
| CN106228528A (en) | A kind of multi-focus image fusing method based on decision diagram Yu rarefaction representation | |
| CN111508069B (en) | Three-dimensional face reconstruction method based on single hand-drawn sketch | |
| CN104156997A (en) | Quick volume data skeleton extraction method based on rendering | |
| CN111028335B (en) | Point cloud data block surface patch reconstruction method based on deep learning | |
| CN116416376A (en) | Three-dimensional hair reconstruction method, system, electronic equipment and storage medium | |
| CN115147545A (en) | Scene three-dimensional intelligent reconstruction system and method based on BIM and deep learning | |
| CN112634429B (en) | Rock core three-dimensional image reconstruction method based on mixed depth generation model | |
| CN113593043B (en) | Point cloud three-dimensional reconstruction method and system based on generation countermeasure network | |
| CN114758070A (en) | Single-image three-dimensional human body fine reconstruction method based on cross-domain multitask | |
| CN117094895B (en) | Image panorama stitching method and system | |
| CN116385667B (en) | Reconstruction method of three-dimensional model, training method and device of texture reconstruction model | |
| CN111191729B (en) | Three-dimensional object fusion feature representation method based on multi-modal feature fusion | |
| CN117315169A (en) | Live-action three-dimensional model reconstruction method and system based on deep learning multi-view dense matching | |
| Guénard et al. | Reconstructing plants in 3D from a single image using analysis-by-synthesis | |
| CN110097615B (en) | Stylized and de-stylized artistic word editing method and system | |
| Bhardwaj et al. | SingleSketch2Mesh: generating 3D mesh model from sketch | |
| CN114332286B (en) | Artificial intelligent drawing method and device and computer storage medium | |
| CN106780722B (en) | A kind of differential mode scale Forest Scene construction method of the same race and system | |
| CN112329799A (en) | Point cloud colorization algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200207 |
|
| WD01 | Invention patent application deemed withdrawn after publication |