CN113345077B - PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for Qin cavity character - Google Patents
PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for Qin cavity character Download PDFInfo
- Publication number
- CN113345077B CN113345077B CN202110648230.9A CN202110648230A CN113345077B CN 113345077 B CN113345077 B CN 113345077B CN 202110648230 A CN202110648230 A CN 202110648230A CN 113345077 B CN113345077 B CN 113345077B
- Authority
- CN
- China
- Prior art keywords
- gan
- pifu
- data
- generator
- picture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000006872 improvement Effects 0.000 claims abstract description 5
- 238000013256 Gubra-Amylin NASH model Methods 0.000 claims description 10
- 230000003044 adaptive effect Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses a method for reconstructing a three-dimensional model of a Qin cavity figure based on PIFU and 3D-GAN improvement, which comprises the steps of firstly, resizing a 2D Qin cavity figure picture; then, background-based matting algorithm is utilized to eliminate Background operation for the 2D dramatic characters; adding a mask to the image, extracting a mask area of the image, and finishing preprocessing operation on the 2D Qin cavity figure picture; then modifying the 3D-GAN network, merging a texture reconstruction part into a Tex-PIFU model of the PIFU network, adding a generator of the 3D-GAN in order to ensure that identities keep consistency before and after generation, and introducing circular consistency loss; finally, inputting the picture obtained by extracting the mask region into a 3D-GAN, and carrying out three-dimensional reconstruction on the 2D dramatic character picture; the method of the invention enables the generated three-dimensional model to be more vivid and lifelike, solves the problems of rough texture and longer training time of the three-dimensional model generated by the 3D-GAN, and greatly improves the fidelity and the operation speed of the three-dimensional reconstruction model.
Description
Technical Field
The invention belongs to the field of computer graphics processing, and particularly relates to a method for reconstructing a three-dimensional model of a Qin cavity character based on PIFU and 3D-GAN improvement.
Background
By reconstructing the three-dimensional model of the 2D ash chamber character picture, a 3D character model is generated for human-computer interaction, development of ash chamber culture is greatly promoted, people are enabled to pick up interest in traditional drama in a more modern mode, and deep artistic accumulation and rich human connotation of the traditional Chinese culture of ash chamber are realized.
Currently, for certain specific fields of objects, such as faces, human bodies or known artificial objects, relatively accurate 3D surfaces have been deduced from images with the aid of parametric models, data-driven techniques or deep neural networks. The traditional method is to search components from CAD libraries to generate new objects, so that the generation is very real and not novel. Recent 3D deep learning advances have shown that general shapes can infer relatively accurate 3D surfaces from few images, sometimes even a single input. However, the resulting resolution and accuracy of these methods is often limited and does not work well even for modeling tasks in a particular area. The sippe network introduces a multi-view reasoning method by synthesizing new silhouette views from a single image. Although the memory efficiency of multi-view silhouette is high, the concave region is difficult to infer, and therefore, the Siclope cannot reliably derive the exact details of the model.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention aims to provide the improved three-dimensional model reconstruction method for the Qin cavity character based on PIFU and 3D-GAN, which has the advantages of less memory occupation, accurate detail capturing and improvement of the fidelity and the operation speed of the three-dimensional reconstruction model.
In order to achieve the above purpose, the invention is realized by adopting the following technical scheme:
the improved three-dimensional model reconstruction method for the Qin cavity character based on PIFU and 3D-GAN comprises the following steps:
step 1: collecting 2D Qin cavity figure pictures, and setting the pictures to be of an adaptive size;
step 2: performing Background removal operation on the Qin cavity figure picture in the step 1 by using a Background-based matting algorithm;
step 3: performing mask processing on the picture obtained in the step 2, adding masks to the picture, and extracting mask areas;
step 4: adding a Tex-PIFU model of a PIFU network in front of a DIB-R frame in an original 3D-GAN model network, adding a generator in the 3D-GAN network, and introducing a cycle consistency loss to restrict training of the generator;
the loop consistency loss is expressed as:
wherein L is cyc Indicated is a loop consistency loss; the parameter G represents the generator of 3D-GAN; the parameter F represents the generator of the added 3D-GAN;
representing the expected x from P data Acquiring in distribution; x represents the real data of the person,
p_data (x) represents a distribution of real data; similarly, E_ (y_P_data (y)) indicates that y is expected to be derived from P data (y) obtaining in a distribution; y represents the generated data, P data (y) represents a distribution of the generated data; g (x) represents the image y' of the input image x generated by the generator G; f (G (x)) represents the image x' of the image y generated by the generator F; u is in the formula 1 Subscript 1 represents a 1-norm, and the whole represents the sum of the absolute values of the elements in the U vector;
step 5: and (3) taking the mask picture obtained in the step (3) as input data of training, wherein the training target is F (G (x)) (x), F (G (y)) (y), and performing three-dimensional reconstruction on the mask picture by using the improved 3D-GAN to obtain the three-dimensional model of the Qin cavity character.
Further, the 3D-GAN model network structure is composed of a generator, a discriminator and a DIB-R framework.
Further, the picture size is set to 512x512mm in the step 1.
The invention has the following beneficial effects:
the invention is based on PIFU and 3D-GAN improved three-dimensional model reconstruction method of the Qin cavity personage, utilize Background-based to scratch the algorithm and carry on the Background operation to 2D dramatic personage picture, through modifying 3D-GAN network, merge the texture rebuilding part into Tex-PIFU model of PIFU network, in order to make the identity keep the consistency before and after producing, the invention increases a generator of 3D-GAN, and introduce the training of the constraint generator of the loss of the cyclic consistency, achieve the purpose that the three-dimensional model produced is more true and vivid; the method ensures that the generated three-dimensional model is more vivid and lifelike, solves the problems of rough texture and long training time of the three-dimensional model generated by the 3D-GAN, and greatly improves the fidelity and the operation speed of the three-dimensional reconstruction model.
The method does not need to divide the space into voxels, and has less memory occupation and higher memory efficiency; the full convolution neural network can be used for retaining local features in the image, the implicit function can express geometric details of the surface, and accurate details appearing in the 2D image can be captured; the color of each vertex of the model can be predicted, and a real and vivid three-dimensional model can be generated by only one 2D picture, so that the process is simple.
Drawings
FIG. 1 is a 2D Qin cavity character picture;
FIG. 2 is a block diagram of a Background-based matting algorithm;
FIG. 3 is an image generated by a matting algorithm;
FIG. 4 is an image obtained by extracting a mask area;
FIG. 5 is an overall block diagram of the 3D-GAN model;
FIG. 6 is a 3D-GAN model incorporating Tex-PIFU;
FIG. 7 is a 3D-GAN model after completion;
FIG. 8 is a schematic diagram of a final generated three-dimensional model;
Detailed Description
The following examples illustrate the invention in further detail.
The invention provides a PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for a Qin cavity figure, which comprises the following steps:
step 1: first, a 2D Qin cavity person picture is obtained by photographing the Qin cavity person under the green screen, and then the image size is changed to 512, as shown in fig. 1, the obtained 2D Qin cavity person picture is obtained.
Step 2: in order to obtain a better three-dimensional model without being interfered by other factors in training, background-based matting algorithm is used for carrying out Background elimination operation on the graph in fig. 1, the structure of the Background-based matting algorithm is shown in fig. 2, and a result obtained after Background elimination is shown in fig. 3.
Step 3: after the background of the person is eliminated, the mask area of fig. 2 is extracted, and the preprocessing operation of the data is completed at this time, and the result is shown in fig. 4.
Step 4: the original 3D-GAN model network structure is constructed by using Pytorch, the 3D-GAN model network structure is composed of a generator, a discriminator and a DIB-R framework, and the 3D-GAN model is shown in figure 5.
The generators and discriminators in the 3D-GAN compete against each other and game with each other. The task of the generator is to generate data that makes the arbiter unable to determine true or false, while the task of the arbiter is to distinguish the true or false of the pictures generated by the generator and the true pictures. In the training process of the 3D-GAN, parameters of a generator, parameters of a training discriminator and parameters of the training generator are required to be fixed.
In order to obtain a finer three-dimensional model, a Tex-PIFU model in PIFU is introduced, a part for extracting picture textures in 3D-GAN is replaced, the Tex-PIFU model firstly predicts a continuous space internal and external probability field, and a model surface of a human body is obtained by extracting 0.5 equivalent surfaces. Then, the texture prediction process predicts an RGB color value for the 3D points of the human geometry surface, which captures the exact details that appear in the 2D image and predicts the color of each vertex of the model. The improved 3D-GAN network structure at this time is shown in fig. 6.
In order to keep identity consistency between the generated three-dimensional model and the original image, the invention adds a generator for the 3D-GAN, and introduces a cycle consistency loss to restrict the training of the generator so as to achieve the purpose that the generated three-dimensional model is more real and lifelike. The improved 3D-GAN network structure at this time is shown in fig. 7.
The loop consistency loss is expressed as:
wherein L is cyc Indicated is a loop consistency loss; the parameter G represents the generator of 3D-GAN; the parameter F represents the generator of the added 3D-GAN;
representing the expected x from P data Acquiring in distribution; x represents the real data of the person,
p_data (x) represents a distribution of real data; similarly, E_ (y_P_data (y)) indicates that y is expected to be derived from P data (y) obtaining in a distribution; y represents the generated data, P data (y) represents a distribution of the generated data; g (x) represents the image y' of the input image x generated by the generator G; f (G (x)) represents the image x' of the image y generated by the generator F; u is in the formula 1 Subscript 1 represents the 1-norm and overall represents the sum of the absolute values of the elements in the U vector. Our training goal is to make F (G (x)). Apprxeq.x, G (F (y)). Apprxeq.y.
Step 5: the image shown in fig. 3 is taken as an original image and is input into the improved 3D-GAN network, the training round is set to be 100 times, other parameters are the same as those of the 3D-GAN and Tex-PIFU, and training is started. The three-dimensional model of the Qin cavity character is shown in (a), (b) and (c) in fig. 8.
Claims (3)
1. The improved three-dimensional model reconstruction method for the Qin cavity character based on PIFU and 3D-GAN is characterized by comprising the following steps:
step 1: collecting 2D Qin cavity figure pictures, and setting the pictures to be of an adaptive size;
step 2: performing Background removal operation on the Qin cavity figure picture in the step 1 by using a Background-based matting algorithm;
step 3: performing mask processing on the picture obtained in the step 2, adding masks to the picture, and extracting mask areas;
step 4: adding a Tex-PIFU model of a PIFU network in front of a DIB-R frame in an original 3D-GAN model network, adding a generator in the 3D-GAN network, and introducing a cycle consistency loss to restrict training of the generator;
the loop consistency loss is expressed as:
wherein L is cyc Indicated is a loop consistency loss; the parameter G represents the generator of 3D-GAN; the parameter F represents the generator of the added 3D-GAN;
representing the expected x from P data Acquiring in distribution; x represents the real data of the person,
p_data (x) represents a distribution of real data; similarly, E_ (y_P_data (y)) indicates that y is expected to be derived from P data (y) obtaining in a distribution; y represents the generated data, P data (y) represents a distribution of the generated data; g (x) represents the image y' of the input image x generated by the generator G; f (G (x)) represents the image x' of the image y generated by the generator F; u is in the formula 1 Subscript 1 represents a 1-norm, and the whole represents the sum of the absolute values of the elements in the U vector;
step 5: and (3) taking the mask picture obtained in the step (3) as input data of training, wherein the training target is F (G (x)) (x), G (F (y)) (y), and the three-dimensional reconstruction is carried out on the mask picture by using the improved 3D-GAN to obtain the three-dimensional model of the Qin cavity character finally.
2. The method for reconstructing the three-dimensional model of the Qin cavity character based on the improvement of PIFU and 3D-GAN as set forth in claim 1, wherein: the 3D-GAN model network structure consists of a generator, a discriminator and a DIB-R framework.
3. The method for reconstructing the three-dimensional model of the Qin cavity character based on the improvement of PIFU and 3D-GAN as set forth in claim 1, wherein: the picture size is set to 512x512mm in the step 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110648230.9A CN113345077B (en) | 2021-06-10 | 2021-06-10 | PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for Qin cavity character |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110648230.9A CN113345077B (en) | 2021-06-10 | 2021-06-10 | PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for Qin cavity character |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113345077A CN113345077A (en) | 2021-09-03 |
| CN113345077B true CN113345077B (en) | 2024-03-15 |
Family
ID=77476415
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110648230.9A Active CN113345077B (en) | 2021-06-10 | 2021-06-10 | PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for Qin cavity character |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113345077B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020029356A1 (en) * | 2018-08-08 | 2020-02-13 | 杰创智能科技股份有限公司 | Method employing generative adversarial network for predicting face change |
| CN112488953A (en) * | 2020-12-10 | 2021-03-12 | 深圳先进技术研究院 | Medical image denoising method, system, terminal and storage medium |
| CN112541864A (en) * | 2020-09-25 | 2021-03-23 | 中国石油大学(华东) | Image restoration method based on multi-scale generation type confrontation network model |
-
2021
- 2021-06-10 CN CN202110648230.9A patent/CN113345077B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020029356A1 (en) * | 2018-08-08 | 2020-02-13 | 杰创智能科技股份有限公司 | Method employing generative adversarial network for predicting face change |
| CN112541864A (en) * | 2020-09-25 | 2021-03-23 | 中国石油大学(华东) | Image restoration method based on multi-scale generation type confrontation network model |
| CN112488953A (en) * | 2020-12-10 | 2021-03-12 | 深圳先进技术研究院 | Medical image denoising method, system, terminal and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于改进的CycleGAN模型非配对的图像到图像转换;何剑华;龙法宁;朱晓姝;;玉林师范学院学报;20180430(第02期);第122-126页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113345077A (en) | 2021-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109255831B (en) | Single-view face three-dimensional reconstruction and texture generation method based on multi-task learning | |
| CN110310319B (en) | Illumination-separated single-view human body clothing geometric detail reconstruction method and device | |
| CN112529999A (en) | Parameter estimation model training method, device, equipment and storage medium | |
| CN116664782B (en) | Neural radiation field three-dimensional reconstruction method based on fusion voxels | |
| US20230230304A1 (en) | Volumetric capture and mesh-tracking based machine learning 4d face/body deformation training | |
| CN115298708A (en) | Multi-view neural human body rendering | |
| Mu et al. | Actorsnerf: Animatable few-shot human rendering with generalizable nerfs | |
| Xiong et al. | Get3dhuman: Lifting stylegan-human into a 3d generative model using pixel-aligned reconstruction priors | |
| Wu et al. | Ganhead: Towards generative animatable neural head avatars | |
| CN113345077B (en) | PIFU and 3D-GAN-based improved three-dimensional model reconstruction method for Qin cavity character | |
| CN111260706B (en) | Dense depth map calculation method based on monocular camera | |
| Hu et al. | Humanliff: Layer-wise 3d human generation with diffusion model | |
| CN117422829A (en) | Face image synthesis optimization method based on nerve radiation field | |
| Yu et al. | Confies: Controllable neural face avatars | |
| CN112002019B (en) | Method for simulating character shadow based on MR mixed reality | |
| CN117501313A (en) | Hair rendering system based on deep neural network | |
| CN112288861B (en) | Single-photo-based automatic construction method and system for three-dimensional model of human face | |
| CN114119874A (en) | Single image reconstruction high-definition 3D face texture method based on GAN | |
| Li et al. | Learning to implicitly represent 3D human body from multi-scale features and multi-view images | |
| CN117593442B (en) | Portrait generation method based on multi-stage fine grain rendering | |
| Beacco et al. | Automatic 3D avatar generation from a single RBG frontal image | |
| Men et al. | En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data | |
| Wang | A Systematic Study on the Extraction and Image Reproduction of Ceramic Sculpture Artworks | |
| Wang et al. | Design of 3D Display System for Intangible Cultural Heritage Based on Generative Adversarial Network | |
| Sevastopolsky et al. | HeadCraft: Modeling High-Detail Shape Variations for Animated 3DMMs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |