WO2021063271A1 - 人体模型重建方法、重建系统及存储介质 - Google Patents

人体模型重建方法、重建系统及存储介质 Download PDF

Info

Publication number
WO2021063271A1
WO2021063271A1 PCT/CN2020/117916 CN2020117916W WO2021063271A1 WO 2021063271 A1 WO2021063271 A1 WO 2021063271A1 CN 2020117916 W CN2020117916 W CN 2020117916W WO 2021063271 A1 WO2021063271 A1 WO 2021063271A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
reconstructed
texture
reconstruction
texture information
Prior art date
Application number
PCT/CN2020/117916
Other languages
English (en)
French (fr)
Inventor
李众
徐毅
全书学
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to CN202080064554.1A priority Critical patent/CN114450719A/zh
Priority to EP20873287.5A priority patent/EP4036863A4/en
Publication of WO2021063271A1 publication Critical patent/WO2021063271A1/zh
Priority to US17/707,681 priority patent/US11928778B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts

Definitions

  • the embodiments of the present application relate to the field of computer vision and three-dimensional human body reconstruction, and in particular, to a human body model reconstruction method, reconstruction system, and storage medium.
  • High-precision human body 3D reconstruction is a key issue in computer vision, computer graphics and other fields. It aims to quickly and accurately reconstruct a 3D human body parametric model from human body images. This technology can be widely used in motion analysis, visual special effects, Many fields such as virtual reality and e-commerce have high research and commercial value.
  • the current model reconstruction method cannot obtain a three-dimensional human body model with high accuracy and containing all texture information through a simple reconstruction process.
  • the embodiments of the present application provide a human body model reconstruction method, reconstruction system and storage medium, which can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore all the texture information of the human body, thereby improving the model reconstruction. Accuracy greatly optimizes the model reconstruction effect.
  • an embodiment of the present application provides a method for reconstructing a human body model, and the method includes:
  • the target image is a frame of frontal image corresponding to the object to be reconstructed;
  • the initial three-dimensional model is a three-dimensional model without texture
  • a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • an embodiment of the present application provides a reconstruction system, the reconstruction system includes: an acquisition part, a segmentation part, a reconstruction part, and a generation part,
  • the acquiring part is configured to acquire a target image; wherein the target image is a frame of frontal image corresponding to the object to be reconstructed;
  • the segmentation part is configured to perform segmentation processing based on the object to be reconstructed in the target image to obtain a segmented image
  • the acquiring part is further configured to respectively acquire the initial estimated shape and texture information of the object to be reconstructed based on the segmented image;
  • the reconstruction part is configured to determine an initial three-dimensional model of the object to be reconstructed through the initial estimated shape; wherein, the initial three-dimensional model is a textureless three-dimensional model;
  • the obtaining part is further configured to obtain complete texture information of the object to be reconstructed according to the texture information and the texture generation model;
  • the generating part is configured to generate a three-dimensional reconstruction model of the object to be reconstructed based on the initial three-dimensional model and the complete texture information; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • an embodiment of the present application provides a reconstruction system.
  • the reconstruction system includes a processor and a memory storing executable instructions of the processor. When the instructions are executed by the processor, the implementation is as described above.
  • the described reconstruction method of the human body model is as described above.
  • an embodiment of the present application provides a computer-readable storage medium with a program stored thereon and applied to a reconstruction system.
  • the program is executed by a processor, the above-mentioned method for reconstructing a human body model is realized.
  • the embodiments of the application provide a human body model reconstruction method, a reconstruction system and a storage medium.
  • the reconstruction system obtains a target image and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is the object to be reconstructed Corresponding frontal image; based on the segmented image, obtain the initial estimated shape and texture information of the object to be reconstructed; determine the initial 3D model of the object to be reconstructed through the initial estimated shape; where the initial 3D model is a textureless 3D model ; According to the texture information and the texture generation model, the complete texture information of the object to be reconstructed is obtained; based on the initial three-dimensional model and the complete texture information, a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein, the three-dimensional reconstruction model is a textured three-dimensional model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation processing. Based on the segmented image, on the one hand, the reconstruction system can obtain Based on the initial estimated shape, an initial 3D model corresponding to the object to be reconstructed, excluding the texture of the human body is generated; on the other hand, the reconstruction system can use the texture information extracted from the segmented image to use the texture generation model to restore the object to be reconstructed The complete texture information of the, and finally a textured three-dimensional reconstruction model can be obtained through the mapping of the initial three-dimensional model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • Figure 1 is the first schematic diagram of the realization process of the human body model reconstruction method
  • Figure 2 is a schematic diagram of AR/VR application demonstration
  • Figure 3 is the second schematic diagram of AR/VR application demonstration
  • Figure 4 is the third schematic diagram of AR/VR application demonstration
  • Figure 5 is a second schematic diagram of the implementation process of the human body model reconstruction method
  • Figure 6 is a schematic diagram of the reconstruction of the human body model
  • Figure 7 is a schematic diagram of two-dimensional non-rigid registration
  • Figure 8 is a schematic diagram of a model generation method
  • Fig. 9 is a schematic diagram before head correction
  • Figure 10 is a schematic diagram of the head after correction
  • Figure 11 is a schematic diagram of the front model
  • Figure 12 is a schematic diagram of the back model
  • Figure 13 is the first schematic diagram after splicing
  • Figure 14 is the second schematic diagram after splicing
  • Figure 15 is a schematic diagram of reconstructing a three-dimensional model
  • Figure 16 is a second schematic diagram of reconstructing a three-dimensional model
  • Figure 17 is a third schematic diagram of reconstructing a three-dimensional model
  • Figure 18 is a fourth schematic diagram of reconstructing a three-dimensional model
  • Figure 19 is a schematic diagram of animation reconstruction results
  • Figure 20 is a schematic diagram of a system for reconstructing a human body model
  • Figure 21 is a schematic diagram of the composition structure of the reconstruction system
  • Figure 22 is a second schematic diagram of the composition structure of the reconstruction system.
  • High-precision three-dimensional human body reconstruction is a key issue in the fields of computer vision, computer graphics, etc. It aims to quickly and accurately reconstruct a three-dimensional human body parametric model from human body images. This technology can be widely used in motion analysis, visual special effects, Many fields such as virtual reality and e-commerce have high research and commercial value.
  • Real-time 3D human body reconstruction mainly includes two aspects: geometric optimization and texture mapping.
  • Geometric optimization refers to the process of accurately generating a three-dimensional human body model; this process determines the geometric topology of the model and gives relatively accurate geometric information, but the texture information of the optimized target is lost.
  • Texture mapping is the process of mapping texture pixels to the surface of a three-dimensional object; through texture mapping, the texture information of the human body model can be restored and the sense of reality of the model can be increased. Therefore, high-quality texture mapping plays a vital role in reconstructing vivid human faces.
  • Kanade and Narayanan used a dome with a diameter of 5 meters and installed 51 cameras on it to digitize real objects into free viewpoint video (fvv).
  • shooting devices tend to use industrial-grade synchronous cameras with higher resolution and speed.
  • the all-optical studio of Carnegie Mellon University consists of 480 Video Graphics Array (VGA) cameras, It consists of 31 high-definition cameras and 10 Kinect sensors, which are used to reconstruct and restore a variety of human activities. It can be seen that the above-mentioned three-dimensional human body reconstruction technology is largely implemented by a huge three-dimensional acquisition system and complex reconstruction algorithms, and it is impractical to use such a complicated device in daily scenes.
  • VGA Video Graphics Array
  • the emerging learning-based method uses Convolutional Neural Networks (CNN) to obtain high-resolution results by training a large number of synthetic images, but there are still problems such as misalignment of the 3D model reconstruction results with the 2D images. And due to the deviation of the training data, this method is not suitable for all types of input.
  • CNN Convolutional Neural Networks
  • the reconstruction system may obtain the segmented image corresponding to the object to be reconstructed through segmentation processing. Based on the segmented image, on the one hand , The reconstruction system can generate an initial three-dimensional model corresponding to the object to be reconstructed, excluding the texture of the human body, on the basis of obtaining the initial estimated shape; on the other hand, the reconstruction system can use the texture information extracted from the partial texture information of the segmented image Generate the model, restore the complete texture information of the object to be reconstructed, and finally obtain a textured three-dimensional reconstruction model through the mapping of the initial three-dimensional model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • the reconstruction system can reconstruct a three-dimensional model of the human body and complete texture information based on the input single-frame frontal image of the human body.
  • the reconstruction system can segment the input single frame image, and then fit the skinned Multi-Person Linear Model (SMPL) model to the segmented image. Since the SMPL model cannot be well aligned with the contour of the human body in the image, in this application, the reconstruction system can continue to deform the SMPL model.
  • the reconstruction system can realize the deformation of the model by searching for the mapping relationship between the silhouette of the human figure and the silhouette of the fitted SMPL model.
  • the depth map of the SMPL model is corrected to the final shape, that is, the frontal model of the human body is obtained.
  • the reconstruction system can use the same method to reconstruct the back model of the human body, and then stitch the front model and the back model together to obtain the initial three-dimensional model of the human body.
  • the initial three-dimensional model cannot reflect all the texture information of the human body. Restore the invisible texture on the back of the human body.
  • the reconstruction system can use the texture generation model, the InferGAN network, to restore other invisible texture information based on the visible part of the human body texture information, and finally build all the human body texture information .
  • the reconstruction system can obtain a three-dimensional reconstruction model containing the complete texture of the human body, thereby completing the reconstruction of the human body model based on one frame of input image.
  • the reconstruction of the human body model can specifically include a three-dimensional human body. Modeling and texture reconstruction.
  • FIG. 1 is a schematic diagram of the realization process of the method for reconstructing a human body model.
  • the method for the reconstruction system to perform three-dimensional body reconstruction can be It includes the following steps:
  • Step 101 Obtain a target image, and perform segmentation processing based on the object to be reconstructed in the target image to obtain a segmented image; wherein the target image is a frame of frontal image corresponding to the object to be reconstructed.
  • the reconstruction system may first obtain the target image, and then perform segmentation processing based on the object to be reconstructed in the target image, so as to obtain the segmented image.
  • the reconstruction system may be composed of at least one terminal, or may be composed of a terminal and a server.
  • the processing flow of the method for reconstructing the three-dimensional model and texture information proposed in this application can be implemented in the terminal, in the server, or in the terminal and the server at the same time.
  • the reconstruction system can develop a mobile application program.
  • the terminal runs the program, it first uses the configured camera to collect the image of the human body, and then the image can be sent to the remote server to complete the reconstruction process of the 3D model and texture information. Then, after receiving the generated 3D reconstruction model, the terminal can directly render the animation sequence in the terminal for playback.
  • Figure 2 is a schematic diagram of AR/VR application demonstration
  • Figure 3 is a schematic diagram of AR/VR application demonstration two
  • Figure 4 is a schematic diagram of AR/VR application demonstration three, as shown in Figures 2, 3, and 4.
  • the terminal can use ARCore to put the generated virtual model, that is, the 3D reconstruction model, into the real scene to obtain the dynamic model video.
  • Figure 2 and Figure 3 show two animation models rendered in a real environment.
  • Figure 4 uses ARCore to demonstrate so that the animation model can be seen from different angles.
  • the above-mentioned terminal may be any terminal with shooting and storage functions, such as: tablet computer, mobile phone, e-reader, remote control, personal computer (PC), notebook Terminals such as computers, in-vehicle devices, Internet TVs, wearable devices, personal digital assistants (PDAs), portable media players (PMP), navigation devices, etc.
  • PC personal computer
  • notebook Terminals such as computers, in-vehicle devices, Internet TVs, wearable devices, personal digital assistants (PDAs), portable media players (PMP), navigation devices, etc.
  • the target image may be a frame of RGB image, specifically, the target image may be a frame of frontal image corresponding to the object to be reconstructed.
  • the object to be reconstructed is the human body in the target image.
  • the reconstruction system can obtain the target image in a variety of ways.
  • the reconstruction system can accept the target image sent by other devices, or it can use the configured shooting device to collect the image of the object to be processed.
  • the object to be reconstructed when the reconstruction system performs segmentation processing, the object to be reconstructed may be used as a reference to segment the object to be processed from the target image. Specifically, after the reconstruction system performs segmentation processing on the target image, the region including the object to be reconstructed and the region not including the object to be reconstructed can be obtained. Wherein, the reconstruction system may determine a part of the image including the object to be reconstructed as the segmented image, and at the same time, determine the part of the image that does not include the object to be reconstructed as other images.
  • the segmented image is the image of the object to be processed that is segmented from the target image.
  • the reconstruction system may use multiple methods to perform segmentation processing on the target image.
  • the reconstruction system can use the human clothing segmentation method to obtain the segmented image.
  • Step 102 Obtain the initial estimated shape and texture information of the object to be reconstructed based on the segmented image.
  • the reconstruction system is acquiring the target image and performing segmentation processing based on the object to be reconstructed in the target image. After the segmented image is obtained, the initial estimation model of the object to be reconstructed can be obtained based on the segmented image, and at the same time Obtain the texture information of the object to be reconstructed.
  • the texture information obtained by the reconstruction system is part of the texture information of the object to be reconstructed.
  • the reconstruction system after the reconstruction system obtains the segmented image through the segmentation process, it can fit the posture and shape of the object to be reconstructed, so as to obtain the initial estimated shape.
  • the posture of the human body is not clear. Therefore, it can rely on the parametric model of the human body shape, such as the Skinned Multi-Person Linear Model (SMPL) or the complete human body modeling and animation simulation (Shape completion and animation of people, SCAPE).
  • SMPL Skinned Multi-Person Linear Model
  • SCAPE complete human body modeling and animation simulation
  • SMPL model is a parametric human body model, which is a human body modeling method proposed by Max Planck, which can carry out arbitrary human body modeling and animation drive.
  • This method proposes a method to image the body surface appearance of the human body posture. This method can simulate the bulge and depression of human muscles during the movement of the limbs. Therefore, the surface distortion of the human body during exercise can be avoided, and the appearance of human muscle stretching and contraction can be accurately described.
  • ⁇ and ⁇ are the input parameters, where ⁇ represents the 10 parameters of the individual's height, short, fat, thin, head-to-body ratio, and ⁇ represents the overall posture and posture of the human body. 75 parameters of the relative angle of 24 joints.
  • the SCAPE model uses a deformation matrix to describe the diversity brought about by changes in the body itself.
  • the reconstruction system when it generates the initial estimated shape of the object to be reconstructed based on the segmented image, it may use the SMPL model to perform fitting processing on the segmented image to obtain the initial estimated shape.
  • the initial estimated shape obtained by fitting may include pose parameters, shape parameters, and error parameters.
  • the reconstruction system when the reconstruction system acquires the texture information of the object to be reconstructed, it can extract the pixel information of the segmented image, so that the pixel information of the object to be reconstructed can be used to determine the corresponding The texture information.
  • the reconstruction system when it obtains the texture information of the object to be reconstructed based on the segmented image, it extracts the pixel information in the segmented image to obtain the texture information.
  • the texture information can represent the front texture of the object to be reconstructed.
  • the segmented image is also the front area of the object to be reconstructed. Obtain the texture of the front side of the object to be reconstructed, while the texture of the back side is invisible.
  • Step 103 Determine the initial three-dimensional model of the object to be reconstructed through the initial estimated shape; wherein, the initial three-dimensional model is a three-dimensional model without texture.
  • the reconstruction system after the reconstruction system obtains the initial estimated shape of the object to be reconstructed based on the segmented image, it can continue to reconstruct the three-dimensional model of the object to be reconstructed on the basis of the initial estimated shape, so as to obtain the object to be reconstructed.
  • the initial 3D model without texture information.
  • the initial three-dimensional model is obtained by the reconstruction system based on the initial estimated shape of the object to be reconstructed, and is only obtained by reconstructing on the basis of the posture and contour of the object to be reconstructed. It does not include the texture of the skin, clothing, etc. of the object to be reconstructed. Therefore, the initial three-dimensional model is a three-dimensional model without texture.
  • the parametric model fitting method obtains the shape and posture of the object to be reconstructed, it does not completely conform to the actual body contour of the object to be reconstructed. That is to say, the initial estimated shape cannot accurately match the real human body contour of the object to be reconstructed. Therefore, the reconstruction system needs to correct the initial estimated shape first, and then reconstruct the initial three-dimensional model.
  • the reconstruction system when the reconstruction system determines the initial three-dimensional model of the object to be reconstructed through the initial estimated shape, it may first obtain the first silhouette of the object to be reconstructed; and then proceed according to the first silhouette and the initial estimated shape.
  • the correction processing can generate the front model and the back model of the object to be reconstructed respectively; finally, the reconstruction system further generates the initial three-dimensional model of the object to be reconstructed by splicing the front model and the back model.
  • the reconstruction system when the reconstruction system performs correction processing according to the first silhouette and the initial estimated shape to generate the frontal model of the object to be reconstructed, it may first determine the second silhouette of the initial estimated shape; and then determine the second silhouette of the initial estimated shape.
  • the reconstruction system uses the first warping function to generate the frontal model, due to the inaccurate head prediction, the face of the object to be reconstructed may be deformed. Therefore, After the reconstruction system generates the frontal model, it needs to correct the face model in the frontal model of the object to be processed, so as to repair the deformation of the face.
  • the reconstruction system may continue to restore and reconstruct the back model of the object to be reconstructed.
  • the reconstruction system may adopt a method similar to that of generating the front model. Method, the back model of the object to be reconstructed is reconstructed.
  • the initial estimated shape is obtained by fitting a segmented image including the frontal image of the object to be reconstructed. Therefore, the initial estimated shape is the frontal shape of the object to be reconstructed, and the reconstruction system The back model of the object to be reconstructed obtained from the initial estimated shape cannot be used directly, but the back shape of the object to be reconstructed needs to be determined based on the initial estimated shape.
  • the reconstruction system when it performs correction processing based on the first silhouette and the initial estimated shape to generate the back model of the object to be reconstructed, it may first perform rendering processing on the initial estimated shape to obtain the back of the object to be reconstructed. Estimate the shape; then determine the third silhouette of the estimated back shape; then determine the second mapping relationship between the first silhouette and the third silhouette; and then generate a second distortion function based on the second mapping relationship; finally use the first The second twist function generates the back model.
  • the reconstruction system when it determines the first mapping relationship and the second mapping relationship, it can use a two-dimensional non-rigid registration method to map the first silhouette to the second silhouette and the third silhouette, respectively. .
  • the reconstruction system may use the back surface culling technology to obtain the estimated back shape of the object to be reconstructed by rendering.
  • Step 104 Obtain complete texture information of the object to be reconstructed according to the texture information and the texture generation model.
  • the reconstruction system may further obtain the completed texture information of the object to be reconstructed according to the texture information and the texture generation model obtained by pre-training.
  • the reconstruction system since the texture information generated for the first time is determined by the front texture of the object to be reconstructed, in order to obtain a three-dimensional model of the object to be reconstructed with texture information, the reconstruction system needs to treat it first. The complete texture information of the reconstructed object is generated.
  • the reconstruction system can first construct a texture generation model for texture information prediction, and then can use the texture generation model to predict and generate other objects of the object to be reconstructed based on the texture information of the object to be reconstructed. With texture information, the finished texture information of the object to be reconstructed can finally be obtained.
  • the texture generation model may be obtained by training using Generative Adversarial Networks (GAN), such as InferGAN.
  • GAN Generative Adversarial Networks
  • the reconstruction system may first obtain the training data set before obtaining the complete texture information of the object to be reconstructed according to the texture information and the texture generation model, and then may use the training data set for model training , And finally complete the construction of the texture generation model.
  • the training data set may include multiple sets of texture data, where each set of texture data may include different texture images obtained by collecting the same object at different angles, that is, the first texture. Image and second texture image.
  • the texture information in the first texture image and the texture information in the second texture image are different. .
  • the reconstruction system when the reconstruction system uses the training data set for model training and generates the texture generation model, it may first determine the first contour corresponding to the first texture image, and determine the second contour corresponding to the second texture ; Then use the first texture image, the first contour, the second texture image and the second contour training to obtain a texture generation model.
  • the training of the texture generation model does not need to use a training data set with strict matching of positive and negative colors, that is to say, the reconstruction system can use any set of photos taken by the same person in different perspectives for texture generation Model training.
  • the reconstruction system when the reconstruction system obtains the complete texture information of the object to be reconstructed according to the texture information and the texture generation model, it can first input the texture information to the texture generation model, so as to output the reconstruction The predicted texture information corresponding to the object can then obtain complete texture information based on the texture information and the predicted texture information.
  • the predicted texture information may be other texture information corresponding to the object to be reconstructed, other than part of the texture information, that is, the predicted texture information is obtained by the reconstruction system through the texture generation model.
  • Other texture information that cannot be obtained from the segmented image for example, the back texture of the object to be reconstructed.
  • the reconstruction system since the predicted texture information is texture information other than the texture information, the reconstruction system combines the texture information with the predicted texture information to generate complete texture information of the object to be reconstructed.
  • Step 105 Generate a three-dimensional reconstruction model of the object to be reconstructed based on the initial three-dimensional model and the complete texture information; wherein, the three-dimensional reconstruction model is a textured three-dimensional model.
  • the reconstruction system determines the initial three-dimensional model of the object to be reconstructed through the initial estimated shape, and at the same time, after obtaining the complete texture information of the object to be reconstructed according to the texture information and the texture generation model, it can pass the initial Three-dimensional model and texture information are completed, and a three-dimensional reconstruction model of the object to be reconstructed with texture information is further generated.
  • FIG. 5 is the second schematic diagram of the realization process of the human body model reconstruction method.
  • the method for the reconstruction system to reconstruct the human body model may include the following steps:
  • Step 106 Use the animation reconstruction model to perform animation processing on the three-dimensional reconstruction model.
  • the reconstruction system after the reconstruction system completes the generation of the three-dimensional reconstruction model of the object to be reconstructed, it can add animation effects to the object to be reconstructed through the pre-trained animation reconstruction model. Specifically, the reconstruction system can use the animation reconstruction model to perform animation processing on the three-dimensional reconstruction model.
  • the reconstruction system can use the linear hybrid skin animation to reconstruct the model, transfer the skin weight of the SMPL model to the reconstructed 3D reconstruction model, and use the existing motion acquisition data to perform animation processing on it. That is to say, in this application, the reconstruction system can convert the SMPL weight parameters used as a reference to drive the final reconstruction model with animation effect, thereby not only reconstructing the static model of the object to be reconstructed, but also achieving the animation effect. Add to.
  • the reconstruction system can take a frame of RGB image as input, and simultaneously reconstruct the three-dimensional human body shape and the whole body texture map. Specifically, the reconstruction system may first perform segmentation processing on the human body shape part in the image, and then fit the segmented image with the parameterized model to obtain the initial estimated shape of the human body shape. Next, the reconstruction system can use contour-based dense correspondence to correct the initial estimated shape to obtain a front model and a back model, and then stitch together to obtain an initial three-dimensional model. Further, in order to infer the invisible back texture from the front image, the reconstruction system can use a pre-trained texture generation model to restore the invisible texture information, and then generate all the textures of the human body. Finally, the reconstruction system can use the initial 3D model and all textures to construct a 3D reconstruction model with texture information.
  • the human body model reconstruction method proposed in the embodiment of the present application uses a two-dimensional non-rigid deformation algorithm to correct the initial estimated shape to obtain the final reconstruction result; at the same time, the reconstruction system is trained to perform texture information
  • the restored texture generation model such as the InferGAN network, realizes the inference and prediction of invisible texture; further, the reconstruction system proposed in this application can reconstruct a human body 3D model with complete texture information only by relying on one frame of input image .
  • FIG. 6 is a schematic diagram of the reconstruction of the human body model.
  • the reconstruction system reconstructs the 3D human body model according to a frame of image. Based on the reconstruction results obtained. It can be seen that comprehensive experiments have been carried out on the existing image data and the collected image data.
  • the reconstruction system can use the action shooting data to manipulate and make animations, and a mobile application that demonstrates this function in the AR/VR device has been developed. program.
  • This application proposes a human body model reconstruction method.
  • the reconstruction system acquires a target image, and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is a frontal image corresponding to the object to be reconstructed; Based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are obtained; the initial estimated shape is used to determine the initial three-dimensional model of the object to be reconstructed; among them, the initial three-dimensional model is a three-dimensional model without texture; generated according to the texture information and texture The model obtains the complete texture information of the object to be reconstructed; based on the initial three-dimensional model and the complete texture information, a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation. Based on the segmented image, on the one hand, the reconstruction system can obtain On the basis of the initial estimated shape, an initial three-dimensional model corresponding to the object to be reconstructed, excluding the human body texture is generated; on the other hand, the reconstruction system can use the texture generation model to restore the reconstruction to be reconstructed by using part of the texture information extracted from the segmented image The complete texture information of the object, and finally a textured 3D reconstruction model can be obtained through the mapping of the initial 3D model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • a two-dimensional non-rigid deformation algorithm is proposed, which can correct the initial estimated shape to obtain the initial three-dimensional model.
  • FIG. 7 is a schematic diagram of two-dimensional non-rigid registration.
  • the purpose of the reconstruction system is to find the correspondence between the SMPL silhouette (black outline) and the segmented human silhouette (white outline), that is, the first silhouette and The first mapping relationship between the second silhouettes.
  • the restored SMPL grid that is, the initial estimated shape, provides a good initial guess for the reconstructed human body shape of the object to be reconstructed.
  • the contour of the initial estimated shape is not consistent with the contour of the human body in the target image. Therefore, the embodiment of the present application proposes a method for optimizing the human body shape grid, which optimizes the human body shape grid by finding the corresponding relationship between the initial SMPL estimated depth map and the restored human body shape depth map.
  • the reconstruction system may use a 2D method to replace the distortion in 3D.
  • a 2D non-rigid registration method is first used to deform the silhouette of the object to be reconstructed in the target image to match the silhouette of the initial estimated shape.
  • the point on the silhouette of the initial estimated shape to the nearest point of the silhouette of the object to be reconstructed is used to guide the deformation.
  • the distortion function generated in the process can be used to distort the depth map of the SMPL model (initial estimated shape) to the final The shape is the corresponding model.
  • the reconstruction system can apply warping functions to both the front and back of the object to be reconstructed, so as to obtain the front model and back model of the object to be reconstructed, where the two restored grids and contours of the front model and the back model are maintained. Consistent, and then based on the front model and the back model, a three-dimensional reconstruction model of the object to be reconstructed can be obtained by splicing.
  • the human silhouette (the first silhouette corresponding to the object to be reconstructed) is first mapped to the SMPL silhouette (ie, the initial estimate) The second silhouette of the shape).
  • the reconstruction system can use the deformed graph to represent the transition from the first silhouette to Movement during the second silhouette. More specifically, in this application, the goal of the reconstruction system is to solve a set of affine transformations with To parameterize the deformable movement of graph nodes. After deformation, the new position of the vertex can be expressed as the following formula:
  • ⁇ t (s) is the weighting factor of the graph node g t on the contour, where,
  • d(s 1 , g i ) is the geodesic distance between points s 1 and g i along the two-dimensional contour line.
  • R is the distance between s 1 and its k nearest neighbor graph nodes in the geodesic domain.
  • the reconstruction system can continue to minimize the following energy functions to solve the set of A and b:
  • E rigid is used to strengthen the rigidity and performance of the rotation matrix, so it can be defined as:
  • I is the identity matrix
  • the second function term E smooth is used to enhance the spatial smoothness of geometric deformation, and can be defined as the following formula:
  • ⁇ (i) is the k neighbors of node i.
  • the data item E fit is similar to the iterative closest point in a two-dimensional form, and it can be used to measure the vertex displacement between the line segments of the source contour (first silhouette) and the target contour (second silhouette).
  • the data item E fit can include two parts: point-to-point distance and point-to-plane situation:
  • s'i f(s i ,a,b), Is the closest point of si in the second silhouette.
  • n is The corresponding normal.
  • the reconstruction system after the reconstruction system determines that there is a first mapping relationship M between the first silhouette and the second silhouette, it can generate a front mesh, that is, generate a frontal model of the object to be reconstructed.
  • a front mesh that is, generate a frontal model of the object to be reconstructed.
  • the reconstruction system needs to construct a function that converts the contour correspondence into the pixel correspondence in the mask.
  • the reconstruction system may use Mean-Value Coordinates (MVC) as the deformation function.
  • MVC represents a point inside a plane triangle as a convex combination of its boundary points.
  • the re-adding system can use the weighted combination of the first silhouette set S to represent the point p:
  • the first warped function can be expressed as the following formula:
  • FIG. 8 is a schematic diagram of the model generation method.
  • the reconstruction system can calculate its corresponding point C in the SMPL mask. (p), where the third column shows the reconstruction results. It can be seen that for each pixel p in the character body, the corresponding pixel C(p) in the SMPL body can be calculated, that is, the reconstruction system can apply this deformation function to the SMPL depth map to complete the final depth map of the front grid. Then generate a frontal model of the object to be reconstructed:
  • the reconstruction system can use geometric constraints to establish a triangular mesh. Specifically, the reconstruction result is shown in Figure 4 above.
  • FIG. 9 is a schematic diagram before head correction
  • FIG. 10 is a schematic diagram after head correction.
  • the frontal model of the object to be reconstructed is generated using the first warping function
  • the face of the reconstructed object may appear deformed. This is especially obvious if the viewpoint is far away from the front camera view.
  • the general part of the body shape of the object to be reconstructed for example, the torso, upper body, and lower body
  • the head posture of the object to be reconstructed is usually predicted Inaccurate, this is because the fitting method does not consider facial feature points in the optimization process.
  • Figure 10 shows the corrected result.
  • the reconstruction system corrects the misalignment of the face area.
  • the reconstruction system can repair the facial shape through a two-step method. Specifically, after the parametric model is fitted, the reconstruction system can first roughly align the head posture. For image I, first detect the face area, and then detect the two-dimensional face landmark points. The reconstruction system can use 7 reference points (corners of eyes, nose, mouth, etc.). Then solve the 3D head pose R head by minimizing the reprojection error of the 3D reference landmark points on the image I:
  • Proj is the projection of the three-dimensional world coordinates to the two-dimensional camera coordinates
  • x is the three-dimensional landmark set
  • x 2d is the two-dimensional projection of the landmark points on the image I.
  • the reconstruction system constructs the frontal model of the object to be reconstructed, it can use the first silhouette corresponding to the object to be reconstructed and the second silhouette of the initial estimated shape to determine the first mapping relationship between the two, and then The corresponding first warping function is generated based on the first mapping relationship, and finally the frontal model can be obtained through reconstruction of the first warping function.
  • the reconstruction system may continue to reconstruct the back model of the object to be reconstructed.
  • the reconstruction system can first restore the estimated back shape of the object to be reconstructed, and then obtain the corresponding back model based on the estimated back shape, and finally obtain the initial three-dimensional model of the object to be reconstructed by stitching the front model and the back model.
  • the reconstruction system uses the same camera viewpoint as the SMPL model (initial estimated shape) that presents the front. Then, instead of drawing the triangle closest to the camera, the furthest triangle is drawn, so that the silhouette corresponding to the back view is obtained, that is, the third silhouette corresponding to the estimated shape of the back is obtained.
  • SMPL model initial estimated shape
  • the reconstruction system can use the first silhouette and the third silhouette to determine both Then, based on the second mapping relationship, a corresponding second warping function is generated; finally, the second warping function can be used to generate the back model of the object to be reconstructed.
  • FIG. 11 is a schematic diagram of a front model
  • FIG. 12 is a schematic diagram of a back model
  • FIG. 13 is a schematic diagram 1 after stitching
  • FIG. 14 is a schematic diagram 2 after stitching, as shown in FIG. 11, As shown in 12, 13, and 14.
  • the reconstruction system completes the construction of the front model and the back model, the two can be stitched together to obtain the final three-dimensional model.
  • the reconstruction system can create a series of triangles connecting the two geometric shapes of the front model and the back model along the boundary between the front model and the back model, and finally complete the construction of the three-dimensional reconstruction model of the object to be reconstructed.
  • This application proposes a human body model reconstruction method.
  • the reconstruction system acquires a target image, and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is a frontal image corresponding to the object to be reconstructed; Based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are obtained; the initial estimated shape is used to determine the initial three-dimensional model of the object to be reconstructed; among them, the initial three-dimensional model is a three-dimensional model without texture; generated according to the texture information and texture The model obtains the complete texture information of the object to be reconstructed; based on the initial three-dimensional model and the complete texture information, a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation. Based on the segmented image, on the one hand, the reconstruction system can obtain On the basis of the initial estimated shape, an initial three-dimensional model corresponding to the object to be reconstructed, excluding the human body texture is generated; on the other hand, the reconstruction system can use the texture generation model to restore the reconstruction to be reconstructed by using part of the texture information extracted from the segmented image The complete texture information of the object, and finally a textured 3D reconstruction model can be obtained through the mapping of the initial 3D model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • the reconstruction system when the reconstruction system performs texture reconstruction, it proposes an inference network that can predict invisible colors in an image, that is, a texture generation model, so that the texture generation model can be used , Use texture information prediction to obtain complete texture information.
  • the present application proposes a method of projecting the image onto the geometric body. Therefore, the terminal needs to obtain the image of the object to be reconstructed.
  • the complete texture information that is, the front texture information and the back texture information, where the front texture information, that is, the texture information obtained from the segmented image, can be used to predict the remaining texture information of the object to be reconstructed.
  • the embodiment of this application proposes a texture generation model, so that the invisible texture of the object to be reconstructed can be automatically generated.
  • the texture generation model may be an InferGAN model.
  • InferGAN can transfer texture from one input RGB image (Io) to another RGB image (It) according to the input image.
  • the intuitive assumption is that in the latent space, the distance between input and output should be approximately equal to the mathematical distance between the human body semantic segmentation Po and Pt corresponding to the images Io and It.
  • the assumptions that can be formulated are:
  • ⁇ img and ⁇ seg are image encoder and analytical encoder respectively.
  • ⁇ R is an image decoder based on hidden features of convolution.
  • this application may incorporate ⁇ img and ⁇ seg into ⁇ .
  • InferGAN encodes the texture information from I o and embeds new texels into the target body part (P t ) (for example, the back part of the inference stage).
  • the training data set may include multiple sets of texture data, and each set of texture data includes the first texture image and the second texture image obtained by collecting the same object at different angles.
  • the size of the first texture image and the second texture image can be 512 ⁇ 512.
  • the terminal may first perform segmentation processing on the first texture image and the second texture image to obtain P o after the first texture image is segmented, and P t after the second texture image is segmented.
  • the clothing of the human body can be defined as 20 parts, so P o and P t can be converted into an effective vector map (the size is 20 ⁇ 512 ⁇ 512).
  • the contour C o of the human body in the first texture image and the contour C t of the human body in the second texture image may also be used as input for training the texture generation model.
  • the contour (with a size of 1 ⁇ 512 ⁇ 512) can make the output of the texture generation network have a clearer boundary.
  • random affine transformation is performed on the input first texture image and the corresponding vector diagram to increase the geometric diversity.
  • the embodiment of this application connects all inputs (I o , Po , P t , C o , C t ) in the channel dimension, and the input size after stitching is 45 ⁇ 512 ⁇ 512.
  • the encoder composed of five convolutional blocks learns texture information from I o , and learns geometric information from P o , Co and P t, Ct. Then the latent features are passed to the processing of 9 residual blocks, which further improves the coding ability.
  • the image decoder consists of five transposed convolutional layers and a supertangent activation function.
  • the total loss L can be expressed as:
  • D and G are discriminator and generator respectively.
  • the perceptual loss l perceptual is calculated based on the output characteristics of the five different layers of VGG19. It is the layer feature of VGG19 to the image I t. In this application, different layer features can be given different scales. Since the loss of GAN is calculated in the feature space, GAN loss and perceptual loss will make the network produce clearer images.
  • the first texture image and the second texture image are not restricted, that is, the first texture image and the second texture image do not need to be restricted to the front view image and the back view image, as long as both are guaranteed It is sufficient if the image contains textures that are not identical.
  • the texture generation model InferGAN can be used to predict texture information (back texture) other than texture information (front texture) of the object to be reconstructed.
  • texture information back texture
  • the input target image I o may be segmented first to determine the semantic analysis P o and the contour C o .
  • invisible semantic analysis and contours in the target image can be analyzed Perform estimation to obtain the remaining other texture information, and then synthesize the back image texture by passing all the information to the texture generation model InferGAN
  • This application proposes a human body model reconstruction method.
  • the reconstruction system acquires a target image, and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is a frontal image corresponding to the object to be reconstructed; Based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are obtained; the initial estimated shape is used to determine the initial three-dimensional model of the object to be reconstructed; among them, the initial three-dimensional model is a three-dimensional model without texture; generated according to the texture information and texture The model obtains the complete texture information of the object to be reconstructed; based on the initial three-dimensional model and the complete texture information, a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation processing. Based on the segmented image, on the one hand, the reconstruction system can obtain On the basis of the initial estimated shape, an initial three-dimensional model corresponding to the object to be reconstructed without human body texture is generated; on the other hand, the reconstruction system can use partial texture information extracted from the segmented image to use the texture generation model to restore the reconstruction to be reconstructed The complete texture information of the object, and finally a textured 3D reconstruction model can be obtained through the mapping of the initial 3D model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • tests can be performed based on public data and collected data.
  • the terminal may use a configured camera to perform image collection for different objects to be reconstructed, and obtain images for verifying the reconstruction method of the human body model.
  • the resolution of the image may be a 4032 ⁇ 3024 image.
  • this application can use a certain specific posture when capturing images. For example, the image can be captured when the object to be reconstructed is in a posture with the feet separated and the hands open at 45 degrees.
  • the reconstruction algorithm can be deployed on the cloud server.
  • the reconstruction time is about 1 minute.
  • Figure 15 is the first schematic diagram of the reconstructed three-dimensional model
  • Figure 16 is the schematic diagram two of the reconstructed three-dimensional model
  • Figure 17 is the schematic diagram three of the reconstructed three-dimensional model
  • Figure 18 is the schematic diagram four of the reconstructed three-dimensional model, as shown in Figure 15 is the input target image
  • Figure 16 shows the reconstructed geometry of the human body model.
  • Figures 17 and 18 show the results of observing the model from different angles after using full texture mapping.
  • the human body model reconstruction method proposed in this application can accurately restore the human body shape of the object to be reconstructed, so that the reconstructed geometric contour is very consistent with the input target image, and at the same time, under the premise that the input target image is a single frame image , Can also predict and generate invisible texture information.
  • the human body model reconstruction method proposed in this application is compared with other 3D human body shape reconstruction methods on the PeopleSnapshot data set.
  • volumetric 3D human body shape prediction (Volumetric inference of 3d human body shapes, Bodynet) is a voxel-based estimation method of human body shape, pose and shape; Silhouette-based clothed people (SICLOPE) )
  • SICLOPE Silhouette-based clothed people
  • HMR End-toend Recovery of Human Shape and Pose
  • the Video Shapes method can use 120 images of the same person from different angles to merge into a complete human shape model.
  • this application can use one frame of image to reconstruct the human body model and texture information.
  • the 3D reconstruction model of the object to be reconstructed obtained by this application has more details and is relatively more detailed.
  • the calculation cost of this application is much smaller.
  • this application in order to make the finally generated 3D reconstruction model have animation effects, this application can transfer the SMPL parameterized model to the reconstruction model, and apply the motion capture in the CMU dataset (cmudataset) data.
  • FIG. 19 is a schematic diagram of the animation reconstruction result. As shown in FIG. 19, this application applies a jogging action sequence to the three-dimensional reconstruction model to generate a jogging animation effect. It can be seen that the animation deformation obtained in this application is reasonable and sufficient for some AR/VR applications.
  • InferGAN texture generation model
  • methods such as random cropping, affine transformation, and flipping are used to increase the amount of data. It can be said that this network is trained for 20 periods, and finally InferGAN is obtained.
  • InferGAN the qualitative results of InferGAN are tested based on the DeepFasion data set, and the reconstruction system can predict the target view image texture through any input view image texture.
  • InferGAN can generate a realistic bottom texture based on the whole body source input.
  • InferGAN is used to synthesize a reasonable upper texture while maintaining the original texture of the bottom.
  • FIG. 20 is a schematic diagram of a human body model reconstruction system.
  • the human body model reconstruction method proposed in an embodiment of the present application can use one frame of image to reconstruct the human body model and texture information.
  • two-dimensional non-rigid registration can be used to deform the geometry from the initial initial estimated shape to the final initial three-dimensional model.
  • a pre-trained texture generation model can be used to infer textures from different perspectives by using textures under the forward perspective, and finally complete texture information of the object to be reconstructed can be obtained.
  • the weights and joint positions of the model can be parameterized to obtain an animation model to generate animation effects.
  • This application proposes a human body model reconstruction method.
  • the reconstruction system acquires a target image, and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is a frontal image corresponding to the object to be reconstructed; Based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are obtained; the initial estimated shape is used to determine the initial three-dimensional model of the object to be reconstructed; among them, the initial three-dimensional model is a three-dimensional model without texture; generated according to the texture information and texture The model obtains the complete texture information of the object to be reconstructed; based on the initial three-dimensional model and the complete texture information, a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation processing. Based on the segmented image, on the one hand, the reconstruction system can obtain On the basis of the initial estimated shape, an initial three-dimensional model corresponding to the object to be reconstructed without human body texture is generated; on the other hand, the reconstruction system can use partial texture information extracted from the segmented image to use the texture generation model to restore the reconstruction to be reconstructed With the complete texture information of the object, a textured 3D reconstruction model can be obtained through texture mapping through the initial 3D model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • FIG. 21 is a schematic diagram 1 of the composition structure of the reconstruction system.
  • the reconstruction system 10 proposed in the embodiment of the present application may include an acquisition part 11 and a segmentation part 12.
  • the acquiring part 11 is configured to acquire a target image; wherein the target image is a frame of frontal image corresponding to the object to be reconstructed;
  • the segmentation part 12 is configured to perform segmentation processing based on the object to be reconstructed in the target image to obtain a segmented image
  • the acquiring part 11 is further configured to respectively acquire the initial estimated shape and texture information of the object to be reconstructed based on the segmented image;
  • the reconstruction part 13 is configured to determine an initial three-dimensional model of the object to be reconstructed through the initial estimated shape; wherein the initial three-dimensional model is a textureless three-dimensional model;
  • the obtaining part 11 is further configured to obtain complete texture information of the object to be reconstructed according to the texture information and the texture generation model;
  • the generating part 14 is configured to generate a three-dimensional reconstruction model of the object to be reconstructed based on the initial three-dimensional model and the complete texture information; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • the acquiring part 11 is specifically configured to use an SMPL model to perform fitting processing on the segmented image to obtain the initial estimated shape; and extract pixels in the segmented image Information to obtain the texture information.
  • the reconstruction part 13 is specifically configured to obtain a first silhouette of the object to be reconstructed; perform correction processing according to the first silhouette and the initial estimated shape to generate the The front model and the back model of the object to be reconstructed; and the splicing process is performed on the front model and the back model to generate the initial three-dimensional model.
  • the reconstruction part 13 is further specifically configured to determine a second silhouette of the initial estimated shape; determine a first mapping between the first silhouette and the second silhouette Relationship; based on the first mapping relationship, generating a first warping function; using the first warping function to generate the frontal model.
  • the correcting part 15 is configured to perform correction processing on the face model in the frontal model after the frontal model is generated by using the first distortion function.
  • the reconstruction part 13 is further specifically configured to perform rendering processing on the initial estimated shape to obtain the estimated back shape of the object to be reconstructed; and determine the first estimated shape of the back Three silhouettes; determining a second mapping relationship between the first silhouette and the third silhouette; generating a second distortion function based on the second mapping relationship; generating the back model using the second distortion function.
  • the obtaining part 11 is further configured to obtain a training data set before obtaining the complete texture information of the object to be reconstructed according to the texture information and the texture generation model; where
  • the training data set includes multiple sets of texture data, and each set of texture data includes a first texture image and a second texture image obtained by collecting the same object at different angles;
  • the generating part 14 is further configured to use the training data set for model training to generate the texture generation model.
  • the generating part 14 is specifically configured to determine a first contour corresponding to the first texture image, and determine a second contour corresponding to the second texture; using the first The texture image, the first contour, the second texture image, and the second contour are trained to obtain the texture generation model.
  • the acquisition part 11 is further specifically configured to input the texture information into the texture generation model and output predicted texture information; wherein, the predicted texture information is the pending texture information.
  • Other texture information other than the texture information corresponding to the reconstructed object; obtaining the complete texture information based on the texture information and the predicted texture information.
  • the processing part 16 is configured to generate a three-dimensional reconstruction model of the object to be reconstructed based on the initial three-dimensional model and the complete texture information, and then use the animation reconstruction model to The three-dimensional reconstruction model is animated.
  • the target image is a frame of RGB image.
  • Fig. 22 is a second schematic diagram of the composition structure of the reconstruction system.
  • the reconstruction system 10 proposed in the embodiment of the present application may also include a processor 17, a memory 18 storing executable instructions of the processor 17, and further, the reconstruction system 10 may also include a memory communication interface 19, and a bus 110 for connecting the processor 17, the memory 18, and the memory communication interface 19.
  • the aforementioned processor 17 may be an Application Specific Integrated Circuit (ASIC), a digital signal processor (Digital Signal Processor, DSP), or a digital signal processing device (Digital Signal Processing Device, DSPD). ), Programmable Logic Device (ProgRAMmable Logic Device, PLD), Field Programmable Gate Array (Field ProgRAMmable Gate Array, FPGA), Central Processing Unit (CPU), Controller, Microcontroller, Microprocessor At least one of. It can be understood that, for different devices, the electronic devices used to implement the above-mentioned processor functions may also be other, which is not specifically limited in the embodiment of the present application.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Signal Processor
  • DSPD Digital Signal Processing Device
  • PLD Programmable Logic Device
  • Field Programmable Gate Array Field ProgRAMmable Gate Array
  • CPU Central Processing Unit
  • Controller Microcontroller
  • Microprocessor At least one of.
  • the reconstruction system 10 may also include a memory 18, which may be connected to the processor 17, where the memory 18 is used to store executable program code, the program code includes computer operation instructions, the memory 18 may include high-speed RAM memory, or may also Including non-volatile memory, for example, at least two disk memories.
  • the bus 110 is used to connect the memory communication interface 19, the processor 17, the memory 18, and the mutual communication among these devices.
  • the memory 18 is used to store instructions and data.
  • the above-mentioned processor 17 is configured to obtain a target image, and perform segmentation processing based on the object to be reconstructed in the target image to obtain a segmented image; wherein, the target image is all A frontal image corresponding to the object to be reconstructed; based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are respectively obtained; through the initial estimated shape, the initial three-dimensional model of the object to be reconstructed is determined Wherein, the initial three-dimensional model is a three-dimensional model without texture; according to the texture information and the texture generation model, the complete texture information of the object to be reconstructed is obtained; and the complete texture information is generated based on the initial three-dimensional model and the complete texture information The three-dimensional reconstruction model of the object to be reconstructed; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • the aforementioned memory 18 may be a volatile memory (volatile memory), such as a random-access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory.
  • volatile memory such as a random-access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory.
  • RAM random-access memory
  • non-volatile memory such as a read-only memory.
  • One storage device Read-Only Memory, ROM), flash memory (flash memory), hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and Instructions and data are provided to the processor 17.
  • the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be realized in the form of hardware or software function module.
  • the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this embodiment is essentially or correct
  • the part that the prior art contributes or all or part of the technical solution can be embodied in the form of a software product.
  • the computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer).
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.
  • An embodiment of the present application proposes a reconstruction system, which acquires a target image and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is a frame of frontal image corresponding to the object to be reconstructed ; Based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are obtained; through the initial estimated shape, the initial three-dimensional model of the object to be reconstructed is determined; among them, the initial three-dimensional model is a three-dimensional model without texture; according to the texture information and texture Generate a model to obtain the complete texture information of the object to be reconstructed; generate a 3D reconstruction model of the object to be reconstructed based on the initial 3D model and the complete texture information; among them, the 3D reconstruction model is a textured 3D model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation processing. Based on the segmented image, on the one hand, the reconstruction system can obtain On the basis of the initial estimated shape, an initial three-dimensional model corresponding to the object to be reconstructed without human body texture is generated; on the other hand, the reconstruction system can use partial texture information extracted from the segmented image to use the texture generation model to restore the reconstruction to be reconstructed With the complete texture information of the object, a textured 3D reconstruction model can be obtained through texture mapping through the initial 3D model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.
  • the embodiment of the present application provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned method for reconstructing a human body model is realized.
  • the program instructions corresponding to the method for reconstructing a human body model in this embodiment can be stored on storage media such as an optical disk, a hard disk, and a USB flash drive.
  • storage media such as an optical disk, a hard disk, and a USB flash drive.
  • the target image is a frame of frontal image corresponding to the object to be reconstructed;
  • the initial three-dimensional model is a three-dimensional model without texture
  • a three-dimensional reconstruction model of the object to be reconstructed is generated; wherein the three-dimensional reconstruction model is a textured three-dimensional model.
  • this application can be provided as a method, a terminal, or a computer program product. Therefore, this application may adopt the form of hardware embodiment, software embodiment, or a combination of software and hardware embodiments. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.
  • These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device realizes the functions specified in one or more processes in the schematic diagram and/or one block or more in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in one or more processes in the schematic diagram and/or one block or more in the block diagram.
  • the embodiment of the application proposes a human body model reconstruction method, a reconstruction system and a storage medium.
  • the reconstruction system obtains a target image, and performs segmentation processing based on the object to be reconstructed in the target image to obtain the segmented image; wherein the target image is to be reconstructed
  • a frontal image corresponding to the object based on the segmented image, the initial estimated shape and texture information of the object to be reconstructed are obtained; the initial estimated shape is used to determine the initial three-dimensional model of the object to be reconstructed; the initial three-dimensional model is a textureless three-dimensional Model;
  • the complete texture information of the object to be reconstructed is obtained; based on the initial 3D model and the complete texture information, a 3D reconstruction model of the object to be reconstructed is generated; among them, the 3D reconstruction model is a textured 3D model.
  • the reconstruction system can obtain the segmented image corresponding to the object to be reconstructed through segmentation. Based on the segmented image, on the one hand, the reconstruction system can obtain On the basis of the initial estimated shape, an initial three-dimensional model corresponding to the object to be reconstructed, excluding the human body texture is generated; on the other hand, the reconstruction system can use the texture generation model to restore the reconstruction to be reconstructed by using part of the texture information extracted from the segmented image The complete texture information of the object, and finally a textured 3D reconstruction model can be obtained through the mapping of the initial 3D model and the complete texture information.
  • the human body model reconstruction method proposed in this application can use one frame of image to accurately reconstruct the three-dimensional model of the human body, and at the same time can restore the complete texture information of the human body, thereby improving the accuracy of model reconstruction and greatly optimizing the model. Rebuild the effect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Architecture (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Image Generation (AREA)

Abstract

一种人体模型重建方法、重建系统及存储介质，人体模型重建方法包括：获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。

Description

人体模型重建方法、重建系统及存储介质
本申请基于申请号为62/908,314、申请日为2019年9月30日、申请名称为“3D HUMAN AVATAR DIGITIZATION FROM A SINGLE IMAGE”的在先美国临时专利申请提出，并要求该在先美国临时专利申请的优先权，该在先美国临时专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请实施例涉及计算机视觉与三维人体重建领域，尤其涉及一种人体模型重建方法、重建系统及存储介质。
背景技术
高精度人体三维重建是计算机视觉、计算机图形学等领域的一个重点问题，其旨在快速、准确地从人体图像中重建出三维人体参数化模型，此技术可广泛应用于动作分析、视觉特效、虚拟现实、电子商务等多个领域，具有很高的研究和商用价值。
在进行人体的模型重建时，一般需要使用复杂的装置和重建算法，而这样的方式无法适应于日常场景。虽然可以仅使用一帧图像进行模型的重建，但是重建获得的三维模型准确性较差，且无法对人体的全部纹理信息进行还原。
可见，目前的模型重建方法，无法通过简单的重建流程获得准确性高，且包含有全部纹理信息的人体三维模型。
发明内容
本申请实施例提供一种人体模型重建方法、重建系统及存储介质，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的全部纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
本申请实施例的技术方案是这样实现的：
第一方面，本申请实施例提供了一种人体模型重建方法，所述方法包括：
获取目标图像，并基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；
基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；
通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；
根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；
基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
第二方面，本申请实施例提供了一种重建系统，所述重建系统包括：获取部分，分割部分，重建部分，生成部分，
所述获取部分，配置为获取目标图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；
所述分割部分，配置为基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；
所述获取部分，还配置为基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；
所述重建部分，配置为通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；
所述获取部分，还配置为根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；
所述生成部分，配置为基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
第三方面，本申请实施例提供了一种重建系统，所述重建系统包括处理器、存储有所述处理器可执行指令的存储器，当所述指令被所述处理器执行时，实现如上所述的人体模型重建方法。
第四方面，本申请实施例提供了一种计算机可读存储介质，其上存储有程序，应用于重建系统中，所述程序被处理器执行时，实现如上所述的人体模型重建方法。
本申请实施例提供一种人体模型重建方法、重建系统及存储介质，重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息的映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
附图说明
图1为人体模型重建方法的实现流程示意图一；
图2为AR/VR应用程序演示的示意图一；
图3为AR/VR应用程序演示的示意图二；
图4为AR/VR应用程序演示的示意图三；
图5为人体模型重建方法的实现流程示意图二；
图6为人体模型重建的示意图；
图7为二维非刚性配准的示意图；
图8为模型生成方法的示意图；
图9为头部校正前的示意图；
图10为头部校正后的示意图；
图11为正面模型的示意图；
图12为背部模型的示意图；
图13为拼接后的示意图一；
图14为拼接后的示意图二；
图15为重建三维模型的示意图一；
图16为重建三维模型的示意图二；
图17为重建三维模型的示意图三；
图18为重建三维模型的示意图四；
图19为动画重建结果的示意图；
图20为人体模型重建的系统示意图；
图21为重建系统的组成结构示意图一；
图22为重建系统的组成结构示意图二。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述。可以理解的是，此处所描述的具体实施例仅仅用于解释相关申请，而非对该申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与有关申请相关的部分。
高精度三维人体重建是计算机视觉、计算机图形学等领域的一个重点问题，其旨在快速、准确地从人体图像中重建出三维人体参数化模型，此技术可广泛应用于动作分析、视觉特效、虚拟现实、电子商务等多个领域，具有很高的研究和商用价值。
随着立体视觉技术的发展和虚拟现实技术(Virtual Reality，VR)、增强现实技术(Augmented Reality，AR)的逐渐普及，如何获得日常生活中真实的人体三维模型已经成为研究的热点和难点，特别是在动态场景中捕捉人体运动；从而可以使真实的人体三维模型应用在数字动画，影视级人物动作特效采集和远程会议等。
实时三维人体重建主要包含两方面：几何优化和纹理贴图。几何优化指准确生成人体三维模型的过程；该过程确定模型的几何拓扑结构，给出相对准确的几何信息，但是丢失了所优化目标的纹理信息。纹理映射是将纹理像素映射到三维物体表面的过程；通过纹理映射可以恢复人体模型的纹理信息，增加模型的真实感。因此，在重建生动的人体面貌时，高质量的纹理映射起到至关重要的作用。
早期，Kanade和Narayanan使用了一个直径为5米的穹顶，并在其上安装了51个摄像头来将真实物体数字化到自由视点视频(free viewpoint video，fvv)。近期的工作中，拍摄装置倾向于使用更高分辨率和速度的工业级同步相机，例如，卡内基梅隆大学的全光学工作室由480个视频图形阵列(Video Graphics Array，VGA)摄像机、31个高清摄像机和10个Kinect传感器组成，用于重建和恢复多种人类活动。可见，上述三维人体重建技术很大程度上是由庞大的三维采集系统和复杂的重建算法实现的，在日常场景中使用如此复杂的装置是不实际的。
有一些研究可以从一张或几张照片中恢复人体的姿态和近似形状，但这样的重建方法精度较低；使用非刚体变形方法可以从视频片段中重建人体形状，但需要预先捕获对象的模板。新兴的基于学习的方法，利用卷积神经网络(Convolutional Neural Networks，CNN)通过对大量合成图像进行训练来获得高分辨率的结果，但仍然存在三维模型重建结果与二维图像不对齐等问题，且由于训练数据的偏差，该方法并不适用于所有类型的输入。
此外，在使用上述方法，利用一帧图像进行三维人体重建时，即使能够重建整个人体形状，也只有部分纹理信息，无法重建完成的纹理信息。Natsume等人使用大量合成图像训练模型，推断被遮挡部分的颜色，然而，由此产生的色彩缺乏光感。
也就是说，目前在重建人体的三维模型时，如果使用单帧图像，就无法获得准确性高的人体三维模型，即重建精度低，且在重建后的三维模型中，无法对人体的完整纹理信息进行还原，即重建效果差。
为解决现有技术中存在的缺陷，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用 纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息的映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
具体地，在本申请中，重建系统可以基于输入的单帧人体正面图像，重建人体的三维模型和完整的纹理信息。首先，重建系统可以将输入的单帧图像进行分割，继而将骨骼蒙皮多人线性模型(Skinned Multi-Person Linear Model，SMPL)模型拟合至分割后的图像。由于SMPL模型不能很好地与图像中的人体轮廓对齐，因此，在本申请中，重建系统可以继续对SMPL模型进行变形。其中，重建系统可以通过寻找人形的剪影与拟合后的SMPL模型的剪影之间的映射关系来实现模型的变形。然后将SMPL模型的深度图校正为最终形状，即获得人体的正面模型。接着，重建系统可以使用相同的方法重建人体的背部模型，接着可以将正面模型和背部模型缝合在一起，获得人体的初始三维模型，然而，该初始三维模型中不能体现人体的全部纹理信息，为了恢复人体背面的不可见纹理，在本申请中，重建系统可以使用纹理生成模型，即InferGAN网络，基于人体的可见的部分纹理信息还原不可见的其他纹理信息，最终便可以构建人体的全部纹理信息。最终，通过人体的初始三维信息和全部纹理信息，重建系统便可以获得包含有人体完整纹理的三维重建模型，从而基于一帧输入图像完成了人体模型重建，其中，人体模型重建具体可以包括三维人体建模和纹理重建。
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述。
本申请一实施例提供了一种人体模型重建方法，图1为人体模型重建方法的实现流程示意图一，如图1所示，在本申请的实施例中，重建系统进行三维人体重建的方法可以包括以下步骤：
步骤101、获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像。
在本申请的实施例中，重建系统可以先获取目标图像，然后基于目标图像中的待重建对象进行分割处理，从而可以获得分割后图像。
进一步地，在本申请的实施例中，重建系统可以由至少一个终端构成，也可以由终端和服务器构成。具体地，本申请所提出的三维模型和纹理信息的重建方法的处理流程可以在终端中实现，也可以在服务器中实现，还可以同时在终端和服务器中实现。
示例性的，重建系统可以开发一个移动端的应用程序，当终端运行该程序时，先使用配置的拍摄装置采集人体图像，然后可以将该图像发送到远程服务器中完成三维模型和纹理信息的重建过程，接着，终端在接收到生成的三维重建模型之后，可以直接在终端中渲染动画序列进行回放。图2为AR/VR应用程序演示的示意图一，图3为AR/VR应用程序演示的示意图二，图4为AR/VR应用程序演示的示意图三，如图2、3、4所示，本申请中，终端可以使用ARCore将生成的虚拟模型，即三维重建模型放到真实场景中，以获得动态模型视频。其中，图2和图3显示了在真实环境中渲染的两个动画模型。图4使用ARCore演示，这样可以从不同的角度看到动画模型。
需要说明的是，在本申请的实施例中，上述终端可以为任何具备拍摄和存储功能的终端，例如：平板电脑、手机、电子阅读器、遥控器、个人计算机(Personal Computer，PC)、笔记本电脑、车载设备、网络电视、可穿戴设备、个人数字助理(Personal Digital Assistant，PDA)、便捷式媒体播放器(Portable Media Player，PMP)、导航装置等终端。
进一步地，在本申请的实施例中，目标图像可以为一帧RGB图像，具体地，目标图像可以为待重建对象对应的一帧正面图像。其中，待重建对象即为目标图像中的人体。
可以理解的是，在本申请的实施例中，重建系统可以通过多种方式获取目标图像，例 如，重建系统可以接受其他设备发送的目标图像，也可以通过配置的拍摄装置对待处理对象进行图像采集，获得目标图像，也可以通过访问网站下载目标图像，还可以从存储地址中读取目标图像。
进一步地，在本申请的实施例中，重建系统在进行分割处理时，可以以待重建对象为基准，将待处理对象从目标图像中分割出来。具体地，重建系统对目标图像进行分割处理以后，可以获得包括待重建对象的区域，和不包括待重建对象的区域。其中，重建系统可以将包括有待重建对象的部分图像确定为分割后图像，同时，将不包括待重建对象的部分图像确定为其他图像。
也就是说，在本申请中，分割后图像为从目标图像中分割而来的、待处理对象的图像。
可以理解的是，在本申请的实施例中，重建系统可以采用多种方式对目标图像进行分割处理。例如，重建系统可以使用人体服装分割方法获取分割后图像。
步骤102、基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息。
在本申请的实施例中，重建系统在获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像之后，可以基于分割后图像，获得待重建对象的初始估计模型，同时获得待重建对象的纹理信息。其中重建系统获得的纹理信息为待重建对象的部分纹理信息。
需要说明的是，在本申请的实施例中，重建系统在通过分割处理获得分割后图像之后，可以对待重建对象的姿态和形状进行拟合，从而可以获得初始估计形状。
进一步地，在本申请中，一般来说，对于分割后图像中的待重建对象，人体的姿势是不明确的。因此，可以依赖于人体形状的参数化模型，例如骨骼蒙皮多人线性模型(Skinned Multi-Person Linear Model，SMPL)或人体完整建模与动画模拟(Shape completion and animation of people，SCAPE)来拟合一个符合分割后图像中的待重建对象的姿态。
SMPL模型是一种参数化人体模型，是马普所提出的一种人体建模方法，该方法可以进行任意的人体建模和动画驱动。这种方法提出了人体姿态影像体表形貌的方法，这种方法可以模拟人的肌肉在肢体运动过程中的凸起和凹陷。因此可以避免人体在运动过程中的表面失真，可以精准的刻画人的肌肉拉伸以及收缩运动的形貌。
在使用SMPL模型进行人体建模的方法中，β和θ是其中的输入参数，其中，β代表是个人体高矮胖瘦、头身比等比例的10个参数，θ是代表人体整体运动位姿和24个关节相对角度的75个参数。SCAPE模型则是用变形矩阵来刻画因形体本身变化带来的多样性。
示例性的，在本申请中，重建系统在基于分割后图像生成待重建对象的初始估计形状时，可以利用SMPL模型对分割后图像进行拟合处理，获得初始估计形状。其中，拟合获得的初始估计形状中可以包含有姿势参数、形状参数以及误差参数。
可以理解的是，在本申请的实施例中，重建系统在进行待重建对象的纹理信息的获取时，可以对分割后的图像进行像素信息的提取，从而可以利用待重建对象的像素信息确定对应的纹理信息。
也就是说，在本申请中，重建系统在基于分割后图像，获得待重建对象的纹理信息时，提取分割后图像中的像素信息，从而获得纹理信息。其中，纹理信息可以表征待重建对象的正面纹理，具体地，由于目标图像为待重建对象的正面图像，因此，分割后的图像也为待重建对象的正面区域，进而在进行像素提取之后只能获得待重建对象的正面的纹理，而背面的纹理则是不可见的。
步骤103、通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型。
在本申请的实施例中，重建系统在基于分割后图像，获得待重建对象的初始估计形状之后，可以继续在初始估计形状的基础上对待重建对象的三维模型进行重建，从而可以获得待重建对象的、不具有纹理信息的初始三维模型。
需要说明的是，在本申请的实施例中，初始三维模型为重建系统基于待重建对象的初始估计形状重建获得的，仅仅是对在待重建对象的姿势和轮廓的基础上进行重建获得的，并不包含待重建对象的皮肤、衣物等的纹理，因此，初始三维模型为无纹理的三维模型。
可以理解的是，在本申请的实施例中，参数化模型拟合方法虽然获取了待重建对象的形状和姿态，但并不是完全符合待重建对象的实际的身体轮廓。也就是说，初始估计形状并不能精确地匹配待重建对象真实的人体轮廓，因此，重建系统需要先对初始估计形状进行校正，进而重建初始三维模型。
进一步地，在本申请的实施例中，重建系统在通过初始估计形状，确定待重建对象的初始三维模型时，可以先获取待重建对象的第一剪影；然后根据第一剪影和初始估计形状进行校正处理，从而可以分别生成待重建对象的正面模型和背部模型；最后，重建系统通过对正面模型和背部模型进行拼接处理，进一步生成待重建对象的初始三维模型。
示例性的，在本申请的实施例中，重建系统在根据第一剪影和初始估计形状进行校正处理，生成待重建对象的正面模型时，可以先确定初始估计形状的第二剪影；然后确定第一剪影和第二剪影之间的第一映射关系；进而可以基于第一映射关系，生成第一扭曲函数；最终便可以利用第一扭曲函数生成正面模型。
需要说明的是，在本申请的实施例中，重建系统在利用第一扭曲函数生成正面模型之后，由于存在头部预测不准确的原因，待重建对象的面部可能会出现变形的情况，因此，重建系统在生成正面模型之后，需要对待处理对象的正面模型中的面部模型进行修正处理，从而修复面部的变形。
进一步地，在本申请的实施例中，重建系统在生成待重建对象的正面模型之后，可以继续对待重建对象的背部模型进行恢复和重建，具体地，重建系统可以采用与生成正面模型相类似的方法，对待重建对象的背部模型进行重建。
需要说明的是，在本申请的实施例中，初始估计形状是对包括待重建对象的正面图像的分割后图像进行拟合获得的，因此，初始估计形状为待重建对象的正面形状，重建系统并不能直接使用初始估计形状获得的待重建对象的背部模型，而是需要先基于初始估计形状确定待重建对象的背部形状。
示例性的，在本申请中，重建系统在根据第一剪影和初始估计形状进行校正处理，生成待重建对象的背部模型时，可以先对初始估计形状进行渲染处理，以获得待重建对象的背部估计形状；然后再确定背部估计形状的第三剪影；再对第一剪影和第三剪影之间的第二映射关系进行确定；进而可以基于第二映射关系，生成第二扭曲函数；最终利用第二扭曲函数生成背部模型。
具体地，在本申请中，重建系统在进行第一映射关系和第二映射关系的确定时，可以采用二维非刚性配准的方法将第一剪影分别对应到第二剪影和第三剪影上。
进一步地，在本申请的实施例中，重建系统在进行背部模型的生成之前，可以使用背面剔除技术，渲染获得待重建对象的背部估计形状。
步骤104、根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息。
在本申请的实施例中，重建系统在基于分割后图像，获得待重建对象的纹理信息之后，可以进一步根据纹理信息和预先训练获得的纹理生成模型，获得待重建对象的完成纹理信息。
需要说明的是，在本申请的实施例中，由于初次生成的纹理信息是对待重建对象的正面纹理进行确定的，为了获得待重建对象的、具有纹理信息的三维模型，重建系统便需要先对待重建对象的完整纹理信息进行生成。
进一步地，在本申请的实施例中，重建系统可以先构建用于进行纹理信息预测的纹理生成模型，然后便可以利用纹理生成模型，基于待重建对象的纹理信息，预测生成待重建对象的其他纹理信息，最终便可以获得待重建对象的完成纹理信息。
示例性的，在本申请的实施例中，纹理生成模型可以为使用生成式对抗网络(Generative Adversarial Networks，GAN)训练获得的，例如InferGAN。
需要说明的是，在本申请的实施例中，重建系统在根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息之前，可以先获取训练数据集，然后可以使用训练数据集进行模型训练，最终便完成纹理生成模型的构建。
可以理解的是，在本申请的实施例中，训练数据集中可以包括多组纹理数据，其中，每组纹理数据中可以包括有按照不同角度采集相同对象所获得的不同纹理图像，即第一纹理图像和第二纹理图像。
具体地，在本申请中，由于第一纹理图像和第二纹理图像是分别基于不同的拍摄角度拍摄获得的，因此，第一纹理图像中纹理信息和第二纹理图像中的纹理信息是不同的。
进一步地，在本申请的实施例中，重建系统在使用训练数据集进行模型训练，生成纹理生成模型时，可以先确定第一纹理图像对应的第一轮廓，确定第二纹理对应的第二轮廓；然后利用第一纹理图像、第一轮廓、第二纹理图像以及第二轮廓训练获得纹理生成模型。
需要说明的是，在本申请中，纹理生成模型的训练不需要使用正反颜色严格匹配的训练数据集，也就是说，重建系统可以使用同一个人在不同视角拍摄的任意一组照片进行纹理生成模型的训练。
需要说明的是，在本申请的实施例中，重建系统在根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息时，可以先将纹理信息输入至纹理生成模型，从而可以输出待重建对象所对应的预测纹理信息，然后可以基于纹理信息和预测纹理信息，获得完整纹理信息。
可以理解的是，在本申请的实施例中，预测纹理信息可以为待重建对象对应的、部分纹理信息以外的其他纹理信息，也就是说，预测纹理信息是重建系统通过纹理生成模型获得的、无法从分割后图像中获取的其他纹理信息，例如，待重建对象的背部纹理。
进一步地，在本申请的实施例中，由于预测纹理信息为纹理信息以外的其他纹理信息，因此，重建系统将纹理信息与预测纹理信息进行结合，便可以生成待重建对象的完整纹理信息。
步骤105、基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。
在本申请的实施例中，重建系统在通过初始估计形状，确定待重建对象的初始三维模型，同时，在根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息之后，便可以通过初始三维模型和完成纹理信息，进一步生成待重建对象的、具有纹理信息的三维重建模型。
进一步地，在本申请的实施例中，图5为人体模型重建方法的实现流程示意图二，如图5所示，基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型之后，即步骤105之后，重建系统进行人体模型重建的方法可以包括以下步骤：
步骤106、利用动画重建模型对三维重建模型进行动画处理。
在本申请的实施例中，重建系统在完成待重建对象的三维重建模型的生成之后，可以通过预先训练的动画重建模型为待重建对象增加动画效果。具体地，重建系统可以利用动画重建模型对三维重建模型进行动画处理。
需要说明的是，在本申请中，重建系统可以使用线性混合皮肤动画重建模型，将SMPL模型的皮肤权重转移到重建的三维重建模型上，并使用现有的动作采集数据对其进行动画处理。也就是说，在本申请中，重建系统可以将作为参考的SMPL权重参数进行转换，来驱动最后的具有动画效果的重建模型，从而不仅重建了待重建对象的静态模型，还实现了动画效果的添加。
通过上述步骤101至步骤111所提出的人体模型重建方法，重建系统可以以一帧RGB 图像为输入，同时重建三维人体形状和全身纹理图。具体地，重建系统可以先对图像中的人体形状部分进行分割处理，然后将分割后的图像与参数化模型进行拟合，得到人体形状的初始估计形状。接下来，重建系统可以使用基于轮廓的密集对应对初始估计形状进行校正处理，得到正面模型和背部模型，进而拼接获得初始三维模型。进一步地，为了从正面图像中推断出不可见的背面纹理，重建系统可以使用预先训练的纹理生成模型，恢复不可见的纹理信息，进而生成人体的全部纹理。最终，重建系统便可以利用初始三维模型和全部纹理，构建具有纹理信息的三维重建模型。
具体地，本申请的实施例提出的人体模型重建方法，使用了一种二维非刚性变形算法，对初始估计形状进行校正，从而得到最终重建结果；同时，重建系统训练了可以对纹理信息进行还原的纹理生成模型，如InferGAN网络，从而实现了对不可见纹理的推理和预测；进一步地，本申请提出的重建系统，仅依赖一帧输入图像便可以重建一个具有完整纹理信息的人体三维模型。
示例性的，在本申请中，还开发了一个使用软件平台ARCore的移动端的应用程序来演示它在AR中的应用。具体地，在本申请中，图6为人体模型重建的示意图，如图6所示，重建系统根据一帧图像重建了3D人体模型其中，图6展示了在采集的图像数据和PeopleSnapshot数据集的基础上获得的重建结果。可见，对已有的图像数据和采集的图像数据进行了综合实验，重建系统可以利用动作拍摄数据来操纵和制作动画，且开发了一个在AR/VR装置中演示了这一功能的移动端的应用程序。
本申请提出了一种人体模型重建方法，重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息的映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
基于上述实施例，在本申请的再一实施例中，重建系统在进行三维人体建模时，提出了一种二维非刚性变形算法，可以将初始估计形状校正得到初始三维模型。
在将待重建对象的人体形状从目标图像中分割出来，获得分割后图像之后，重建系统可以将SMPL模型拟合到分割后图像。图7为二维非刚性配准的示意图，如图7所示，本重建系统的目的是找到SMPL剪影(黑色轮廓)和分割的人类剪影(白色轮廓)之间的对应，即第一剪影和第二剪影之间的第一映射关系。恢复的SMPL网格，即初始估计形状，为待重建对象的重建的人体形状提供了良好的初步猜测。但是，初始估计形状的轮廓与目标图像中的人体轮廓并不一致。因此，本申请实施例提出了一种优化人体形状网格的方法，该方法通过寻找初始SMPL估计深度图与复原人体形状深度图的对应关系来对人体形状网格进行优化。
具体地，在本申请的实施例中，重建系统可以使用2D方法来代替3D中的扭曲。在本申请中，首先使用2D非刚性配准方法使目标图像中待重建对象的剪影变形，以匹配初始估计形状的剪影。从初始估计形状的剪影上的点到待重建对象的剪影的最近点被用于引导 变形，在此过程中产生的扭曲函数，可用于将SMPL模型(初始估计形状)的深度图扭曲到最终的形状，即对应的模型。在本申请中，重建系统可以对待重建对象的正面和背部都应用扭曲函数，从而获得待重建对象的正面模型和背部模型，其中，正面模型和背部模型这两个恢复的网格与轮廓都保持一致，进而可以基于正面模型和背部模型，拼接获得待重建对象的三维重建模型。
需要说明的是，在本申请的实施例中，为了应用扭曲函数来重建最终的形状，在本申请中，首先将人体剪影(待重建对象对应的第一剪影)对应到SMPL剪影(即初始估计形状的第二剪影)中。
可以理解的是，基于第一剪影和第二剪影，假设第一剪影为S集合，和第二剪影为T集合，其中，S有k S个顶点{s i∣i＝1，…，k S}，T有k T个顶点{t i∣i＝1，…，k T}，其中s i，t i∈R 2。然后可以从集合S里均匀抽样m个图节点G＝{g 1，g 2，...，g m}，进一步地，在本申请中，重建系统可以使用变形图来表示从第一剪影到第二剪影过程中的运动。更具体地说，在本申请中，重建系统的目标是求解一组仿射变换
Figure PCTCN2020117916-appb-000001
和
Figure PCTCN2020117916-appb-000002
来参数化表示图节点的可变形移动。在变形后，顶点的新位置可以表示为如下公式：
Figure PCTCN2020117916-appb-000003
其中,ω t(s)为轮廓上图节点g t的权重因子，其中，
ω i(s 1)＝max(0,(1-d(s 1,g i) 2/R 2) 3)      (2)
其中，d(s 1,g i)为点s 1与g i沿二维轮廓线的测地线距离。R是测地线域中s 1与其k个最近邻图节点之间的距离。在本申请中，可以设置k＝4。
在完成变形图的构建之后，为了使人体形状第一剪影S变形以匹配SMPL模型的第二剪影T，在本申请中，重建系统可以继续最小化以下能量函数来求解A和b的集合：
E total＝E rigidλ rigid+E smoothλ smooth+E fit      (3)
其中，函数项E rigid用于加强旋转矩阵的刚度和性能，因此可以定义为：
Figure PCTCN2020117916-appb-000004
其中，I是单位矩阵。
第二个函数项E smooth用于加强几何变形的空间光滑性，可以定义为如下公式：
E smooth＝∑ G∑ k∈Ω(i)‖A i(g k-g i)+g i+b i-(g k+b k)‖ 2      (5)
其中，Ω(i)为节点i的k个近邻。
进一步地，数据项E fit类似于二维形式的迭代最近点，它可以用于测量源轮廓(第一剪影)和目标轮廓(第二剪影)线段之间的顶点位移。数据项E fit可以包括两部分：点对点距离和点对平面情况：
Figure PCTCN2020117916-appb-000005
其中，s’ i＝f(s i,a,b)，
Figure PCTCN2020117916-appb-000006
是s i在第二剪影中的最近点。n为
Figure PCTCN2020117916-appb-000007
对应的法线。在本申请中，可以使用λ point＝0.1和λ plane＝1。
进一步地，在本申请中，重建系统可以利用迭代高斯-牛顿算法可对整体能量函数E total进行优化。具体地，可以使用λ rigid＝50和λ smooth＝25。结果如图7所示。
需要说明的是，在本申请的实施例中，重建系统在确定第一剪影和第二剪影之间具有第一映射关系M之后，就可以生成前网格，即生成待重建对象的正面模型。在本申请中，对于S＝{s i∣i＝1，…，k S}中的每个s i，在初始估计模型中，即SMPL轮廓边界M(S)＝{M(s 1)，M(s 2)，..，(S kS)}上有一个对应点。
进一步地，在本申请中，对于待重建对象的人体掩码内的每个像素p，在SMPL掩码内可以找到其对应的像素C(p)。为了计算密集的对应关系，重建系统需要构造一个将轮廓对应关系转换为掩码内像素对应关系的函数。具体地，在本申请中，重建系统可以使用均值坐标(Mean-Value Coordinates，MVC)作为变形函数。MVC将平面三角内部的一个 点表示为其边界点的凸组合。更具体地说，重加系统可以使用第一剪影集合S的加权组合来表示点p：
Figure PCTCN2020117916-appb-000008
进一步地，在本申请中，在完成上述对应的M和MVC函数的计算之后，可以将s i替换为M(s i)，因此，第一扭曲函数可以表示为如下公式：
Figure PCTCN2020117916-appb-000009
示例性的，在本申请中，图8为模型生成方法的示意图，如图8所示，对于人形图像掩码内的每个点p，重建系统可以计算它在SMPL掩码中的对应点C(p)，其中，第三列显示重构结果。可见，对于人物体内的每个像素p，可以计算SMPL体内的对应像素C(p)，也就是说，重建系统可以将此变形函数应用于SMPL深度图中来完成最终前网格的深度图，进而生成待重建对象的正面模型：
Z(p)＝Z SMPL(C(p))        (9)
可以理解的是，在本申请中，在获得每个像素的深度之后，重建系统可以使用几何约束建立三角形网格。具体地，重建结果如上述图4所示。
进一步地，在本申请中，图9为头部校正前的示意图，图10为头部校正后的示意图，如图9所示，在使用第一扭曲函数生成待重建对象的正面模型之后，待重建对象的面部可能会出现变形。如果视点是远离正面相机视图时，这一点特别明显。具体地，在参数模型拟合的过程中，虽然待重建对象的体型的一般部分(例如，躯干，上半身，下半身)是合理的，如图9所示，待重建对象的头部姿势通常预测的不准确，这是由于拟合方法没有在优化过程中考虑面部特征点。如图10展示了校正后的结果，重建系统对人脸区域的不对准进行了校正。
为了解决这一问题，在本申请中，重建系统可以通过两步方法来修复面部形状。具体地，经过参数化模型拟合后，重建系统可以先对头部姿态进行粗略的对齐。对于图像I，首先检测人脸区域，然后检测二维的人脸地标点。重建系统可以使用7个基准点(眼角，鼻子，嘴角等)。然后通过最小化图像I上的3D基准地标点的再投影误差来求解3D头部姿态R head：
min‖Proj(R head·x)-x 2d‖          (10)
其中Proj为三维世界坐标到二维摄像机坐标的投影，x为三维地标集，x 2d为地标点在图像I上的二维投影。
综上所述，重建系统在进行待重建对象的正面模型的构建时，可以使用待重建对象对应的第一剪影和初始估计形状的第二剪影，确定两者之间的第一映射关系，然后基于第一映射关系生成对应的第一扭曲函数，最终便可以通过第一扭曲函数重建获得正面模型。
进一步地，在本申请的实施例中，重建系统在构建正面网格模型，即待重建对象的正面模型之后，重建系统可以继续重建待重建对象的背部模型。
具体地，重建系统可以先恢复待重建对象的背部估计形状，然后基于背部估计形状获得对应的背部模型，最后可以通过对正面模型和背部模型的缝合处理，获得待重建对象的初始三维模型。
可以理解的是，在本申请中，为了生成与第一剪影匹配的背部估计形状，可以选择直接把虚拟相机设置在面对人的背部的位置，并渲染相应的SMPL模型。但是，这种使用透视投影并渲染获得的背部估计形状的剪影并不一定能与初始估计形状的第二剪影匹配。因此，在本申请中，重建系统可以选择使用背面剔除技术。
具体地，在本申请中，重建系统使用与呈现正面的SMPL模型(初始估计形状)的相同摄相机视点。然后，不绘制离相机最近的三角形，而是绘制最远的三角形，这样就得到后视图相应的剪影，即获得了背部估计形状对应的第三剪影。
进一步地，在本申请的实施例中，与构建正面模型的方法相似，重建系统在获得待重 建对象的背部估计形状对应的第三剪影之后，可以使用第一剪影和第三剪影，确定两者之间的第二映射关系；然后基于第二映射关系，生成对应的第二扭曲函数；最终便可以利用第二扭曲函数生成待重建对象的背部模型。
示例性的，在本申请的实施例中，图11为正面模型的示意图，图12为背部模型的示意图，图13为拼接后的示意图一，图14为拼接后的示意图二，如图11、12、13、14所示。重建系统在完成对正面模型和背部模型的构建之后，可以对两者进行缝合处理以获得最终的三维模型。具体地，重建系统可以沿正面模型和背部模型的边界创建一系列连接正面模型和背部模型两个几何形状的三角形，最终完成对待重建对象的三维重建模型的构建。
本申请提出了一种人体模型重建方法，重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息的映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
基于上述实施例，在本申请的再一实施例中，重建系统在进行纹理重建时，提出了一种能够预测图像中的不可见颜色的推理网络，即纹理生成模型，从而可以通过纹理生成模型，利用纹理信息预测获得完整纹理信息。
需要说明的是，在本申请的实施例中，为了使构建的待重建对象的三维重建模型具有纹理信息，本申请提出了将图像投影到几何体上的方法，因此，终端需要获得待重建对象的完整纹理信息，即正面纹理信息和背部纹理信息，其中，正面纹理信息，即从分割后图像中获得的纹理信息，可以用于对待重建对象的剩余的其他纹理信息的预测。
进一步地，在本申请中，对于待重建对象被遮挡的、不可见的背部纹理，本申请实施例提出了一种纹理生成模型，从而可以自动生成待重建对象的不可见纹理。其中，纹理生成模型可以为InferGAN模型。
具体地，InferGAN可以根据输入的图像，将纹理从一个输入RGB图像(Io)转移到另一个RGB图像(It)。直观的假设是，在潜在的空间中，输入输出之间的距离应该约等于图像Io和It相对应的人体语义分割Po和Pt之间的数学距离，在本申请中，可以制定的假设为：
Θ img(I o)-Θ img(I t)≈Θ seg(P o)-Θ seg(P t)        (11)
I t≈Θ R(Θ img(I o)+(Θ seg(P o)-Θ seg(P t)))      (12)
其中，Θ img、Θ seg分别为图像编码器和解析编码器。Θ R是一种基于卷积隐藏特征的图像解码器。为了节省计算资源，本申请可以将Θ img、Θ seg合并到Θ中。具体来说，InferGAN对来自I o的纹理信息进行编码，并将新的纹理像素嵌入到目标身体部分(P t)中(例如，推理阶段的背部部分)。
在进行纹理生成模型，即InferGAN的训练过程中，训练数据集可以包括多组纹理数据，每组纹理数据中包括按照不同角度采集相同对象所获得的第一纹理图像和第二纹理图像，示例性的，第一纹理图像和第二纹理图像的大小可以为512×512。在数据预处理部分， 终端可以先对第一纹理图像和第二纹理图像进行分割处理，获得到第一纹理图像分割后的P o，和第二纹理图像分割后的P t。示例性的，人体的服装可以被定义为20个部分，因此可以将P o和P t转化为有效编矢量图(尺寸为20×512×512)。
进一步地，在本申请的实施例中，还可以利用第一纹理图像中的人体的轮廓C o和第二纹理图像中人体的轮廓C t作为训练纹理生成模型时的输入。具体地，在本申请中，轮廓(大小为1×512×512)可以使得纹理生成网络的输出具有更清晰的边界。
需要说明的时，在本申请的实施例中，对输入的第一纹理图像和对应的矢量图进行随机仿射变换，以增加几何的多样性。
在InferGAN的训练过程中，为了节省计算资源，本申请实施例在通道维度上连接所有输入(I o，P o，P t，C o，C t)，拼接后的输入大小为45×512×512。由五个卷积块组成的编码器从I o中学习纹理信息，从P o，C o和P t，Ct中学习几何信息。然后将潜在特征传递给9个残差块的处理，进一步提高了编码能力。图像解码器由五个转置卷积层和一个超切线激活函数组成。在本申请中，可以采用三个损失函数训练纹理生成网络InferGAN：具体包括GAN损失，VGG特征提取器(FVGG)计算的感知损失，L1像素损失。因此，总损失L可表示为：
Figure PCTCN2020117916-appb-000010
其中，D和G分别为鉴别器和发生器。感知损失l perceptual是根据VGG19的五个不同层的输出特征计算的。
Figure PCTCN2020117916-appb-000011
是VGG19对图像I t的层特征。在本申请中，可以给不同的层特征以不同的尺度。由于GAN的损失是在特征空间中计算的，因此，GAN损失和感知损失会使网络产生更清晰的图像。
值得注意的是，在训练阶段，并不对第一纹理图像和第二纹理图像进行限制，即第一纹理图像和第二纹理图像并不需要限制为前视图图像和后视图图像，只要保证两者为包含有不完全相同的纹理的图像即可。
进一步地，在本申请的实施例中，使用纹理生成模型InferGAN可以进行待重建对象的、纹理信息(正面纹理)以外的其他纹理信息(背部纹理)的预测。具体地，在纹理信息的预测过程中，可以先对输入的目标图像I o进行分割处理，确定出语义解析P o和轮廓C o。在此基础上，可以对目标图像中的、不可见的语义解析和轮廓
Figure PCTCN2020117916-appb-000012
进行估计，从而获得剩余的其他纹理信息，然后可以通过将所有的信息传递给纹理生成模型InferGAN来合成背部图纹理
Figure PCTCN2020117916-appb-000013
本申请提出了一种人体模型重建方法，重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息的映射获得有纹 理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
基于上述实施例，在本申请的再一实施例中，为了对本申请提出的人体模型重建方法的可靠性和有效性进行验证，可以基于公开数据和采集的数据进行测试。
具体地，在本申请中，终端可以使用配置的摄像头针对不同的待重建对象进行图像采集，获得用于验证人体模型重建方法的图像。其中，该图像的分辨率可以为4032×3024的图像。具体地，为了便于后续的动画重建，本申请在采集图像时可以使用某种特定的姿势，例如，可以在待重建对象处于两脚分开、双手45度张开的姿势时进行图像采集。
可以理解的是，在本申请中，可以在云服务器上部署重建算法。例如，在一台配备英特尔酷睿i7-5820K处理器、32GB内存和Titan X GPU的PC上，重建时间约为1分钟。
图15为重建三维模型的示意图一，图16为重建三维模型的示意图二，图17为重建三维模型的示意图三，图18为重建三维模型的示意图四，如图15所示为输入的目标图像，如图16所示为人体模型的重建几何体，如图17和18所示分别为使用全纹理映射后从不同角度观察模型的结果。可见，本申请提出的人体模型重建方法，能够精确地恢复待重建对象的人体形状，使得重建后的几何轮廓与输入的目标图像非常吻合，同时，在输入的目标图像为单帧图像的前提下，还可以对不可见的纹理信息进行预测和生成。
将本申请提出的人体模型重建方法与其他三维人体形状重建方法在PeopleSnapshot数据集上进行了比较。具体地，体积式3D人体形状预测(Volumetric inference of 3d human body shapes，Bodynet)是一种基于体素的人体形状姿态和形状估计方法；基于剪影的穿衣人体重建(Silhouette-based clothed people，SICLOPE)可以依靠不同视角的合成掩码来重建人体形状细节；端对端人体形状姿态预测(End-toend Recovery of Human Shape and Pose，HMR)根据SMPL参数模型估计人体的姿势和细节。视频人体形状(Video Shapes)的方法方法可以使用同一个人的120个不同角度的图像合并成一个完整的人体形状模型，然而，由于需要优化每帧的姿态，因此计算过程比较缓慢。相比之下，本申请可以利用一帧图像进行人体模型的和纹理信息的重建，与上述前三种方法相比，本申请获得的待重建对象的三维重建模型具有更多的细节，且相较于120帧图像的输入的第四种方法，本申请的计算成本要小得多。
进一步地，在本申请的实施例中，为了使最后生成的三维重建模型具有动画效果，本申请可以将SMPL参数化模型传输到重构模型中，并应用CMU数据集(cmudataset)中的运动捕捉数据。示例性的，图19为动画重建结果的示意图，如图19所示，本申请对三维重建模型应用了慢跑动作序列，生成了慢跑的动画效果。可见，本申请获得的动画变形是合理的，并且对于一些AR/VR应用来说也是足够的。
进一步地，在本申请的实施例中，可以选择在DeepFasion数据集上训练和测试纹理生成模型InferGAN。InferGAN的学习率为0.0002，所有实验均采用Adam优化器(β 1＝0.5，β 2＝0.999)。此外，还采用了随机裁剪、仿射变换和翻转等方法来增加数据量，可以讲这个网络训练20个时期，最终获得InferGAN。
在本申请中，基于DeepFasion数据集测试了InferGAN的定性结果，重建系统可以通过任何输入的视图图像纹理来预测目标视图图像纹理。例如，InferGAN可以基于全身源输入生成逼真的底部纹理，具体地，在本申请中，利用InferGAN，在保持底部原始纹理的同时，可以合成出合理的上部纹理。
需要说明的是，在本申请中，由于纹理生成模型的目标是合成缺少的纹理，因此在这一过程中没有对合成的脸部区域的质量进行正则化。
图20为人体模型重建的系统示意图，如图20所示，本申请实施例提出的人体模型重建方法，可以利用一帧图像进行人体模型和纹理信息的重建。具体地，在本申请中，可以 利用二维非刚性配准将几何体从最初的初始估计形状变形到最终的初始三维模型。而在生成其他纹理信息的过程中，可以使用预先训练的纹理生成模型，利用正向视角下的纹理来推断不同视角下的纹理，最终便可以获得待重建对象的完整纹理信息。进一步地，在本申请中，不仅可以利用一帧图像完成了人体的完整几何结构和细节纹理的重建，还可以通过参数化传递模型的权值和关节位置获得动画模型，生成动画效果。
本申请提出了一种人体模型重建方法，重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息通过纹理映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
基于上述实施例，在本申请的另一实施例中，图21为重建系统的组成结构示意图一，如图21所示，本申请实施例提出的重建系统10可以包括获取部分11，分割部分12，重建部分13，生成部分14，修正部分15，处理部分16，
所述获取部分11，配置为获取目标图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；
所述分割部分12，配置为基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；
所述获取部分11，还配置为基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；
所述重建部分13，配置为通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；
所述获取部分11，还配置为根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；
所述生成部分14，配置为基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
进一步地，在本申请的实施例中，所述获取部分11，具体配置为利用SMPL模型对所述分割后图像进行拟合处理，获得所述初始估计形状；提取所述分割后图像中的像素信息，获得所述纹理信息。
进一步地，在本申请的实施例中，所述重建部分13，具体配置为获取所述待重建对象的第一剪影；根据所述第一剪影和所述初始估计形状进行校正处理，生成所述待重建对象的正面模型和背部模型；对所述正面模型和所述背部模型进行拼接处理，生成所述初始三维模型。
进一步地，在本申请的实施例中，所述重建部分13，还具体配置为确定所述初始估计形状的第二剪影；确定所述第一剪影和所述第二剪影之间的第一映射关系；基于所述第一映射关系，生成第一扭曲函数；利用所述第一扭曲函数生成所述正面模型。
进一步地，在本申请的实施例中，所述修正部分15，配置为所述利用所述第一扭曲函 数生成所述正面模型之后，对所述正面模型中的面部模型进行修正处理。
进一步地，在本申请的实施例中，所述重建部分13，还具体配置为对所述初始估计形状进行渲染处理，获得所述待重建对象的背部估计形状；确定所述背部估计形状的第三剪影；确定所述第一剪影和所述第三剪影之间的第二映射关系；基于所述第二映射关系，生成第二扭曲函数；利用所述第二扭曲函数生成所述背部模型。
进一步地，在本申请的实施例中，所述获取部分11，还配置为根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息之前，获取训练数据集；其中，所述训练数据集包括多组纹理数据，每组纹理数据中包括按照不同角度采集相同对象所获得的第一纹理图像和第二纹理图像；
所述生成部分14，还配置为使用所述训练数据集进行模型训练，生成所述纹理生成模型。
进一步地，在本申请的实施例中，所述生成部分14，具体配置为确定所述第一纹理图像对应的第一轮廓，确定所述第二纹理对应的第二轮廓；利用所述第一纹理图像、所述第一轮廓、所述第二纹理图像以及所述第二轮廓训练获得所述纹理生成模型。
进一步地，在本申请的实施例中，所述获取部分11，还具体配置为将所述纹理信息输入至所述纹理生成模型，输出预测纹理信息；其中，所述预测纹理信息为所述待重建对象对应的、所述纹理信息以外的其他纹理信息；基于所述纹理信息和所述预测纹理信息，获得所述完整纹理信息。
进一步地，在本申请的实施例中，所述处理部分16，配置为基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型之后，利用动画重建模型对所述三维重建模型进行动画处理。
进一步地，在本申请的实施例中，所述目标图像为一帧RGB图像。
图22为重建系统的组成结构示意图二，如图22所示，本申请实施例提出的重建系统10还可以包括处理器17、存储有处理器17可执行指令的存储器18，进一步地，重建系统10还可以包括存通信接口19，和用于连接处理器17、存储器18以及存通信接口19的总线110。
在本申请的实施例中，上述处理器17可以为特定用途集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Processor，DSP)、数字信号处理装置(Digital Signal Processing Device，DSPD)、可编程逻辑装置(ProgRAMmable Logic Device，PLD)、现场可编程门阵列(Field ProgRAMmable Gate Array，FPGA)、中央处理器(Central Processing Unit，CPU)、控制器、微控制器、微处理器中的至少一种。可以理解地，对于不同的设备，用于实现上述处理器功能的电子器件还可以为其它，本申请实施例不作具体限定。重建系统10还可以包括存储器18，该存储器18可以与处理器17连接，其中，存储器18用于存储可执行程序代码，该程序代码包括计算机操作指令，存储器18可能包含高速RAM存储器，也可能还包括非易失性存储器，例如，至少两个磁盘存储器。
在本申请的实施例中，总线110用于连接存通信接口19、处理器17以及存储器18以及这些器件之间的相互通信。
在本申请的实施例中，存储器18，用于存储指令和数据。
进一步地，在本申请的实施例中，上述处理器17，用于获取目标图像，并基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
在实际应用中，上述存储器18可以是易失性存储器(volatile memor)，例如随机存取存储器(Random-Access Memory，RAM)；或者非易失性存储器(non-volatile memory)，例如只读第一存储y器(Read-Only Memory，ROM)，快闪存储器(flash memory)，硬盘(Hard Disk Drive，HDD)或固态硬盘(Solid-State Drive，SSD)；或者上述种类的存储器的组合，并向处理器17提供指令和数据。
另外，在本实施例中的各功能模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
集成的单元如果以软件功能模块的形式实现并非作为独立的产品进行销售或使用时，可以存储在一个计算机可读取存储介质中，基于这样的理解，本实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或processor(处理器)执行本实施例方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
本申请实施例提出的一种重建系统，该重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息通过纹理映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。
本申请实施例提供一种计算机可读存储介质，其上存储有程序，该程序被处理器执行时实现如上所述的人体模型重建方法。
具体来讲，本实施例中的一种人体模型重建方法对应的程序指令可以被存储在光盘，硬盘，U盘等存储介质上，当存储介质中的与一种人体模型重建方法对应的程序指令被一电子设备读取或被执行时，包括如下步骤：
获取目标图像，并基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；
基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；
通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；
根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；
基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
本领域内的技术人员应明白，本申请的实施例可提供为方法、终端、或计算机程序产品。因此，本申请可采用硬件实施例、软件实施例、或结合软件和硬件方面的实施例的形 式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器和光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的实现流程示意图和/或方框图来描述的。应理解可由计算机程序指令实现流程示意图和/或方框图中的每一流程和/或方框、以及实现流程示意图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在实现流程示意图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在实现流程示意图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在实现流程示意图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
以上所述，仅为本申请的较佳实施例而已，并非用于限定本申请的保护范围。
工业实用性
本申请实施例提出了一种人体模型重建方法、重建系统及存储介质，重建系统获取目标图像，并基于目标图像中的待重建对象进行分割处理，获得分割后图像；其中，目标图像为待重建对象对应的一帧正面图像；基于分割后图像，分别获得待重建对象的初始估计形状和纹理信息；通过初始估计形状，确定待重建对象的初始三维模型；其中，初始三维模型为无纹理的三维模型；根据纹理信息和纹理生成模型，获得待重建对象的完整纹理信息；基于初始三维模型和完整纹理信息，生成待重建对象的三维重建模型；其中，三维重建模型为有纹理的三维模型。由此可见，在本申请的实施例中，重建系统在获得一帧目标图像之后，可以通过分割处理获得待重建对象对应的分割后图像，基于该分割后图像，一方面，重建系统可以在获取初始估计形状的基础上，生成待重建对象对应的、不包括人体纹理的初始三维模型；另一方面，重建系统可以通过从分割后图像中提取的部分纹理信息，利用纹理生成模型，还原待重建对象的完整纹理信息，最后便可以通过初始三维模型和完整纹理信息的映射获得有纹理的三维重建模型。可见，本申请提出的人体模型重建方法，能够使用一帧图像，准确的对人体的三维模型进行重建，同时可以还原出人体的完整纹理信息，从而提高了模型重建的准确性，大大优化了模型重建效果。

Claims (14)

  1. 一种人体模型重建方法，所述方法包括：
    获取目标图像，并基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；
    基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；
    通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；
    根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；
    基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
  2. 根据权利要求1所述的方法，其中，所述基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息，包括：
    利用SMPL模型对所述分割后图像进行拟合处理，获得所述初始估计形状；
    提取所述分割后图像中的像素信息，获得所述纹理信息。
  3. 根据权利要求1所述的方法，其中，所述通过所述初始估计形状，确定所述待重建对象的初始三维模型，包括：
    获取所述待重建对象的第一剪影；
    根据所述第一剪影和所述初始估计形状进行校正处理，生成所述待重建对象的正面模型和背部模型；
    对所述正面模型和所述背部模型进行拼接处理，生成所述初始三维模型。
  4. 根据权利要求3所述的方法，其中，所述根据所述第一剪影和所述初始估计形状进行校正处理，生成所述待重建对象的正面模型和背部模型，包括：
    确定所述初始估计形状的第二剪影；
    确定所述第一剪影和所述第二剪影之间的第一映射关系；
    基于所述第一映射关系，生成第一扭曲函数；
    利用所述第一扭曲函数生成所述正面模型。
  5. 根据权利要求4所述的方法，其中，所述利用所述第一扭曲函数生成所述正面模型之后，所述方法还包括：
    对所述正面模型中的面部模型进行修正处理。
  6. 根据权利要求3所述的方法，其中，所述根据所述第一剪影和所述初始估计形状进行校正处理，生成所述待重建对象的正面模型和背部模型，包括：
    对所述初始估计形状进行渲染处理，获得所述待重建对象的背部估计形状；
    确定所述背部估计形状的第三剪影；
    确定所述第一剪影和所述第三剪影之间的第二映射关系；
    基于所述第二映射关系，生成第二扭曲函数；
    利用所述第二扭曲函数生成所述背部模型。
  7. 根据权利要求1所述的方法，其中，所述根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息之前，所述方法还包括：
    获取训练数据集；其中，所述训练数据集包括多组纹理数据，每组纹理数据中包括按照不同角度采集相同对象所获得的第一纹理图像和第二纹理图像；
    使用所述训练数据集进行模型训练，生成所述纹理生成模型。
  8. 根据权利要求7所述的方法，其中，所述使用所述训练数据集进行模型训练，生成所述纹理生成模型，包括：
    确定所述第一纹理图像对应的第一轮廓，确定所述第二纹理对应的第二轮廓；
    利用所述第一纹理图像、所述第一轮廓、所述第二纹理图像以及所述第二轮廓训练获得所述纹理生成模型。
  9. 根据权利要求7所述的方法，其中，所述根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息，包括：
    将所述纹理信息输入至所述纹理生成模型，输出预测纹理信息；其中，所述预测纹理信息为所述待重建对象对应的、所述纹理信息以外的其他纹理信息；
    基于所述纹理信息和所述预测纹理信息，获得所述完整纹理信息。
  10. 根据权利要求1所述的方法，其中，所述基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型之后，所述方法还包括：
    利用动画重建模型对所述三维重建模型进行动画处理。
  11. 根据权利要求1所述的方法，其中，所述目标图像为一帧RGB图像。
  12. 一种重建系统，所述重建系统包括：获取部分，分割部分，重建部分，生成部分，
    所述获取部分，配置为获取目标图像；其中，所述目标图像为所述待重建对象对应的一帧正面图像；
    所述分割部分，配置为基于所述目标图像中的待重建对象进行分割处理，获得分割后图像；
    所述获取部分，还配置为基于所述分割后图像，分别获得所述待重建对象的初始估计形状和纹理信息；
    所述重建部分，配置为通过所述初始估计形状，确定所述待重建对象的初始三维模型；其中，所述初始三维模型为无纹理的三维模型；
    所述获取部分，还配置为根据所述纹理信息和纹理生成模型，获得所述待重建对象的完整纹理信息；
    所述生成部分，配置为基于所述初始三维模型和所述完整纹理信息，生成所述待重建对象的三维重建模型；其中，所述三维重建模型为有纹理的三维模型。
  13. 一种重建系统，所述重建系统包括处理器、存储有所述处理器可执行指令的存储器，当所述指令被所述处理器执行时，实现如权利要求1-11任一项所述的方法。
  14. 一种计算机可读存储介质，其上存储有程序，应用于重建系统中，所述程序被处理器执行时，实现如权利要求1-11任一项所述的方法。
PCT/CN2020/117916 2019-09-30 2020-09-25 人体模型重建方法、重建系统及存储介质 WO2021063271A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202080064554.1A CN114450719A (zh) 2019-09-30 2020-09-25 人体模型重建方法、重建系统及存储介质
EP20873287.5A EP4036863A4 (en) 2019-09-30 2020-09-25 METHOD AND SYSTEM FOR RECONSTRUCTION OF THE HUMAN BODY MODEL AND STORAGE MEDIUM
US17/707,681 US11928778B2 (en) 2019-09-30 2022-03-29 Method for human body model reconstruction and reconstruction system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201962908314P 2019-09-30 2019-09-30
US62/908,314 2019-09-30

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/707,681 Continuation US11928778B2 (en) 2019-09-30 2022-03-29 Method for human body model reconstruction and reconstruction system

Publications (1)

Publication Number Publication Date
WO2021063271A1 true WO2021063271A1 (zh) 2021-04-08

Family

ID=75337652

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/117916 WO2021063271A1 (zh) 2019-09-30 2020-09-25 人体模型重建方法、重建系统及存储介质

Country Status (4)

Country Link
US (1) US11928778B2 (zh)
EP (1) EP4036863A4 (zh)
CN (1) CN114450719A (zh)
WO (1) WO2021063271A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113051420A (zh) * 2021-04-15 2021-06-29 山东大学 一种基于文本生成视频机器人视觉人机交互方法及系统
CN113570707A (zh) * 2021-07-30 2021-10-29 集美大学 三维人体重建方法、装置、计算机设备及存储介质
CN114373033A (zh) * 2022-01-10 2022-04-19 腾讯科技（深圳）有限公司 图像处理方法、装置、设备、存储介质及计算机程序

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114998514A (zh) * 2022-05-16 2022-09-02 聚好看科技股份有限公司 一种虚拟角色的生成方法及设备
CN115147508B (zh) * 2022-06-30 2023-09-22 北京百度网讯科技有限公司 服饰生成模型的训练、生成服饰图像的方法和装置
CN115239865A (zh) * 2022-08-03 2022-10-25 中国电信股份有限公司 模型构建方法、模型构建装置、介质及电子设备
CN116152051B (zh) * 2023-02-27 2023-12-01 上海福柯斯智能科技有限公司 一种x射线图像的分块配准图像剪影方法和装置
CN116957919B (zh) * 2023-07-12 2024-07-16 珠海凌烟阁芯片科技有限公司 一种基于rgbd图像的3d人体模型生成方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101082765A (zh) * 2006-06-01 2007-12-05 高宏 一种三维人像摄影系统及其实现方法
CN101404091A (zh) * 2008-11-07 2009-04-08 重庆邮电大学 基于两步形状建模的三维人脸重建方法和系统
US20100097375A1 (en) * 2008-10-17 2010-04-22 Kabushiki Kaisha Square Enix (Also Trading As Square Enix Co., Ltd.) Three-dimensional design support apparatus and three-dimensional model display system
US20130321413A1 (en) * 2012-05-31 2013-12-05 Microsoft Corporation Video generation using convict hulls
CN109191554A (zh) * 2018-09-04 2019-01-11 清华-伯克利深圳学院筹备办公室 一种超分辨图像重建方法、装置、终端和存储介质
CN110189408A (zh) * 2019-06-04 2019-08-30 西安科技大学 一种根据人体照片得到人体外表数据的系统及方法

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011045768A2 (en) * 2009-10-15 2011-04-21 Yeda Research And Development Co. Ltd. Animation of photo-images via fitting of combined models
US9196089B2 (en) * 2012-05-17 2015-11-24 Disney Enterprises, Inc. Techniques for processing reconstructed three-dimensional image data
EP2852932A1 (en) * 2012-05-22 2015-04-01 Telefónica, S.A. A method and a system for generating a realistic 3d reconstruction model for an object or being
JP6608165B2 (ja) * 2015-05-12 2019-11-20 国立大学法人京都大学 画像処理装置及び方法、並びにコンピュータプログラム
US11508107B2 (en) * 2018-02-26 2022-11-22 Didimo, Inc. Additional developments to the automatic rig creation process
US10321728B1 (en) * 2018-04-20 2019-06-18 Bodygram, Inc. Systems and methods for full body measurements extraction
US11250639B2 (en) * 2018-12-19 2022-02-15 Seddi, Inc. Learning-based animation of clothing for virtual try-on
CN109636831B (zh) * 2018-12-19 2023-08-04 安徽大学 一种估计三维人体姿态及手部信息的方法
WO2020141800A1 (ko) * 2019-01-03 2020-07-09 주식회사 미즈 이미지를 이용하여 신체 사이즈를 측정하고, 의류 사이즈를 추천하는 방법, 서버 및 프로그램
CN109840939B (zh) 2019-01-08 2024-01-26 北京达佳互联信息技术有限公司 三维重建方法、装置、电子设备及存储介质
US10839586B1 (en) * 2019-06-07 2020-11-17 Snap Inc. Single image-based real-time body animation
CN110223370B (zh) 2019-05-29 2023-03-07 南京大学 一种从单视点图片生成完整人体纹理贴图的方法
EP3983996A1 (en) * 2019-06-17 2022-04-20 Ariel Ai, Inc. Scene reconstruction in three-dimensions from two-dimensional images
CN110298916B (zh) 2019-06-21 2022-07-01 湖南大学 一种基于合成深度数据的三维人体重建方法
KR20210019185A (ko) * 2019-08-12 2021-02-22 엘지전자 주식회사 멀티미디어 디바이스 및 그 제어 방법
US10813715B1 (en) * 2019-10-16 2020-10-27 Nettelo Incorporated Single image mobile device human body scanning and 3D model creation and analysis
CN110827342B (zh) * 2019-10-21 2023-06-02 中国科学院自动化研究所 三维人体模型重建方法及存储设备、控制设备
CN111784821B (zh) * 2020-06-30 2023-03-14 北京市商汤科技开发有限公司 三维模型生成方法、装置、计算机设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101082765A (zh) * 2006-06-01 2007-12-05 高宏 一种三维人像摄影系统及其实现方法
US20100097375A1 (en) * 2008-10-17 2010-04-22 Kabushiki Kaisha Square Enix (Also Trading As Square Enix Co., Ltd.) Three-dimensional design support apparatus and three-dimensional model display system
CN101404091A (zh) * 2008-11-07 2009-04-08 重庆邮电大学 基于两步形状建模的三维人脸重建方法和系统
US20130321413A1 (en) * 2012-05-31 2013-12-05 Microsoft Corporation Video generation using convict hulls
CN109191554A (zh) * 2018-09-04 2019-01-11 清华-伯克利深圳学院筹备办公室 一种超分辨图像重建方法、装置、终端和存储介质
CN110189408A (zh) * 2019-06-04 2019-08-30 西安科技大学 一种根据人体照片得到人体外表数据的系统及方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ALLDIECK THIEMO; MAGNOR MARCUS; BHATNAGAR BHARAT LAL; THEOBALT CHRISTIAN; PONS-MOLL GERARD: "Learning to Reconstruct People in Clothing From a Single RGB Camera", 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 15 June 2019 (2019-06-15), pages 1175 - 1186, XP033687140, DOI: 10.1109/CVPR.2019.00127 *
See also references of EP4036863A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113051420A (zh) * 2021-04-15 2021-06-29 山东大学 一种基于文本生成视频机器人视觉人机交互方法及系统
CN113570707A (zh) * 2021-07-30 2021-10-29 集美大学 三维人体重建方法、装置、计算机设备及存储介质
CN114373033A (zh) * 2022-01-10 2022-04-19 腾讯科技（深圳）有限公司 图像处理方法、装置、设备、存储介质及计算机程序

Also Published As

Publication number Publication date
US11928778B2 (en) 2024-03-12
CN114450719A (zh) 2022-05-06
EP4036863A1 (en) 2022-08-03
EP4036863A4 (en) 2023-02-08
US20220222895A1 (en) 2022-07-14

Similar Documents

Publication Publication Date Title
WO2021063271A1 (zh) 人体模型重建方法、重建系统及存储介质
US10679046B1 (en) Machine learning systems and methods of estimating body shape from images
CN109636831B (zh) 一种估计三维人体姿态及手部信息的方法
Yu et al. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor
Ichim et al. Dynamic 3D avatar creation from hand-held video input
Vlasic et al. Articulated mesh animation from multi-view silhouettes
CN106023288B (zh) 一种基于图像的动态替身构造方法
WO2022205760A1 (zh) 三维人体重建方法、装置、设备及存储介质
CN107240129A (zh) 基于rgb‑d相机数据的物体及室内小场景恢复与建模方法
CN113421328B (zh) 一种三维人体虚拟化重建方法及装置
Tu et al. Consistent 3d hand reconstruction in video via self-supervised learning
Li et al. Animated 3D human avatars from a single image with GAN-based texture inference
US20230126829A1 (en) Point-based modeling of human clothing
Li et al. 3d human avatar digitization from a single image
Caliskan et al. Multi-view consistency loss for improved single-image 3d reconstruction of clothed people
Huang et al. Object-occluded human shape and pose estimation with probabilistic latent consistency
Li et al. Image-guided human reconstruction via multi-scale graph transformation networks
CN114998514A (zh) 一种虚拟角色的生成方法及设备
Li et al. Robust 3D human motion reconstruction via dynamic template construction
Gan et al. Fine-grained multi-view hand reconstruction using inverse rendering
Gomes et al. Do as I do: Transferring human motion and appearance between monocular videos with spatial and temporal constraints
Gong Application and Practice of Artificial Intelligence Technology in Interior Design
Yang et al. Human bas-relief generation from a single photograph
CN115049764A (zh) Smpl参数预测模型的训练方法、装置、设备及介质
CN116071485A (zh) 一种人体三维重建方法及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20873287

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020873287

Country of ref document: EP

Effective date: 20220429