TW202244853A - 3d reconstruction method, apparatus and system, storage medium and computer device - Google Patents

Abstract

The present disclosure provides a 3D reconstruction method, an apparatus and a system, a medium and a computer device. The method includes: performing 3D reconstruction of a target object in an image by a 3D reconstruction network to obtain initial values of parameters of the target object, wherein the initial values of the parameters are used to construct a 3D model of the target object; optimizing the initial values of the parameters based on pre-obtained supervision information for representing features of the target object to obtain optimized values of the parameters; and performing skinned mesh based on the optimized values of the parameters to construct the 3D model of the target object.

Description

Three-dimensional reconstruction method, device and system, media and computer equipment

The present disclosure relates to the technical field of computer vision, in particular to a three-dimensional reconstruction method, device and system, media and computer equipment.

3D reconstruction is one of the important technologies in computer vision, and has many potential applications in augmented reality, virtual reality and other fields. By performing three-dimensional reconstruction on the target object, the posture and limb rotation of the target object can be reconstructed. However, traditional 3D reconstruction methods cannot balance the accuracy and reliability of reconstruction results.

The present disclosure provides a three-dimensional reconstruction method, device and system, media and computer equipment.

According to the first aspect of the embodiments of the present disclosure, there is provided a 3D reconstruction method, the method comprising: performing 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain an initial value of a parameter of the target object, wherein, The initial value of the parameter is used to establish the three-dimensional model of the target object; the initial value of the parameter is optimized based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain the optimized value of the parameter; based on The optimized values of the parameters are subjected to bone skinning processing to establish a three-dimensional model of the target object.

In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: the target object's initial Two-dimensional key points, semantic information of multiple pixel points on the target object in the image; the second supervisory information includes an initial three-dimensional point cloud of the surface of the target object. In the embodiment of the present disclosure, only the initial two-dimensional key points of the target object or the semantic information of the pixels can be used as the supervisory information to optimize the initial value of the parameter, so that the optimization efficiency is high and the optimization complexity is low; or, the The initial 3D point cloud of the surface of the target object and the semantic information of the aforementioned initial 2D key points or pixels are used together as supervisory information, thereby improving the accuracy of the optimized value of the obtained parameters.

In some embodiments, the method further includes: extracting initial two-dimensional keypoint information of the target object from the image through a keypoint extraction network. Using the information of the initial 2D key points extracted by the key point extraction network as supervision information can generate more natural and reasonable actions for the 3D model.

In some embodiments, the image includes a depth image of the target object; the method further includes: extracting depth information of multiple pixels on the target object from the depth image; based on the The depth information back-projects multiple pixel points on the target object in the depth image to a three-dimensional space to obtain an initial three-dimensional point cloud on the surface of the target object. By extracting the depth information and back-projecting the pixels on the 2D image to the 3D space based on the depth information, the initial 3D point cloud of the surface of the target object can be obtained, so that the initial 3D point cloud can be used as the supervisory information to optimize the parameters. The initial value further improves the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; the extracting the depth information of multiple pixels on the target object from the depth image includes: Carry out image segmentation on the image, determine the image area where the target object is located in the RGB image based on the results of the image segmentation, and determine the target in the depth image based on the image area where the target object is located in the RGB image An image area where the object is located; obtaining depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.

In some embodiments, the method further includes: filtering outliers from the initial 3D point cloud, and using the filtered initial 3D point cloud as the second supervisory information. By filtering the outliers, the interference of the outliers is reduced, and the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is collected by an image acquisition device, and the parameters include: global rotation parameters of the target object, key point rotation parameters of each key point of the target object, the The body parameters of the target object and the displacement parameters of the image acquisition device; the optimization of the initial value of the parameters based on the pre-acquired supervision information representing the characteristics of the target object includes: value and the initial value of the key point rotation parameter remain unchanged, based on the supervisory information and the initial value of the displacement parameter, the current value of the displacement parameter of the image acquisition device and the global rotation parameter Optimizing the initial value to obtain the optimal value of the displacement parameter and the optimal value of the global rotation parameter; Optimize the initial value of the key point rotation parameter and the optimal value of the body shape parameter. During the optimization process, changing the position of the image acquisition device and changing the position of the 3D key points can lead to changes in the 2D projection of the 3D key points, which will lead to an unstable optimization process. By using a two-stage optimization method, firstly fix the initial value of the key point rotation parameter and the initial value of the body shape parameter to optimize the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter, and then fix the displacement parameter The initial value and the initial value of the global rotation parameter optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter, which improves the stability of the optimization process.

In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the current displacement parameter of the image acquisition device based on the supervision information and the initial value of the displacement parameter value and the initial value of the global rotation parameter, including: obtaining the target two-dimensional projection key point corresponding to the three-dimensional key point of the target object in the two-dimensional projection key point belonging to the preset part of the target object; wherein , the 3D key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, and the 2D projection key point is based on the current value of the displacement parameter and The initial value of the global rotation parameter is obtained by projecting the three-dimensional key point of the target object; obtaining the first loss between the target two-dimensional projection key point and the initial two-dimensional key point; obtaining the initial value of the displacement parameter A second loss between the value and the current value of the displacement parameter; optimizing the current value of the displacement parameter and the initial value of the global rotation parameter based on the first loss and the second loss. The preset part can be the torso and other parts. Since different actions have little influence on the key points of the torso, the first loss can be determined by using the key points of the torso, which can reduce the influence of different actions on the position of the key points and improve Optimize the accuracy of the results. Since the two-dimensional key points are supervisory information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, it is possible to reduce the optimization result falling into the local optimal point on the two-dimensional plane, thereby Situations that deviate from the true point.

In some embodiments, the supervisory information includes initial two-dimensional key points of the target object; the optimal value based on the displacement parameter and the global rotation parameter is based on the initial value of the key point rotation parameter Optimizing with the initial value of the posture parameter includes: obtaining the third loss between the optimized two-dimensional projection key point of the target object and the initial two-dimensional key point, and the optimized two-dimensional projection key point is based on the The optimized value of the displacement parameter and the optimized value of the global rotation parameter are obtained by projecting the optimized three-dimensional key point of the target object, and the optimized three-dimensional key point is based on the optimized value of the global rotation parameter and the initial value of the key point rotation parameter And the initial value of the posture parameter is obtained; the fourth loss is obtained, and the fourth loss is used to characterize the rationality of the posture corresponding to the optimal value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter; Optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the third loss and the fourth loss. This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, which improves the stability of the optimization process. At the same time, the fourth loss ensures the optimization The latter parameters correspond to the rationality of the pose.

In some embodiments, the method further includes: after optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter , performing joint optimization on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. In this embodiment, on the basis of the aforementioned optimization, the optimized parameters are jointly optimized, thereby further improving the accuracy of the optimization result.

In some embodiments, the supervisory information includes initial 2D key points of the target object and an initial 3D point cloud of the surface of the target object; based on the supervisory information and the initial value of the displacement parameter, the Optimizing the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter includes: obtaining the predicted value of the target object in the two-dimensional projection key point corresponding to the three-dimensional key point of the target object. Set the target two-dimensional projection key point of the part; wherein, the three-dimensional key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, and the two-dimensional projection The key points are obtained by projecting the three-dimensional key points of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the distance between the target two-dimensional projected key point and the initial two-dimensional key point The first loss; obtaining the second loss between the initial value of the displacement parameter and the current value of the displacement parameter; obtaining the first loss between the first three-dimensional point cloud of the surface of the target object and the initial three-dimensional point cloud Five losses; the first three-dimensional point cloud is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter; based on the first loss, the second loss and the fifth loss pair The current value of the displacement parameter and the initial value of the global rotation parameter are optimized. In this embodiment, the three-dimensional pointfish is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

In some embodiments, the joint optimization of the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter includes: obtaining the A sixth loss between the optimized two-dimensional projection keypoint of the target object and the initial two-dimensional keypoint, the optimized two-dimensional projection keypoint is based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter on the The optimized three-dimensional key points of the target object are obtained by projection, and the optimized three-dimensional key points are obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the body shape parameter; the seventh loss is obtained, and the first Seven losses are used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the posture parameter; obtain the second 3D point cloud of the surface of the target object and the initial The eighth loss among three-dimensional point clouds; the second three-dimensional point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the body shape parameter; based on the sixth loss, the first The seventh loss and the eighth loss jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. In this embodiment, the three-dimensional pointfish is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

According to a second aspect of an embodiment of the present disclosure, there is provided a 3D reconstruction device, the device comprising: a first 3D reconstruction group, configured to perform 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain the target The initial value of the parameter of the object, wherein the initial value of the parameter is used to establish the three-dimensional model of the target object; the optimization module is used to optimize the parameter based on the pre-acquired supervision information used to represent the characteristics of the target object Optimize the initial value of the parameter to obtain the optimized value of the parameter; the second three-dimensional reconstruction group is used to perform bone skinning processing based on the optimized value of the parameter, and establish a three-dimensional model of the target object.

In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: the target object's initial Two-dimensional key points, semantic information of multiple pixel points on the target object in the image; the second supervisory information includes an initial three-dimensional point cloud of the surface of the target object. In the embodiment of the present disclosure, only the initial two-dimensional key points of the target object or the semantic information of the pixels can be used as the supervisory information to optimize the initial value of the parameter, so that the optimization efficiency is high and the optimization complexity is low; or, the The initial 3D point cloud of the surface of the target object and the semantic information of the aforementioned initial 2D key points or pixels are used together as supervisory information, thereby improving the accuracy of the optimized value of the obtained parameters.

In some embodiments, the device further includes: a 2D key point extraction module, configured to extract initial 2D key point information of the target object from the image through a key point extraction network. Using the information of the initial 2D key points extracted by the key point extraction network as supervision information can generate more natural and reasonable actions for the 3D model.

In some embodiments, the image includes a depth image of the target object; the device further includes: a depth information extraction module, configured to extract a plurality of pixels on the target object from the depth image Depth information of points; a backprojection module, used to backproject multiple pixel points on the target object in the depth image to three-dimensional space based on the depth information, to obtain the initial surface of the target object 3D point cloud. By extracting the depth information and back-projecting the pixels on the 2D image to the 3D space based on the depth information, the initial 3D point cloud of the surface of the target object can be obtained, so that the initial 3D point cloud can be used as the supervisory information to optimize the parameters. The initial value further improves the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; the depth information extraction module includes: an image segmentation unit for performing image segmentation on the RGB image, the image An area determining unit, configured to determine the image area where the target object is located in the RGB image based on the image segmentation result, and determine the target object in the depth image based on the image area where the target object is located in the RGB image The image area where the target object is located; a depth information acquisition unit, configured to acquire depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.

In some embodiments, the device further includes: a filtering module, configured to filter out outliers from the initial 3D point cloud, and use the filtered initial 3D point cloud as the second supervisory information. By filtering the outliers, the interference of the outliers is reduced, and the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is collected by an image acquisition device, and the parameters include: global rotation parameters of the target object, key point rotation parameters of each key point of the target object, the The posture parameters of the target object and the displacement parameters of the image acquisition device; the optimization module includes: a first optimization unit, which is used to keep the initial value of the posture parameter and the initial value of the key point rotation parameter unchanged In this case, based on the supervision information and the initial value of the displacement parameter, the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter are optimized to obtain the optimized value of the displacement parameter and the global rotation parameter. The optimized value of the rotation parameter; the second optimization unit is used to optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, The optimal values of the key point rotation parameters and the optimal values of the body parameters are obtained. During the optimization process, changing the position of the image acquisition device and changing the position of the 3D key points can lead to changes in the 2D projection of the 3D key points, which will lead to an unstable optimization process. By using a two-stage optimization method, firstly fix the initial value of the key point rotation parameter and the initial value of the body shape parameter to optimize the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter, and then fix the displacement parameter The initial value and the initial value of the global rotation parameter optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter, which improves the stability of the optimization process.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the first optimization unit is configured to: obtain the two-dimensional projection key points corresponding to the three-dimensional key points of the target object belonging to the The target two-dimensional projection key point of the preset part of the target object; wherein, the three-dimensional key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, The two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the target two-dimensional projection key point and the initial two-dimensional a first loss between key points; obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; based on the first loss and the second loss on the current value of the displacement parameter and the initial value of the global rotation parameter are optimized. The preset part can be the torso and other parts. Since different actions have little influence on the key points of the torso, the first loss can be determined by using the key points of the torso, which can reduce the influence of different actions on the position of the key points and improve Optimize the accuracy of the results. Since the two-dimensional key points are supervisory information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, it is possible to reduce the optimization result falling into the local optimal point on the two-dimensional plane, thereby Situations that deviate from the true point.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the second optimization unit is configured to: obtain the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points The third loss between points, the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized three-dimensional key point The point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the body shape parameter; obtain the fourth loss, and the fourth loss is used to characterize the optimized value of the global rotation parameter, the key point The rationality of the attitude corresponding to the initial value of the rotation parameter and the initial value of the posture parameter; based on the third loss and the fourth loss, the initial value of the key point rotation parameter and the initial value of the posture parameter are optimized . This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, which improves the stability of the optimization process. At the same time, the fourth loss ensures the optimization The latter parameters correspond to the rationality of the pose.

In some embodiments, the device further includes: a joint optimization module, which is used to optimize the initial value of the key point rotation parameter and the body posture based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter. After the initial value of the parameter is optimized, the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized. In this embodiment, on the basis of the aforementioned optimization, the optimized parameters are jointly optimized, thereby further improving the accuracy of the optimization result.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; the first optimization unit is configured to: acquire the three-dimensional key points of the target object Among the two-dimensional projection key points corresponding to the point, the target two-dimensional projection key points belonging to the preset part of the target object; wherein, the three-dimensional key points of the target object are based on the initial value of the global rotation parameter, the key point rotation parameter The initial value of the initial value and the initial value of the posture parameter are obtained, and the key points of the two-dimensional projection are obtained by projecting the three-dimensional key points of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the target The first loss between the two-dimensional projection key point and the initial two-dimensional key point; obtain the second loss between the initial value of the displacement parameter and the current value of the displacement parameter; obtain the target object surface The fifth loss between the first 3D point cloud and the initial 3D point cloud; the first 3D point cloud is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter ; optimizing the current value of the displacement parameter and the initial value of the global rotation parameter based on the first loss, the second loss and the fifth loss. In this embodiment, the three-dimensional pointfish is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

In some embodiments, the joint optimization module includes: a first acquisition unit, configured to acquire a sixth loss between the optimized 2D projection keypoint of the target object and the initial 2D keypoint, the The optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized three-dimensional key point is based on the optimized value of the global rotation parameter , the optimized value of the key point rotation parameter and the optimized value of the posture parameter are obtained; the second acquisition unit is used to obtain the seventh loss, and the seventh loss is used to characterize the optimized value of the global rotation parameter and the key point rotation parameter The optimal value and the rationality of the posture corresponding to the optimal value of the body parameters; the third acquisition unit is used to acquire the eighth loss between the second 3D point cloud on the surface of the target object and the initial 3D point cloud; the The second three-dimensional point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the body shape parameter; a joint optimization unit is used for based on the sixth loss, the seventh loss and the eighth loss Jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. In this embodiment, the three-dimensional pointfish is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

According to a third aspect of the embodiments of the present disclosure, there is provided a three-dimensional reconstruction system, the system comprising: an image acquisition device for acquiring an image of a target object; and a processing unit connected in communication with the image acquisition device for performing three-dimensional reconstruction on the target object in the image through a three-dimensional reconstruction network to obtain an initial value of a parameter of the target object, and the initial value of the parameter is used to establish a three-dimensional model of the target object; based on The pre-acquired supervisory information used to represent the characteristics of the target object optimizes the initial value of the parameter to obtain the optimized value of the parameter; performs bone skinning processing based on the optimized value of the parameter to establish a three-dimensional image of the target object Model.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any embodiment is implemented.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, any A method described in one embodiment.

According to a sixth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product is stored in a storage medium and includes a computer program that can run on a processor, and when the processor executes the computer program, any The method described in the examples.

In the embodiment of the present disclosure, the initial value of the parameter is obtained by performing three-dimensional reconstruction on the image of the target object through the three-dimensional reconstruction network, and then optimizes the initial value of the parameter based on the supervision information, and the optimized value of the parameter obtained based on the parameter optimization to create a 3D model of the target object. The advantage of the parameter optimization method is that it can give more accurate 3D reconstruction results that conform to the 2D observation characteristics of the image, but it often gives unnatural and unreasonable action results with low reliability. Network regression through the 3D reconstruction network can give more natural and reasonable action results. Therefore, optimizing the output results of the 3D reconstruction network as the initial value of the parameters can ensure the reliability of the 3D reconstruction results. On the other hand, taking into account the accuracy of 3D reconstruction.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

This disclosure claims the priority of the Chinese patent application with application number 202110506464X and titled "3D reconstruction method, device and system, media and computer equipment" filed on May 10, 2021, which is incorporated by reference into this article.

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as detailed in the appended claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only, and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a", "the", and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality.

It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various pieces of information, these pieces of information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present disclosure, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure, and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are described below in conjunction with the accompanying drawings The program is described in further detail.

The 3D reconstruction of the target object needs to reconstruct the body posture and limb rotation of the target object. Usually, a parametric model is used to express the body posture and limb rotation of the target object, not just the 3D key points. For example, when performing 3D reconstruction on different people, the 3D model of a thinner person (as shown in Figure 1A) and the 3D model of a fatter person (as shown in Figure 1B) are respectively reconstructed. The person shown in FIG. 1B is in the same posture as the person shown in FIG. 1B , and the key point information is the same, and the difference in posture between the two cannot be represented only through the key point information.

In related technologies, three-dimensional reconstruction is generally carried out by means of parameter optimization and network regression. The parameter optimization method usually selects a set of standard parameters, and uses the gradient descent method to iteratively optimize the initial values of the parameters of the 3D model of the target object according to the 2D visual features of the image of the target object. Features can select 2D keypoints, etc. The advantage of the parameter optimization method is that it can give more accurate parameter estimation results that conform to the two-dimensional visual characteristics of the image, but it often gives unnatural and unreasonable action results, and the final performance of parameter optimization is very dependent on the initial parameters. value, resulting in low reliability of the 3D reconstruction method based on parameter optimization.

The methods of network regression usually train an end-to-end neural network to learn the mapping from images to 3D model parameters. The advantage of the network regression method is that it can give more natural and reasonable action results. However, due to the lack of a large amount of training data, the 3D reconstruction results may not match the 2D visual features in the image. Therefore, the 3D reconstruction method based on network regression Less accurate. The 3D reconstruction method in the related art cannot take into account the accuracy and reliability of the 3D reconstruction results.

Based on this, an embodiment of the present disclosure provides a three-dimensional reconstruction method, as shown in FIG. 2 , the method includes:

Step 201: Perform 3D reconstruction on the target object in the image through the 3D reconstruction network to obtain the initial value of the parameter of the target object, wherein the initial value of the parameter is used to establish the 3D model of the target object;

Step 202: Optimizing the initial value of the parameter based on the pre-acquired supervision information representing the characteristics of the target object to obtain the optimized value of the parameter;

Step 203: Perform bone skinning processing based on the optimized values of the parameters, and establish a 3D model of the target object.

In step 201, the target object may be a three-dimensional object, such as a person, an animal, a robot, etc. in a physical space, or one or more regions on the three-dimensional object, such as a human face or a limb. For the convenience of description, the target object is a human being, and the three-dimensional reconstruction performed on the target object is a human body reconstruction as an example for description. The image of the target object may be a single image, or may include multiple images obtained by shooting the target object from multiple different angles of view. 3D human body reconstruction based on a single image is called monocular 3D human body reconstruction, and 3D human body reconstruction based on multiple images from different perspectives is called multi-eye 3D human body reconstruction. Each image can be a grayscale image, RGB image or RGBD image. The image may be an image captured instantly by an image acquisition device (for example, a camera or a camera) around the target object, or may be an image acquired and stored in advance.

將體態參數的初始值和姿態參數的初始值映射為人體表面的三維模型，該三維模型包括6890個頂點，頂點之間通過固定的連接關係構成三角面片。可以使用一個預訓練的回歸器W，從人體表面模型的頂點進一步回歸出人體的三維關鍵點

，即：

。 The image of the target object can be reconstructed in 3D through a 3D reconstruction network, wherein the 3D reconstruction network can be a pre-trained neural network. The 3D reconstruction network can perform 3D reconstruction based on images, and estimate the initial values of natural and reasonable parameters. The initial values of the parameters here can be represented by a vector. The dimension of the vector can be 85 dimensions, for example. The vector contains the rotation information of the human body's moving limbs (that is, the initial value of the posture parameters, including the initial values of the global rotation parameters of the human body and the initial values of the key point rotation parameters of 23 key points), the initial values of the body parameters and the parameters of the camera The initial value of these three parts of information. The human body can be represented by key points and limb bones connecting these key points. The key points of the human body can include the top of the head, nose, neck, left and right eyes, left and right ears, chest, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right buttocks, One or more of key points such as left and right knees, left and right ankles, etc., the initial value of the pose parameter is used to determine the position of the key points of the human body in three-dimensional space. The initial values of the body parameters are used to determine body information such as height, shortness, fatness, and thinness of the human body. The initial value of the parameter of the camera is used to determine the absolute position of the human body in the three-dimensional space under the camera coordinate system, and the parameter of the camera includes a displacement parameter between the camera and the human body and a posture parameter of the camera, wherein the posture parameter of the camera is The initial value can be replaced by the initial value of the global rotation parameter of the human body. The parameters of the human body can be expressed using a parametric form of a Skinned Multi-Person Linear (SMPL) model (referred to as SMPL parameters). After obtaining the value of the SMPL parameter, bone skinning can be performed based on the value of the SMPL parameter, that is, using a mapping function

The initial value of the body shape parameter and the initial value of the posture parameter are mapped to a three-dimensional model of the human body surface, the three-dimensional model includes 6890 vertices, and the vertices form a triangular patch through a fixed connection relationship. A pre-trained regressor W can be used to further regress the 3D key points of the human body from the vertices of the human surface model

,which is:

.

In step 202, the supervisory information may be two-dimensional visual features of the image (also called two-dimensional observation features), for example, two-dimensional key points of the target object in the image and multiple pixel points on the target object At least one of the semantic information for . The semantic information of a pixel is used to represent which area the pixel is located on the target object, and the area may be, for example, the area where the head, arm, torso, leg, etc. are located. In the case of using two-dimensional key point information as supervision information, the two-dimensional key point extraction network can be used to estimate the position of the key point of the human body in the image. Here, any two-dimensional pose estimation method can be used, such as OpenPose. In addition to using 2D visual features as supervisory information, 2D visual features and the initial 3D point cloud on the surface of the target object can also be used as supervisory information to further improve the accuracy of 3D reconstruction.

In the case that the image includes a depth image (for example, the image is an RGBD image), depth information of multiple pixels on the target object may be extracted from the depth image, based on the Depth information projects a plurality of pixel points on the target object in the depth image to a three-dimensional space to obtain an initial three-dimensional point cloud on the surface of the target object.

The plurality of pixels may be part or all of the pixels on the target object in the image. For example, it may include pixel points of various areas on the target object that need to be three-dimensionally reconstructed, and the number of pixel points in each area should be greater than or equal to the number required for three-dimensional reconstruction.

Because the image generally includes both the target object and the background area. Therefore, image segmentation can be performed on the RGB image included in the image, the image area where the target object is located in the RGB image is acquired, and the image area where the target object is located in the RGB image is determined based on the image area where the target object is located. The image area where the target object is located in the depth image; acquiring the depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation, the image area where the target object that needs to be three-dimensionally reconstructed is located can be extracted from the image, and the influence of the background area in the image on the three-dimensional reconstruction can be avoided. In some embodiments, the pixels in the depth image correspond one-to-one to the pixels in the RGB image. For example, the image may also be an RGBD image.

Further, outliers can also be filtered out from the 3D point cloud (ie, the initial 3D point cloud), and the supervisory information can include the filtered 3D point cloud. The filtering can be implemented using a point cloud filter. By filtering out outliers, a finer 3D point cloud of the surface of the target object can be obtained, thereby further improving the accuracy of 3D reconstruction. For each target 3D point in the 3D point cloud, obtain the average distance from the n 3D points closest to the target 3D point to the target 3D point, assuming that the average distance corresponding to each target 3D point obeys a statistical distribution (for example, Gaussian distribution), the mean and variance of the statistical distribution can be calculated, and a threshold s can be set based on the mean and variance, then the 3D points whose average distance is outside the range of the threshold s can be regarded as outliers and obtained from the 3D points Filtered out in the cloud.

In practical applications, if the image is an RGB image, the initial values of the parameters can be iteratively optimized using the two-dimensional observation features as supervisory information. If the image is an RGBD image, the two-dimensional observation features and the three-dimensional point cloud on the surface of the target object can be used as supervisory information to iteratively optimize the initial value of the parameter. The optimization method may, for example, use a gradient descent method, which is not limited in the present disclosure.

In step 203, bone skinning processing may be performed based on the optimized values of the parameters to obtain a three-dimensional model of the target object.

As shown in FIG. 3 , it is an overall flowchart of the embodiment of the present disclosure. In the case that the input is an RGB image, the RGB image can be 3D reconstructed through the 3D reconstruction network to obtain the human body parameter values of the person in the image, and the key points of the person in the image can be obtained by using the key point extraction network. Extract to obtain the two-dimensional key points of the human body. Then, the human body parameter value is used as the initial value of the parameter, and the two-dimensional key points of the human body are used as the supervision information, and the initial value of the human body parameter is optimized through the parameter optimization module to obtain the optimized value of the human body parameter, and based on the optimized value of the human body parameter. Skeleton skinning processing to obtain the human body reconstruction model.

In the case of an RGBD image input, the image can be decomposed into an RGB image and a TOF (Time of Flight, time of flight) depth map. The TOF depth map includes the depth information of each pixel in the RGB image. The RGB image can be reconstructed three-dimensionally through the three-dimensional reconstruction network to obtain the human body parameter value of the person in the image, and the key point extraction network can be used to extract the key point of the person in the image to obtain the two-dimensional key point of the human body. The point cloud reconstruction group can also be used to reconstruct the surface point cloud of the human body based on the depth information in the TOF depth map. Then, the human body parameter value is used as the initial value of the parameter, and the two-dimensional key points of the human body and the point cloud of the human body surface are used as supervision information, and the initial value of the human body parameter is optimized through the parameter optimization module to obtain the optimized value of the human body parameter, and based on The optimized value of the human body parameters is processed by bone skinning to obtain the human body reconstruction model.

Further, after obtaining the human body reconstruction model, color processing may be performed on the human body reconstruction model based on the color information in the RGB image or the RGBD image, so that the human body reconstruction model matches the color information of the person in the image.

In the embodiment of the present disclosure, the target object in the image is reconstructed three-dimensionally through the three-dimensional reconstruction network, thereby obtaining the initial value of the parameter, and then optimizing the initial value of the parameter based on the supervision information, and establishing the parameter based on the optimized value of the parameter. 3D model of the target object. The advantage of the parameter optimization method is that it can give more accurate 3D reconstruction results that conform to the 2D observation characteristics of the image, but it often gives unnatural and unreasonable action results with low reliability. The network regression through the 3D reconstruction network can give more natural and reasonable action results. Therefore, using the output of the 3D reconstruction network as the initial value of the parameters for parameter optimization can ensure the reliability of the 3D reconstruction results. On the basis of taking into account the accuracy of 3D reconstruction.

In some embodiments, in the parameter optimization stage, a multi-stage optimization method may be used. The multi-stage optimization method may include a camera optimization stage and a pose optimization stage. In the camera optimization stage, the optimization targets are the value R of the global rotation parameter and the current value t of the displacement parameter between the image acquisition device and the target object. Among them, t and R are three-dimensional vectors, and R is expressed in the form of axis and angle. In the pose optimization stage, the optimization targets are the values of key point rotation parameters and body posture parameters.

Because in the optimization process, changing the position of the camera and changing the position of the 3D key points of the human body can cause changes in the 2D projection of the 3D key points, which will make the optimization process very unstable. Therefore, in the camera optimization stage, the human body pose is fixed, and in the pose optimization stage, the camera position is fixed, thereby improving the stability of the optimization process. That is, when the initial value of the body posture parameter and the initial value of the key point rotation parameter remain unchanged, based on the supervision information and the initial value of the displacement parameter, the displacement parameter of the image acquisition device The current value and the initial value of the global rotation parameter are optimized to obtain the optimized value of the displacement parameter and the optimized value of the global rotation parameter; then keep the optimized value of the displacement parameter and the optimized value of the global rotation parameter unchanged, based on the displacement parameter The optimized value of the key point rotation parameter and the optimized value of the global rotation parameter are optimized, and the initial value of the key point rotation parameter and the initial value of the body shape parameter are optimized to obtain the optimized value of the key point rotation parameter and the optimized value of the body shape parameter.

Further, among the 2D projection key points corresponding to the 3D key points of the target object, target 2D projection key points belonging to preset parts of the target object can be acquired; wherein, the 3D key points of the target object are based on the The initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter are obtained; the two-dimensional projection key point is based on the current value of the displacement parameter and the initial value of the global rotation parameter for the target object The 3D key points of are obtained by projection. A first loss between the target 2D projection keypoint and the initial 2D keypoint is acquired. A second loss between an initial value of the displacement parameter and a current value of the displacement parameter is obtained. The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss and the second loss.

（1）；

（2）； 其中，

和

分別表示第一損失和第二損失，

和

分別表示目標二維投影關鍵點和初始二維關鍵點，

和

分別表示所述圖像採集裝置與所述目標對象之間的位移參數的當前值以及所述位移參數的初始值。可以基於第一損失和第二損失確定第一目標損失

，例如，所述第一目標損失可以確定為所述第一損失與所述第二損失之和，可通過下述公式（3）確定：

（3）。 Wherein, the preset part may be a trunk part, and the key points of the target two-dimensional projection may include left and right shoulder points, left and right hip points, spine center points and other key points. Since different actions have less influence on the key points of the torso, by using the key points of the torso to establish the first loss, the influence of different actions on the position of the key points can be reduced and the accuracy of the optimization result can be improved. The first loss can also be called torso key point projection loss, and the second loss can also be called camera displacement regularization loss. The first loss can be obtained by the following formula (1), and the second loss can be obtained by the following formula (2) get:

(1);

(2); where,

with

denote the first loss and the second loss, respectively,

with

represent the target 2D projection keypoint and the initial 2D keypoint, respectively,

with

Respectively represent the current value of the displacement parameter between the image acquisition device and the target object and the initial value of the displacement parameter. The first target loss can be determined based on the first loss and the second loss

, for example, the first target loss can be determined as the sum of the first loss and the second loss, which can be determined by the following formula (3):

(3).

A third loss between an optimized 2D projection keypoint of the target object and the initial 2D keypoint may be obtained, wherein the optimized 2D projection keypoint is based on an optimized value of the displacement parameter and a global rotation parameter The optimized value of is obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter. A fourth loss is obtained, and the fourth loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter. Optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the third loss and the fourth loss.

（4）； 其中，

為第三損失，

和

分別表示所述優化二維投影關鍵點以及所述初始二維關鍵點。可以基於第三損失和第四損失確定第二目標損失，例如，所述第二目標損失可以確定為所述第三損失與所述第四損失之和，可通過下述公式（5）確定：

（5）； 其中，

為第二目標損失，

為第四損失，可以採用高斯混合模型（Gaussian Mixture Model，GMM）來獲取，用於判斷全域旋轉參數的優化值、關鍵點旋轉參數的初始和體態參數的初始值對應的姿態是否合理，對不合理的姿態輸出較大的損失。 The third loss can also be called the two-dimensional key point projection loss, the fourth loss can also be called the attitude rationality loss, and the third loss can be determined by the following formula (4):

(4); where,

for the third loss,

with

represent the optimized two-dimensional projection key points and the initial two-dimensional key points respectively. The second target loss may be determined based on the third loss and the fourth loss, for example, the second target loss may be determined as the sum of the third loss and the fourth loss, which may be determined by the following formula (5):

(5); where,

for the second target loss,

It is the fourth loss, which can be obtained by using the Gaussian Mixture Model (GMM), which is used to judge whether the optimal value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the body shape parameter correspond to a reasonable attitude, right? Reasonable poses output larger losses.

After optimizing the initial value of the key point rotation parameter and the initial value of the posture parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the optimized value of the global rotation parameter may also be The optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized, that is, a three-stage optimization method is adopted. For the case where the supervision information includes the 3D point cloud information of the surface of the target object, the three-stage optimization method can be adopted, including camera optimization stage, pose optimization stage and point cloud optimization stage.

（6）； 式中，

為所述第五損失，將所述初始三維點雲看作點雲P，將所述第一三維點雲看作點雲Q，

為點雲P中的每個點到點雲Q中距離最近的點構成的點對集合，

為點雲Q中的每個點到點雲P中距離最近的點構成的點對集合。第一損失和第二損失分別通過如下公式（7）和公式（8）表示：

（7）；

（8）； 其中，

和

分別表示第一損失和第二損失，

和

分別表示目標二維投影關鍵點和初始二維關鍵點，

和

分別表示所述位移參數的當前值以及所述位移參數的初始值。可以基於第一損失、第二損失和第五損失之和確定第一目標損失

，再基於第一目標損失對所述位移參數的當前值和全域旋轉參數的初始值進行優化，即，如以下公式（9）：

（9）。 In the camera optimization stage, among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object, the target two-dimensional projection key points belonging to the preset parts of the target object can be obtained; wherein, the three-dimensional key points of the target object Based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, the key point of the two-dimensional projection is based on the current value of the displacement parameter and the initial value of the global rotation parameter. The 3D key points of the target object are obtained by projection. A first loss between the target 2D projection keypoint and the initial 2D keypoint is acquired. A second loss between an initial value of the displacement parameter and a current value of the displacement parameter is obtained. Acquiring the fifth loss between the first 3D point cloud of the surface of the target object and the initial 3D point cloud; wherein, the first 3D point cloud is based on the initial value of the global rotation parameter and the key point rotation parameter Initial values and initial values of body parameters are obtained. The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss. The fifth loss can also be called the Iterative Closest Point (ICP) point cloud registration loss, which can be determined by the following formula (6):

(6); where,

For the fifth loss, the initial 3D point cloud is regarded as point cloud P, and the first 3D point cloud is regarded as point cloud Q,

For each point in the point cloud P to the closest point in the point cloud Q, the set of point pairs,

It is a set of point pairs formed from each point in point cloud Q to the nearest point in point cloud P. The first loss and the second loss are expressed by the following formula (7) and formula (8) respectively:

(7);

(8); where,

with

denote the first loss and the second loss, respectively,

with

represent the target 2D projection keypoint and the initial 2D keypoint, respectively,

with

represent the current value of the displacement parameter and the initial value of the displacement parameter respectively. The first target loss can be determined based on the sum of the first loss, the second loss and the fifth loss

, and then optimize the current value of the displacement parameter and the initial value of the global rotation parameter based on the first target loss, that is, as the following formula (9):

(9).

The attitude optimization stage in the three-stage optimization process is the same as the attitude optimization stage in the two-stage optimization process, and will not be repeated here.

（10）；

（11）。 式中，

為第六損失，

為優化二維投影關鍵點，

為初始二維關鍵點。第七損失可以採用高斯混合模型來獲取，用於判斷全域旋轉參數的優化值、關鍵點旋轉參數的優化值和體態參數的優化值對應的姿態是否合理，對不合理的姿態輸出較大的損失。

為第八損失，P為所述初始三維點雲看作點雲，

為所述第二三維點雲，

為點雲P中的每個點到點雲

中距離最近的點構成的點對集合，

為點雲

中的每個點到點雲P中距離最近的點構成的點對集合。進一步地，可以將第六損失、第七損失和第八損失之和確定為第三目標損失

，並基於第三目標損失對所述全域旋轉參數的優化值、所述關鍵點旋轉參數的優化值、體態參數的優化值以及所述位移參數的優化值進行聯合優化，可通過以下公式（12）進行聯合優化：

（12）。 In the point cloud optimization stage, the sixth loss between the optimized 2D projection keypoint of the target object and the initial 2D keypoint can be obtained, wherein the optimized 2D projection keypoint is based on the displacement parameter The optimized value and the optimized value of the global rotation parameter are obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the body shape parameter. The optimized value is obtained. A seventh loss is obtained, and the seventh loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter. Obtain an eighth loss between the second 3D point cloud of the surface of the target object and the initial 3D point cloud; wherein, the second 3D point cloud is based on the optimized value of the global rotation parameter and the key point rotation parameter The optimal value and the optimal value of body parameters are obtained. Based on the sixth loss, the seventh loss and the eighth loss, jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter , can be optimized by the following formula (10) and formula (11):

(10);

(11). In the formula,

for the sixth loss,

To optimize the 2D projection keypoints,

is the initial two-dimensional keypoint. The seventh loss can be obtained by using a Gaussian mixture model, which is used to judge whether the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter is reasonable, and outputs a large loss for an unreasonable posture. .

is the eighth loss, P is the initial 3D point cloud as a point cloud,

is the second 3D point cloud,

For each point in the point cloud P to the point cloud

A set of point pairs consisting of the closest points in the middle,

for the point cloud

A set of point pairs from each point in point cloud P to the nearest point in point cloud P. Further, the sum of the sixth loss, the seventh loss and the eighth loss can be determined as the third target loss

, and based on the third objective loss, jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter, which can be obtained by the following formula (12 ) for joint optimization:

(12).

In the case where the image of the target object is an RGB image, parameter optimization can be performed based on the aforementioned two-stage optimization method including the camera optimization stage and the attitude optimization stage; The aforementioned three-stage optimization method including camera optimization stage, attitude optimization stage and point cloud optimization stage performs parameter optimization.

This solution can be used in a wide range of scenarios, and can provide natural, reasonable and accurate human body reconstruction models in scenarios such as virtual fitting rooms, virtual anchors, and video action migration.

As shown in FIG. 4A , it is a schematic diagram of an application scene of a virtual fitting room according to an embodiment of the present disclosure. The image of the user 401 can be collected by the camera 403, and the collected image can be sent to a processor (not shown in the figure) for three-dimensional human body reconstruction, so as to obtain the human body reconstruction model 404 corresponding to the user 401, and the human body reconstruction model 404 displayed on the display interface 402 for the user 401 to watch. At the same time, the user 401 can select the required clothing 405, including but not limited to clothing 4051 and hat 4052, etc., and the clothing 405 can be displayed on the display interface 402 based on the human body reconstruction model 404, so that the user 401 can watch the wearing effect of the clothing 405.

As shown in FIG. 4B , it is a schematic diagram of an application scenario of a virtual live broadcast room according to an embodiment of the present disclosure. During the live broadcast, the image of the anchor user 406 can be collected by the anchor client 407, and the image of the anchor user 406 can be sent to the server 408 for three-dimensional reconstruction to obtain the human body reconstruction model of the anchor user, that is, the virtual anchor. The server 408 can return the human body reconstruction model of the anchor user to the anchor client 407 for display, as shown in the model 4071 in the figure. In addition, the host client 407 can also collect the voice information of the host user, and send the voice information to the server 408, so that the server 408 can fuse the reconstruction model of the human body and the voice information. The server 408 can send the fused human body reconstruction model and voice information to the viewer client 409 watching the live program for display and playback, wherein the displayed human body reconstruction model is shown as model 4091 in the figure. Through the above method, the live broadcast screen of the virtual anchor can be displayed on the viewer client 409 .

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

As shown in FIG. 5 , the present disclosure also provides a three-dimensional reconstruction device, which includes:

The first 3D reconstruction group 501 is configured to perform 3D reconstruction on the target object in the image through the 3D reconstruction network to obtain the initial value of the parameter of the target object, and the initial value of the parameter is used to establish the target object 3D model of

An optimization module 502, configured to optimize the initial value of the parameter based on the pre-acquired supervisory information representing the characteristics of the target object, to obtain the optimized value of the parameter;

The second 3D reconstruction group 503 is configured to perform bone skinning processing based on the optimized values of the parameters, and establish a 3D model of the target object.

In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: the target object's initial Two-dimensional key points, semantic information of multiple pixel points on the target object in the image; the second supervisory information includes an initial three-dimensional point cloud of the surface of the target object. In the embodiment of the present disclosure, only the initial two-dimensional key points of the target object or the semantic information of the pixels can be used as the supervisory information to optimize the initial value of the parameter, so that the optimization efficiency is high and the optimization complexity is low; or, the The initial 3D point cloud of the surface of the target object and the semantic information of the aforementioned initial 2D key points or pixels are used together as supervisory information, thereby improving the accuracy of the optimized value of the obtained parameters.

In some embodiments, the device further includes: a 2D key point extraction module, configured to extract initial 2D key point information of the target object from the image through a key point extraction network. Using the information of the initial 2D key points extracted by the key point extraction network as supervision information can generate more natural and reasonable actions for the 3D model.

In some embodiments, the image includes a depth image of the target object; the device further includes: a depth information extraction module, configured to extract a plurality of pixels on the target object from the depth image Depth information of points; a backprojection module, used to backproject multiple pixel points on the target object in the depth image to three-dimensional space based on the depth information, to obtain the initial surface of the target object 3D point cloud. By extracting the depth information and back-projecting the pixels on the 2D image to the 3D space based on the depth information, the initial 3D point cloud of the surface of the target object can be obtained, so that the initial 3D point cloud can be used as the supervisory information to optimize the parameters. The initial value further improves the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; the depth information extraction module includes: an image segmentation unit for performing image segmentation on the RGB image, the image An area determining unit, configured to determine the image area where the target object is located in the RGB image based on the image segmentation result, and determine the target object in the depth image based on the image area where the target object is located in the RGB image The image area where the target object is located; a depth information acquisition unit, configured to acquire depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.

In some embodiments, the device further includes: a filtering module, configured to filter out outliers from the initial 3D point cloud, and use the filtered initial 3D point cloud as the second supervisory information. By filtering the outliers, the interference of the outliers is reduced, and the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is collected by an image acquisition device, and the parameters include: global rotation parameters of the target object, key point rotation parameters of each key point of the target object, the The posture parameters of the target object and the displacement parameters of the image acquisition device; the optimization module includes: a first optimization unit, which is used to keep the initial value of the posture parameter and the initial value of the key point rotation parameter unchanged In this case, based on the supervision information and the initial value of the displacement parameter, the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter are optimized to obtain the optimized value of the displacement parameter and the global rotation parameter. The optimized value of the rotation parameter; the second optimization unit is used to optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, The optimal values of the key point rotation parameters and the optimal values of the body parameters are obtained. During the optimization process, changing the position of the image acquisition device and changing the position of the 3D key points can lead to changes in the 2D projection of the 3D key points, which will lead to an unstable optimization process. By using a two-stage optimization method, firstly fix the initial value of the key point rotation parameter and the initial value of the body shape parameter to optimize the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter, and then fix the displacement parameter The initial value and the initial value of the global rotation parameter optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter, which improves the stability of the optimization process.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the first optimization unit is configured to: obtain the location of the two-dimensional projected key points corresponding to the three-dimensional key points of the target object The target two-dimensional projection key point of the preset part of the target object; wherein, the three-dimensional key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, The two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the target two-dimensional projection key point and the initial two-dimensional a first loss between key points; obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; based on the first loss and the second loss on the current value of the displacement parameter and the initial value of the global rotation parameter are optimized. The preset part can be the torso and other parts. Since different actions have little influence on the key points of the torso, the first loss can be determined by using the key points of the torso, which can reduce the influence of different actions on the position of the key points and improve Optimize the accuracy of the results. Since the two-dimensional key points are supervisory information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, it is possible to reduce the optimization result falling into the local optimal point on the two-dimensional plane, thereby Situations that deviate from the true point.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the second optimization unit is configured to: obtain the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points The third loss between points, the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized three-dimensional key point The point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the body shape parameter; obtain the fourth loss, and the fourth loss is used to characterize the optimized value of the global rotation parameter, the key point The rationality of the attitude corresponding to the initial value of the rotation parameter and the initial value of the posture parameter; based on the third loss and the fourth loss, the initial value of the key point rotation parameter and the initial value of the posture parameter are optimized . This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, which improves the stability of the optimization process. At the same time, the fourth loss ensures the optimization The latter parameters correspond to the rationality of the pose.

In some embodiments, the device further includes: a joint optimization module, which is used to optimize the initial value of the key point rotation parameter and the body posture based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter. After the initial value of the parameter is optimized, the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized. In this embodiment, on the basis of the aforementioned optimization, the optimized parameters are jointly optimized, thereby further improving the accuracy of the optimization result.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; the first optimization unit is configured to: acquire the three-dimensional key points of the target object Among the two-dimensional projection key points corresponding to the point, the target two-dimensional projection key point belonging to the preset part of the target object; wherein, the three-dimensional key point of the target object is based on the initial value of the global rotation parameter, the key point rotation parameter The initial value of the initial value and the initial value of the posture parameter are obtained, and the key points of the two-dimensional projection are obtained by projecting the three-dimensional key points of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the target The first loss between the two-dimensional projection key point and the initial two-dimensional key point; obtain the second loss between the initial value of the displacement parameter and the current value of the displacement parameter; obtain the target object surface The fifth loss between the first 3D point cloud and the initial 3D point cloud; the first 3D point cloud is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter ; optimizing the current value of the displacement parameter and the initial value of the global rotation parameter based on the first loss, the second loss and the fifth loss. In this embodiment, the three-dimensional pointfish is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

In some embodiments, the joint optimization module includes: a first acquisition unit, configured to acquire a sixth loss between the optimized 2D projection keypoint of the target object and the initial 2D keypoint, the The optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized three-dimensional key point is based on the optimized value of the global rotation parameter , the optimized value of the key point rotation parameter and the optimized value of the posture parameter are obtained; the second acquisition unit is used to obtain the seventh loss, and the seventh loss is used to characterize the optimized value of the global rotation parameter and the key point rotation parameter The optimal value and the rationality of the posture corresponding to the optimal value of the body parameters; the third acquisition unit is used to acquire the eighth loss between the second 3D point cloud on the surface of the target object and the initial 3D point cloud; the The second three-dimensional point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the body shape parameter; a joint optimization unit is used for based on the sixth loss, the seventh loss and the eighth loss Jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. In this embodiment, the three-dimensional pointfish is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments, and its specific implementation can refer to the description of the above method embodiments. For the sake of brevity, I won't go into details here.

As shown in FIG. 6, the present disclosure also provides a three-dimensional reconstruction system, which includes:

An image acquisition device 601, configured to acquire an image of a target object; and

The processing unit 602 communicated with the image acquisition device 601 is configured to perform three-dimensional reconstruction on the target object in the image through the three-dimensional reconstruction network to obtain the initial value of the parameter of the target object, and the parameter The initial value is used to establish the three-dimensional model of the target object; the initial value of the parameter is optimized based on the pre-acquired supervisory information representing the characteristics of the target object to obtain the optimized value of the parameter; based on the parameter The optimized value is subjected to bone skinning processing to establish a 3D model of the target object.

The image acquisition device 601 in the embodiment of the present disclosure may be a device with an image acquisition function such as a camera or a camera, and the images collected by the image acquisition device 601 may be transmitted to the processing unit 602 in real time, or stored, and when needed From the storage space to the processing unit 602. The processing unit 602 may be a single server or a server cluster composed of multiple servers. For the method executed by the processing unit 602, refer to the above-mentioned embodiment of the three-dimensional reconstruction method for details, and details are not repeated here.

The embodiment of this specification also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein, when the processor executes the program, any of the foregoing embodiments is realized the method described.

FIG. 7 shows a schematic diagram of a more specific hardware structure of a computing device provided by the embodiment of this specification. The device may include: a processor 701, a storage 702, an input/output interface 703, a communication interface 704, and a bus 705 . The processor 701 , the storage 702 , the input/output interface 703 and the communication interface 704 are connected to each other within the device through the bus bar 705 .

The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more substrate circuits, and is used to execute Relevant programs to realize the technical solutions provided by the embodiments of this specification. The processor 701 may also include a graphics card, and the graphics card may be an Nvidia titan X graphics card or a 1080Ti graphics card.

The storage 702 may be implemented in the form of ROM (Read Only Memory, read-only memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, and the like. The storage 702 can store operating systems and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the storage 702 and called and executed by the processor 701. .

The input/output interface 703 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be connected externally to the device to provide corresponding functions. The input devices may include keyboards, mice, touch screens, microphones, and various sensors, and the output devices may include displays, speakers, vibrators, and indicator lights.

The communication interface 704 is used to connect a communication module (not shown in the figure), so as to realize the communication interaction between the device and other devices. Among them, the communication module can realize communication through wired methods (such as USB, network cable, etc.), and can also realize communication through wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

The bus 705 includes a path for transferring information between various components of the device (eg, the processor 701 , the memory 702 , the input/output interface 703 and the communication interface 704 ).

It should be noted that although the above device only shows the processor 701, the storage 702, the input/output interface 703, the communication interface 704 and the bus bar 705, in the specific implementation process, the device may also include Additional components required. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

An embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the foregoing embodiments is implemented.

Computer-readable media includes both volatile and non-permanent, removable and non-removable media and can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM) , Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash memory or other memory technologies, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassette, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media, such as modulated data signals and carrier waves.

It can be known from the above description of the implementation manners that those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus a necessary general-purpose hardware platform. Based on this understanding, the essence of the technical solutions of the embodiments of this specification or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of this specification.

The systems, devices, modules or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a notebook computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email sending and receiving device, a game console, A tablet computer, a wearable device, or a combination of any of these devices.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated. When implementing the embodiments of this specification, the functions of each module may be integrated implemented in one or more software and/or hardware. Part or all of the modules can also be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

201: Perform three-dimensional reconstruction on the target object in the image through the three-dimensional reconstruction network to obtain the initial value of the parameter of the target object, and the initial value of the parameter is used to establish the three-dimensional model of the target object 202: Optimizing the initial value of the parameter based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain the optimized value of the parameter 203: Carry out bone skinning processing based on the optimized value of the parameter, and establish a three-dimensional model of the target object 401: user 402: display interface 403: camera 404: Human reconstruction model 405: Apparel 4051: clothes 4052: hat 406: anchor user 407: Anchor client 4071: model 408: server 409: audience client 4091: model 501: The first 3D reconstruction group 502: Optimize the module 503: Second 3D reconstruction group 601: image acquisition device 602: processing unit 701: Processor 702: Storage 703: Input/Output Interface 704: communication interface 705: Bus

1A and 1B are schematic illustrations of three-dimensional models of some embodiments. Fig. 2 is a flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure. FIG. 3 is an overall flowchart of an embodiment of the present disclosure. FIG. 4A and FIG. 4B are schematic diagrams of application scenarios of embodiments of the present disclosure, respectively. FIG. 5 is a block diagram of a three-dimensional reconstruction device according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram of a three-dimensional reconstruction system according to an embodiment of the present disclosure. FIG. 7 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.

201: Perform three-dimensional reconstruction on the target object in the image through the three-dimensional reconstruction network to obtain the initial value of the parameter of the target object, and the initial value of the parameter is used to establish the three-dimensional model of the target object

202: Optimizing the initial value of the parameter based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain the optimized value of the parameter

203: Carry out bone skinning processing based on the optimized value of the parameter, and establish a three-dimensional model of the target object

Claims (13)

A three-dimensional reconstruction method, said method comprising: Performing three-dimensional reconstruction on the target object in the image through a three-dimensional reconstruction network to obtain an initial value of a parameter of the target object, wherein the initial value of the parameter is used to establish a three-dimensional model of the target object; Optimizing the initial value of the parameter based on the pre-acquired supervisory information representing the characteristics of the target object to obtain the optimized value of the parameter; Skeleton skinning processing is performed based on the optimized values of the parameters, and a three-dimensional model of the target object is established. The method according to claim 1, wherein the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; The first supervisory information includes at least one of the following: initial two-dimensional key points of the target object, semantic information of multiple pixels on the target object in the image; The second supervisory information includes an initial 3D point cloud of the surface of the target object. The method according to claim 2, wherein the image includes a depth image of the target object; the method also includes: extracting depth information of the plurality of pixel points on the target object from the depth image; Back-projecting the plurality of pixel points on the target object in the depth image to a three-dimensional space based on the depth information to obtain the initial three-dimensional point cloud of the surface of the target object. The method according to claim 3, wherein the image further includes an RGB image of the target object; the depth information of the plurality of pixels on the target object is extracted from the depth image ,include: Carry out image segmentation to described RGB image; Determining the image area where the target object is located in the RGB image based on the image segmentation result; determining the image area where the target object is located in the depth image based on the image area where the target object is located in the RGB image; Depth information of the plurality of pixels in the image area where the target object is located in the depth image is obtained. The method according to any one of claims 2 to 4, wherein the method further comprises: Filter out outliers from the initial 3D point cloud, and use the filtered initial 3D point cloud as the second supervisory information. The method according to any one of claim items 1 to 5, wherein the image of the target object is collected by an image acquisition device, and the parameters include: global rotation parameters of the target object, the target Key point rotation parameters of each key point of the object, body posture parameters of the target object, and displacement parameters of the image acquisition device; Optimizing initial values of the parameters based on pre-acquired supervisory information representing features of the target object, including: Under the condition that the initial value of the posture parameter and the initial value of the key point rotation parameter remain unchanged, based on the supervision information and the initial value of the displacement parameter, the displacement of the image acquisition device is optimizing the current value of the parameter and the initial value of the global rotation parameter to obtain the optimized value of the displacement parameter and the optimized value of the global rotation parameter; Based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, optimize the initial value of the key point rotation parameter and the initial value of the body posture parameter, and obtain the optimized value of the key point rotation parameter and the optimal value of the key point rotation parameter. Optimum values for the body parameters. The method according to claim 6, wherein the supervisory information includes initial two-dimensional key points of the target object; Optimizing the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter based on the supervisory information and the initial value of the displacement parameter, including: Obtaining the target two-dimensional projection key points corresponding to the three-dimensional key points of the target object, which belong to the preset part of the target object, among the two-dimensional projection key points; wherein, the three-dimensional key points of the target object are based on the global rotation parameter The initial value of the initial value of the key point rotation parameter and the initial value of the body shape parameter are obtained, and the key point of the two-dimensional projection is based on the current value of the displacement parameter and the initial value of the global rotation parameter. The three-dimensional key points of the object are obtained by projection; obtaining a first loss between the target 2D projected keypoint and the initial 2D keypoint; obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss and the second loss. The method according to claim 6 or 7, wherein the supervisory information includes the initial two-dimensional key points of the target object; based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the The initial value of the key point rotation parameter and the initial value of the body shape parameter are optimized, including: Obtaining a third loss between the optimized two-dimensional projection keypoint of the target object and the initial two-dimensional keypoint, wherein the optimized two-dimensional projection keypoint is based on the optimized value of the displacement parameter and the global rotation The optimized value of the parameter is obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter worth it; Obtaining a fourth loss, the fourth loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the body posture parameter; Optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the third loss and the fourth loss. The method according to any one of claim items 6 to 8, wherein the initial value of the key point rotation parameter and the body posture are adjusted based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter After the initial values of the parameters are optimized, the method further includes: A joint optimization is performed on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. The method according to claim 9, wherein the supervision information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; based on the supervision information and the displacement parameter The initial value of the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter are optimized, including: Obtaining the target two-dimensional projection key points corresponding to the three-dimensional key points of the target object, which belong to the preset part of the target object, among the two-dimensional projection key points; wherein, the three-dimensional key points of the target object are based on the global rotation parameter The initial value of the initial value of the key point rotation parameter and the initial value of the body shape parameter are obtained, and the key point of the two-dimensional projection is based on the current value of the displacement parameter and the initial value of the global rotation parameter. The 3D key points of the target object are obtained by projection; obtaining a first loss between the target 2D projected keypoint and the initial 2D keypoint; obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; Obtain a fifth loss between the first 3D point cloud of the surface of the target object and the initial 3D point cloud; wherein, the first 3D point cloud is based on the initial value of the global rotation parameter, the key point rotation The initial value of the parameter and the initial value of the posture parameter are obtained; The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss. The method as described in claim item 9 or 10, characterized in that, the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter Perform joint optimization, including: Obtaining a sixth loss between the optimized two-dimensional projection keypoint of the target object and the initial two-dimensional keypoint, wherein the optimized two-dimensional projection keypoint is based on the optimized value of the displacement parameter and the global rotation The optimized value of the parameter is obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized body shape parameter worth it; Obtaining a seventh loss, the seventh loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter; Obtain an eighth loss between the second 3D point cloud of the surface of the target object and the initial 3D point cloud; the second 3D point cloud is based on the optimized value of the global rotation parameter and the key point rotation parameter Optimal value and the optimal value of described posture parameter are obtained; Based on the sixth loss, the seventh loss and the eighth loss, the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the displacement parameter The optimized value is jointly optimized. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of claims 1 to 11 is implemented. A computer device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, when the processor executes the program, the method as described in any one of claims 1 to 11 is implemented .

Images