KR20230078777A - 3D reconstruction methods, devices and systems, media and computer equipment - Google Patents

Abstract

The present invention provides a 3D reconstruction method, apparatus and system, medium and computer equipment, wherein the method performs 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain initial values of parameters of the target object. obtaining - initial values of the parameters are used to build a three-dimensional model of the target object; obtaining an optimized value of the parameter by optimizing an initial value of the parameter based on previously obtained supervision information representing characteristics of a target object; and constructing a 3D model of the target object by performing a skeletal skinning process based on the optimized values of the parameters.

Description

3D reconstruction methods, devices and systems, media and computer equipment

[Cross-citation of related applications]

The present invention claims priority of a Chinese patent application filed on May 10, 2021, with application number 202110506464.X, entitled "Three-dimensional reconstruction method, apparatus and system, medium and computer apparatus", All contents of this Chinese patent application are incorporated into the present invention by reference.

[Technical field]

The present invention relates to the field of computer vision technology, and more particularly to three-dimensional reconstruction methods, devices and systems, media and computer equipment.

3D reconstruction is one of the important techniques in computer vision and has many potential applications in augmented reality, virtual reality and other fields. A posture and limb rotation of the target object may be reconstructed by performing 3D reconstruction on the target object. However, existing 3D reconstruction methods cannot simultaneously consider the accuracy and reliability of reconstruction results.

The present invention provides a 3D reconstruction method, apparatus and system, medium and computer equipment.

According to a first aspect of the embodiments of the present invention, there is provided a 3D reconstruction method, comprising: performing 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain initial values of parameters of the target object obtaining - the initial values of the parameters are used to build a 3D model of the target object; obtaining an optimized value of the parameter by optimizing an initial value of the parameter based on pre-obtained supervision information representing characteristics of a target object; and constructing a 3D model of the target object by performing a skeleton skinning process based on the optimized values of the parameters.

In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of an initial two-dimensional key point of the target object and semantic information of a plurality of pixel points of the target object in the image; The second supervision information includes an initial 3D point cloud of the surface of the target object. The embodiments of the present invention optimize the initial values of the parameters by using only the semantic information of the initial two-dimensional key points or pixel points of the target object as supervision information, so that the optimization efficiency is high and the optimization complexity is low; Alternatively, the initial 3D point cloud of the target object surface and the aforementioned initial 2D key point or pixel point semantic information are used together as supervision information to improve the accuracy of the optimized value of the acquired parameter.

In some embodiments, the method further includes extracting information of an initial two-dimensional key point of the target object from the image through a key point extraction network. A natural and reasonable motion can be created for a 3D model by using the initial 2D key point information extracted by the key point extraction network as supervision information.

In some embodiments, the image includes a depth map of the target object; The method may include: extracting depth information of a plurality of pixel points of the target object from the depth map; The method further comprises obtaining an initial 3D point cloud of a surface of the target object by back-projecting a plurality of pixel points of the target object in the depth map into a 3D space based on the depth information. Depth information is extracted, and based on the depth information, pixel points in the 2D image are back-projected onto a 3D space to obtain an initial 3D point cloud of the surface of the target object, and the initial 3D point cloud is used as supervision information. Optimize the initial values of parameters, and further improve the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; The step of extracting depth information of a plurality of pixel points of the target object from the depth map: performing image segmentation on the RGB image, and based on a result of the image segmentation, an image area where the target object in the RGB image is located. determining an image area where the target object in the depth map is located based on an image area where the target object in the RGB image is located; and acquiring depth information of a plurality of pixel points of an image area in which the target object is located in the depth map. By performing image segmentation on the RGB image to accurately determine the location of the target object, depth information of the extraction target object can be accurately extracted.

In some embodiments, the method further includes filtering outliers in the initial 3D point cloud and using the filtered initial 3D point cloud as the second supervisory information. Through outlier filtering, the interference of outliers is mitigated, and furthermore, the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is collected and obtained by an image acquisition device, and the parameters include a global rotation parameter of the target object, a key point rotation parameter of each key point of the target object, and a state of the target object. parameters and displacement parameters of the image acquisition device; The step of optimizing the initial values of the parameters based on the pre-obtained supervision information used to represent the target object characteristics: in a situation where the initial values of the shape parameters and the initial values of the key point rotation parameters do not change, the Optimizing a current value of a displacement parameter and an initial value of the global rotation parameter of the image acquisition device based on supervision information and an initial value of the displacement parameter to obtain an optimized value of the displacement parameter and an optimized value of the global rotation parameter. ; Based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the shape parameter are optimized to obtain the optimized value of the key point rotation parameter and the optimized value of the shape parameter. Including getting the value. In the optimization process, changing the position of the image acquisition device or changing the position of the 3D key point may cause a change in the 2D projection of the 3D key point, and thus the optimization process becomes unstable. By using a two-step optimization method, the initial values of the key point rotation parameters and the initial values of the shape parameters are fixed in advance to optimize the initial values of the displacement parameters and the initial values of the global rotation parameters of the image acquisition device. The initial values of the initial values and the global rotation parameters are fixed, and the initial values of the key point rotation parameters and the initial values of the shape parameters are optimized to improve the stability of the optimization process.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object; Optimizing the current value of the displacement parameter and the initial value of the global rotation parameter of the image capture device based on the supervision information and the initial value of the displacement parameter includes: Acquiring a target 2D projection key point belonging to a preset portion of the target object among projection key points - the 3D key point of the target object includes the initial value of the global rotation parameter, the initial value of the key point rotation parameter, and the shape obtained based on an initial value of a parameter, and the 2D projection key point is obtained by performing projection on a 3D key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining a first loss between the target 2D projection key point and the initial 2D key point; obtaining a second loss between the initial value of the displacement parameter and the current value of the displacement parameter; and optimizing a current value of the displacement parameter and an initial value of the global rotation parameter based on the first loss and the second loss. The preset part may be a torso or the like, and since different motions have a small effect on key points of the torso, the first loss is determined using the key points of the torso to reduce the influence of different motions on key point positions. and improve the accuracy of optimization results. Since the two-dimensional key point is the supervision information of the two-dimensional plane, and the displacement parameter of the image acquisition device is the parameter of the three-dimensional plane, by obtaining the second loss, the optimization result falls to the local optimization point of the two-dimensional plane at the actual point. You can reduce the chances of getting out of the way.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object; The step of optimizing the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter includes: an optimized two-dimensional projection key point of the target object and obtaining a third loss between the initial 2D key points - the optimized 2D projection key point is an optimized 3D key of the target object based on an optimized value of the displacement parameter and an optimized value of a global rotation parameter; point is obtained by performing projection, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the shape parameter -; obtaining a fourth loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter; and optimizing an initial value of the key point rotation parameter and an initial value of the shape parameter based on the third loss and the fourth loss. This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter to improve the stability of the optimization process, and at the same time reduce the fourth loss. Through this, the rationality of the shape corresponding to the parameters after optimization is secured.

In some embodiments, the method further includes optimizing an initial value of the key point rotation parameter and an initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and then the global rotation. and performing joint optimization on the optimized value of the parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter. This embodiment further improves the accuracy of the optimization result by performing coalition optimization on each parameter after optimization based on the above optimization.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object and an initial three-dimensional point cloud of a surface of the target object; Optimizing the current value of the displacement parameter and the initial value of the global rotation parameter of the image capture device based on the supervision information and the initial value of the displacement parameter includes: Obtaining a target 2D projection key point belonging to a preset portion of the target object among projection key points, wherein the 3D key point of the target object includes an initial value of the global rotation parameter, an initial value of a key point rotation parameter, and a shape parameter. -; obtaining a first loss between the target 2D projection key point and the initial 2D key point; obtaining a second loss between the initial value of the displacement parameter and the current value of the displacement parameter; obtaining a fifth loss between a first 3D point cloud of the surface of the target object and the initial 3D point cloud; obtaining the first 3-dimensional point cloud based on the initial values of the global rotation parameters, the initial values of the key point rotation parameters, and the initial values of the shape parameters; and optimizing the current value of the displacement parameter and the initial value of the global rotation parameter based on the first loss, the second loss and the fifth loss. This embodiment improves the accuracy of the optimization result by optimizing each initial parameter by adding a 3D point cloud to the supervisory information.

In some embodiments, performing joint optimization on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter, and the optimized value of the displacement parameter: obtaining a sixth loss between an optimized 2D projection key point of an object and the initial 2D key point, wherein the optimized 2D projection key point is based on an optimized value of the displacement parameter and an optimized value of a global rotation parameter; to obtain by performing projection on an optimized 3D key point of the target object, wherein the optimized 3D key point corresponds to an optimized value of the global rotation parameter, an optimized value of a key point rotation parameter, and an optimized value of a shape parameter. obtained based on -; obtaining a seventh loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter; obtaining an eighth loss between a second 3D point cloud of the target object surface and the initial 3D point cloud, wherein the second 3D point cloud is an optimized value of the global rotation parameter, a key point rotation parameter Obtained based on the optimized value and the optimized value of the shape parameter -; For the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter based on the sixth loss, the seventh loss and the eighth loss. and performing federation optimization. This embodiment improves the accuracy of the optimization result by optimizing each initial parameter by adding a 3D point cloud to the supervisory information.

According to a second aspect of the embodiments of the present invention, a 3D reconstruction device is provided, and the device includes a first 3D reconstruction module, an optimization module, and a second 3D reconstruction module. The first 3D reconstruction module is used to perform 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain an initial value of a parameter of the target object, wherein the initial value of the parameter is the target object. used to build a three-dimensional model of an object; the optimization module is used to obtain an optimized value of the parameter by optimizing the initial value of the parameter according to pre-obtained supervision information representing characteristics of the target object; The second 3D reconstruction module is used to construct a 3D model of the target object by performing skeletal skinning processing based on the optimized values of the parameters.

In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of an initial two-dimensional key point of the target object and semantic information of a plurality of pixel points of the target object in the image; The second supervision information includes an initial 3D point cloud of the surface of the target object. In an embodiment of the present invention, only semantic information of an initial two-dimensional key point or pixel point of a target object is used as supervision information to optimize the initial values of the parameters so that optimization efficiency is high and optimization complexity is low; Alternatively, the initial 3D point cloud of the target object surface and the aforementioned initial 2D key point or pixel point semantic information are used together as supervision information to improve the accuracy of the optimized value of the acquired parameter.

In some embodiments, the device further includes a 2D key point extraction module, which is used to extract information of an initial 2D key point of the target object from the image through a key point extraction network. A natural and reasonable motion can be created for a 3D model by using the initial 2D key point information extracted by the key point extraction network as supervision information.

In some embodiments, the image includes a depth map of the target object; The device further includes a depth information extraction module and a back-projection module. The depth information extraction module is used to extract depth information of a plurality of pixel points of the target object from the depth map; The back-projection module is used to back-project a plurality of pixel points of the target object in the depth map into a 3-dimensional space based on the depth information to obtain an initial 3-dimensional point cloud of a surface of the target object. Depth information is extracted, and based on the depth information, pixel points in the 2D image are back-projected onto a 3D space to obtain an initial 3D point cloud of the surface of the target object, and the initial 3D point cloud is used as supervision information. Optimize the initial values of parameters, and further improve the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; The depth information extraction module includes an image segmentation unit, an image area determination unit, and a depth information acquisition unit. the image segmentation unit is used to perform image segmentation on the RGB image; The image area determination unit determines an image area where a target object in the RGB image is located based on a result of image segmentation, and a target object in the depth map is determined based on the image area where the target object in the RGB image is located. used to determine the region of the image to be located; The depth information acquiring unit is used to acquire depth information of a plurality of pixel points of an image area where the target object is located in the depth map. By performing image segmentation on the RGB image to accurately determine the location of the target object, depth information of the extraction target object can be accurately extracted.

In some embodiments, the device further includes a filtering module for filtering outliers in the initial 3D point cloud and using the filtered initial 3D point cloud as the second supervisory information. Through outlier filtering, the interference of outliers is mitigated, and furthermore, the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is collected and obtained by an image collecting device, and the parameters include a global rotation parameter of the target object, a key point rotation parameter of each key point of the target object, and a shape parameter of the target object. and a displacement parameter of the image acquisition device; The optimization module determines the current value of the displacement parameter of the image acquisition device and the initial value of the displacement parameter based on the supervision information and the initial value of the displacement parameter in a situation where the initial value of the figure parameter and the initial value of the key point rotation parameter do not change. a first optimization unit used to optimize an initial value of a global rotation parameter to obtain an optimized value of a displacement parameter and an optimized value of a global rotation parameter; Based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the shape parameter are optimized to obtain the optimized value of the key point rotation parameter and the optimized value of the shape parameter. and a second optimization unit used to obtain a value. In the optimization process, changing the position of the image acquisition device or changing the position of the 3D key point may cause a change in the 2D projection of the 3D key point, and thus the optimization process becomes unstable. By using a two-step optimization method, the initial values of the key point rotation parameters and the initial values of the shape parameters are fixed in advance to optimize the initial values of the displacement parameters and the initial values of the global rotation parameters of the image acquisition device. The initial values of the initial values and the global rotation parameters are fixed, and the initial values of the key point rotation parameters and the initial values of the shape parameters are optimized to improve the stability of the optimization process.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object; the first optimization unit obtains a target 2D projection key point belonging to a preset part of the target object from among the 2D projection key points corresponding to the 3D key points of the target object; Here, the 3D key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the shape parameter, and the 2D projection key point is the current value of the displacement parameter. Obtain by performing projection on the three-dimensional key points of the target object based on initial values of and global rotation parameters; obtain a first loss between the target two-dimensional projection key point and the initial two-dimensional key point; obtain a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; A current value of the displacement parameter and an initial value of the global rotation parameter are optimized based on the first loss and the second loss. The preset part may be a torso or the like, and since different motions have a small effect on key points of the torso, the first loss is determined using the key points of the torso to reduce the influence of different motions on key point positions. and improve the accuracy of optimization results. Since the two-dimensional key point is the supervision information of the two-dimensional plane, and the displacement parameter of the image acquisition device is the parameter of the three-dimensional plane, by obtaining the second loss, the optimization result falls to the local optimization point of the two-dimensional plane at the actual point. You can reduce the chances of getting out of the way.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object; the second optimization unit obtains a third loss between an optimized 2D projection key point of the target object and the initial 2D key point, wherein the optimized 2D projection key point corresponds to an optimized value of the displacement parameter and a global rotation; Obtained by performing projection on an optimized 3D key point of the target object based on the optimized value of the parameter, wherein the optimized 3D key point includes the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the shape. Obtained based on the initial value of the parameter -; obtain a fourth loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the posture parameter; An initial value of the key point rotation parameter and an initial value of the shape parameter are optimized based on the third loss and the fourth loss. This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter to improve the stability of the optimization process, and at the same time reduce the fourth loss. Through this, the rationality of the shape corresponding to the parameters after optimization is secured.

In some embodiments, the device optimizes the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and then the global rotation parameter. and a federated optimization module that performs coalesced optimization on the optimized value of the key point rotation parameter, the optimized value of the shape parameter, and the optimized value of the displacement parameter. This embodiment further improves the accuracy of the optimization result by performing coalition optimization on each parameter after optimization based on the above optimization.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object and an initial three-dimensional point cloud of a surface of the target object; The first optimization unit obtains a target 2D projection key point belonging to a preset part of the target object from among the 2D projection key points corresponding to the 3D key point of the target object - the 3D key point of the target object. is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the shape parameter, and the two-dimensional projection key point is based on the current value of the displacement parameter and the initial value of the global rotation parameter. Obtained by performing projection on the 3D key point of the target object -; obtain a first loss between the target two-dimensional projection key point and the initial two-dimensional key point; obtain a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; obtain a fifth loss between the first 3-dimensional point cloud of the surface of the target object and the initial 3-dimensional point cloud; the first three-dimensional point cloud is obtained based on the initial values of the global rotation parameters, the initial values of the key point rotation parameters and the initial values of the shape parameters; A current value of the displacement parameter and an initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss. This embodiment improves the accuracy of the optimization result by optimizing each initial parameter by adding a 3D point cloud to the supervisory information.

In some embodiments, the joint optimization module comprises: a first acquisition unit used to obtain a sixth loss between an optimized 2D projection key point of the target object and the initial 2D key point - the optimized 2D projection A key point is obtained by performing projection on an optimized 3D key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the optimized 3D key point of the global rotation parameter. Obtained based on the optimized value, the optimized value of the key point rotation parameter and the optimized value of the shape parameter -; A second acquisition unit used to obtain a seventh loss, wherein the seventh loss is the rationality of a posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter. Used to indicate -; a third acquisition unit used to obtain an eighth loss between a second 3-D point cloud of the target object surface and the initial 3-D point cloud, wherein the second 3-D point cloud is optimized for the global rotation parameter; values, obtained based on the optimized values of the key point rotation parameters and the optimized values of the shape parameters -; For the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter based on the sixth loss, the seventh loss and the eighth loss. Contains a coalition optimization unit used to perform coalition optimization. This embodiment improves the accuracy of the optimization result by optimizing each initial parameter by adding a 3D point cloud to the supervisory information.

According to a third aspect of the embodiments of the present invention, a three-dimensional reconstruction system is provided, wherein the system includes an image acquisition device and a processing unit. the image collection device is used to collect images of target objects; The processing unit is communicatively connected with the image acquisition device, and performs 3D reconstruction on the target object in the image through a 3D reconstruction network to obtain an initial value of a parameter of the target object - an initial value of the parameter. is used to build a 3D model of the target object; optimize an initial value of the parameter according to supervision information used for representing a target object feature to obtain an optimized value of the parameter; A skeletal skinning process is performed based on the optimized values of the parameters to build a 3D model of the target object.

According to a fourth aspect of the embodiments of the present invention, a computer readable storage medium in which a computer program is stored is provided, and the method according to any embodiment is implemented when the program is executed by a processor.

According to a fifth aspect of the embodiments of the present invention, there is provided a computer device, wherein the computer device includes a memory, a processor, and a computer program stored in the memory and executable by the processor. When the program is executed by a processor, the method according to any of the embodiments described above is implemented.

According to a sixth aspect of the embodiments of the present invention, there is provided a computer program product, wherein the computer program product is stored in a storage medium and includes a computer program executable by a processor, and the computer program is executed by the processor. In case a method according to any embodiment is implemented.

In an embodiment of the present invention, after acquiring initial values of parameters by performing 3D reconstruction on an image of a target object through a 3D reconstruction network, initial values of parameters are optimized based on supervision information, and parameter optimization is performed. A 3D model of the target object is built based on the optimized values of the parameters obtained through The advantage of the parameter optimization method is that it is more accurate and can provide a 3D reconstruction result that matches the 2D observation characteristics of the image, but generally provides unnatural and unreasonable operation results, resulting in low reliability. Since network regression through a 3D reconstruction network can provide more natural and reasonable operating results, the output of the 3D reconstruction network is used as the initial value of the parameter to be optimized to secure the reliability of the 3D reconstruction result and at the same time 3D reconstruction accuracy can also be taken into account.

It is to be understood that the foregoing general description and the following detailed description are illustrative and explanatory only and do not limit the present disclosure.

The accompanying drawings herein are included in the specification as part of the specification, and these accompanying drawings illustrate embodiments corresponding to the embodiments of the present invention, and describe technical solutions of the present invention together with the specification.
1A and 1B are schematic diagrams of three-dimensional models of some embodiments.
2 is a flowchart of a 3D reconstruction method according to an embodiment of the present invention.
3 is an overall flowchart of an embodiment of the present invention.
4A and 4B are respectively schematic diagrams of application scenarios of an embodiment of the present invention.
5 is a block diagram of a 3D reconstruction device according to an embodiment of the present invention.
6 is a schematic diagram of a three-dimensional reconstruction system of an embodiment of the present invention.
7 is a structural schematic diagram of a computer device in an embodiment of the present invention.

Examples will be described in detail herein, examples of which are shown in the drawings. When the following description includes drawings, like numbers in different drawings refer to the same or similar elements unless otherwise indicated. The embodiments described in the examples below do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods as recited in the appended claims and consistent with some aspects of the present disclosure.

The terms used in this disclosure are only for describing specific examples and are not intended to limit the disclosure. Terms in the singular form ("a", "the" and "said") in this disclosure and the appended claims are also intended to include the plural form unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein includes any or all possible combinations of one or more of the associated listed items. Also, as used herein, the term "at least one" refers to any one of several or any combination of at least two of several.

Although various information may be described using terms such as "first", "second", and "third" in the present disclosure, it should be understood that such information should not be limited to these terms. These terms are only intended to distinguish information of the same type from each other. For example, without departing from the scope of the present disclosure, first information may be referred to as second information, and similarly, second information may be referred to as first information. Also, depending on the context, the word "if" as used herein may be interpreted as "when" or "upon" or "in response to a decision". can

In order to help those skilled in the art to better understand the technical solutions according to the embodiments of the present invention, and to make the objects, features, and advantages of the embodiments of the present disclosure clearer and easier to understand, the drawings are attached below. Thus, the technical solution according to the embodiment of the present disclosure will be described in more detail.

3D reconstruction of the target object requires reconstruction of the posture and rotation of the limbs of the target object, and generally uses parametric models as well as 3D key points to represent the posture and rotation of the limbs of the target object. For example, 3D reconstructions for different people reconstruct a 3D model of a lean person (shown in FIG. 1A ) and a 3D model of a fat person (shown in FIG. 1B ) respectively, but FIG. 1A Since the key point information in the same posture is the same for the person shown in and the person shown in FIG.

Related technologies generally perform 3D reconstruction through two methods: parameter optimization and network regression. The parameter optimization method generally selects a series of standard parameters and iteratively optimizes the initial values of the parameters of the 3D model according to the two-dimensional visual characteristics of the image of the target object using gradient descent, wherein the two-dimensional visual characteristics of the image As for the feature, a two-dimensional key point or the like can be selected. The advantage of the parameter optimization method is that it is more accurate and can provide parameter estimation results that conform to the two-dimensional visual characteristics of the image, but it often provides unnatural and irrational operating results, and the final performance of parameter optimization is the initial value of the parameter. , so the reliability of the 3D reconstruction method based on parameter optimization is low.

Network regression methods typically train an end-to-end neural network to learn the mapping from images to 3D model parameters. The advantage of the network regression method is that more natural and reasonable motion results can be obtained, but the amount of training data is not large, so the 3D reconstruction result may not match the 2D visual features of the image. The accuracy of the dimensional reconstruction method is low. Therefore, the conventional 3D reconstruction method cannot simultaneously consider the accuracy and reliability of the 3D reconstruction result.

In view of the above, an embodiment of the present invention provides a 3D reconstruction method, as shown in FIG. 2 , the method includes the following steps:

Step 201: Perform 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain initial values of parameters of the target object, where the initial values of the parameters are used to build a 3D model of the target object. used for;

Step 202: Optimizing initial values of the parameters according to previously obtained supervision information representing characteristics of the target object to obtain optimized values of the parameters;

Step 203: A skeletal skinning process is performed according to the optimized values of the parameters to build a three-dimensional model of the target object.

In step 201, the target object may be a 3D object such as a person, animal, or robot in a physical space, or may be a region of one or more of the 3D objects, such as a face or limb of a person. For convenience of description, a person is used as a target object, and 3D reconstruction of the target object is human body reconstruction as an example. The image of the target object may be one image or may include a plurality of images obtained by photographing the target object from a plurality of different viewing angles. 3D human body reconstruction based on one image is referred to as monocular 3D human body reconstruction, and 3D human body reconstruction of a plurality of images of different viewing angles is referred to as multiocular 3D human body reconstruction. Each image can be a grayscale map, RGB image or RGBD image. The image may be an image captured in real time by an image capture device (eg, a camera) around the target object, or may be a pre-photographed and stored image.

를 사용하여 자태 파라미터의 초기값 및 자세 파라미터의 초기값을 인체 표면의 3차원 모델에 매핑할 수 있는 바, 당해 3차원 모델은 6890개의 정점을 포함하고, 정점 사이는 고정된 연결 관계를 통해 삼각 패치(Triangular patch)를 형성한다. 미리 훈련된 회귀자(Regressor) W를 사용하여 인체 표면 모델의 정점으로부터 인체의 3차원 키 포인트

를 추가로 회귀할 수 있다.A 3D reconstruction may be performed on an image of an object through a 3D reconstruction network, where the 3D reconstruction network may be a pretrained neural network. The 3D reconstruction network can perform 3D reconstruction based on the image, and can also estimate natural and reasonable initial values of parameters, where the initial values of the parameters can be represented by a vector, and the vector The dimension is, for example, 85 dimensions, and the motion limb rotation information of the human body (that is, the initial value of the posture parameter, the initial value of the global rotation parameter of the human body and the initial value of the key point rotation parameter including 23 key points) is added to the vector. ), the initial value of the self-portrait parameter, and the initial value of the camera parameter. The human body can be represented by key points and the limb bones connecting these key points. The key points of the human body are the crown, nose, neck, left eye and right eye, left ear and right ear, chest, left shoulder and right shoulder, left elbow It can contain one or more of the following key points: right elbow, left and right wrist, left and right hip, left and right hip, left and right knee, left and right ankle, and the initial value of the posture parameter is the key point of the human body. It is used to determine position in 3D space. The initial values of the figure parameters are used to determine body information such as height, fatness, and thinness of the human body. The initial values of the camera parameters are used to determine the absolute position of the camera coordinate system of the human body in a 3D space, and the parameters of the camera include a displacement parameter between the camera and the human body and a posture parameter of the camera. The initial values of the posture parameters may be replaced with the initial values of global rotation parameters of the human body. Human body parameters can be expressed using a parameter format (referred to as SMPL parameters) of a Skinned Multi-Person Linear (SMPL) model. After obtaining the SMPL parameter value, based on the SMPL parameter value, skeleton skinning process is performed, that is, one mapping function

The initial values of the posture parameters and the initial values of the posture parameters can be mapped to a 3-dimensional model of the human body surface using . It forms a triangular patch. 3D key points of the human body from the vertices of the human body surface model using the pretrained regressor W

can be further regressed.

In step 202, the supervision information is at least one of two-dimensional visual characteristics (also referred to as two-dimensional observation characteristics), for example, two-dimensional key points of a target object in an image and semantic information of a plurality of pixel points of the target object. can Semantic information of one pixel point is used to indicate which region of the target object the pixel point is located in, and the region may be, for example, a region where a head, arm, torso, or leg is located. In the case of using 2D key point information as supervision information, a 2D key point extraction network can be used to estimate the location of human body key points in the image, where an arbitrary 2D self-image estimation method such as OpenPose is selected. and can be used. In addition to using the 2D visual features as supervision information, the accuracy of 3D reconstruction can be further improved by using the 2D visual features and the initial 3D point cloud of the target object surface together as supervision information.

When the image includes a depth map (for example, the image is an RGBD image), depth information of a plurality of pixel points of the target object is extracted from the depth map, and based on the depth information, the depth information in the depth map is extracted. An initial 3D point cloud of a surface of the target object may be obtained by projecting a plurality of pixel points of the target object onto a 3D space.

The plurality of pixel points may be some or all pixel points of the target object in the image. For example, the target object may include pixel points in each area requiring 3D reconstruction, and the number of pixel points in each area may be greater than or equal to the number requiring 3D reconstruction.

In general, since an image includes a target object and a background area, image segmentation is performed on an RGB image included in the image to obtain an image area where the target object in the RGB image is located, and the target object in the RGB image is located. determining an image area where a target object is located in the depth map based on an image area where the object is located; Depth information of a plurality of pixel points of an image area in which the target object is located in the depth map may be obtained. Through image segmentation, an image region in which a target object requiring 3D reconstruction is located is extracted from an image, thereby avoiding the influence of the 3D reconstruction of the background region in the image. In some embodiments, there is a one-to-one correspondence between pixel points in the depth map and pixel points in the RGB image. For example, the image may be an RGBD image.

Furthermore, outliers may be filtered from the 3D point cloud (ie, the initial 3D point cloud), and the supervision information may include the 3D point cloud after filtering. The filtering may be implemented as a point cloud filter. By filtering outliers, a more precise 3D point cloud of the target object surface can be obtained, further improving the accuracy of 3D reconstruction. For each target 3D point in the 3D point cloud, an average distance from n 3D points closest to the target 3D point to the target 3D point is obtained, and the average distance corresponding to each target 3D point is Assuming that one statistical distribution (eg, Gaussian distribution) is followed, the average value and variance of the statistical distribution can be calculated, and when one threshold value s is set based on the average value and variance, the average distance 3D points outside the range of the threshold value s can be regarded as outliers, and filtered in the 3D point cloud.

In practical applications, when the image is an RGB image, the initial values of the parameters may be repeatedly optimized using the two-dimensional observation characteristics as supervision information. If the image is an RGBD image, the initial values of the parameters may be optimized by using both the 2D observation features and the 3D point cloud of the surface of the target object as supervision information. The optimization method may use, for example, a gradient descent method, but the present invention is not limited thereto.

In step 203, a skeletal skinning process is performed according to the optimized values of the parameters to obtain a 3D model of the target object.

As shown in Fig. 3, this is the overall flow chart of an embodiment of the present invention. If the input is an RGB image, 3D reconstruction is performed on the RGB image through a 3D reconstruction network to obtain human body parameter values of the person in the image, and key points are extracted for the person in the image using the keypoint extraction network. to obtain the 2D key points of the human body. Use the following human body parameter values as the initial values of the parameters, use the two-dimensional key points of the human body as supervision information, optimize the initial values of the human body parameters through the parameter optimization module to obtain optimized values of the human body parameters, Based on the optimized values, skeletal skinning is performed to obtain a human body reconstruction model.

If the input is an RGBD image, the image can be decomposed into an RGB image and a Time of Flight (TOF) depth map, and the TOF depth map includes depth information of each pixel point in the RGB image. By performing 3D reconstruction on the RGB image through a 3D reconstruction network, the human body parameter values of the person in the image can be obtained, and by performing key point extraction on the person in the image using the key point extraction network, the 2D key of the human body can be obtained. points can be earned. A point cloud reconstruction module may be used to reconstruct a human body surface point cloud based on depth information in a TOF depth map. The following human body parameter values are used as the initial values of the parameters, the human body 2-dimensional key points and the human body surface point cloud are used as supervision information, and the initial values of the human body parameters are optimized through the parameter optimization module to obtain the optimized values of the human body parameters. skeletal skinning process is performed based on the optimized values of human body parameters to obtain a human body reconstruction model.

Furthermore, after obtaining the human body reconstruction model, color processing may be performed on the human body reconstruction model based on color information in the RGB image or RGBD image so that the human body reconstruction model and the color information of the person in the image match.

In an embodiment of the present invention, 3D reconstruction is performed on a target object in an image through a 3D reconstruction network to obtain initial values of parameters, and the initial values of the parameters are optimized again based on supervision information, and the parameters A 3D model of the target object is built based on the optimized values. The advantage of the parameter optimization method is that it can provide a 3D reconstruction result that is accurate and conforms to the 2D image observation characteristics, but often provides unnatural and unreasonable operation results, and reliability is low. Since network regression through a 3D reconstruction network can provide more natural and reasonable operating results, if parameter optimization is performed using the output result of the 3D reconstruction network as the initial value of a parameter, the reliability of the 3D reconstruction result can be secured. and the accuracy of 3D reconstruction can be considered at the same time.

In some embodiments, a multi-step optimization method may be used in the parameter optimization step. The multi-step optimization method may include a camera optimization step and a shape optimization step. In the camera optimization step, an optimization target is a value R of a global rotation parameter and a current value t of a displacement parameter between the image acquisition device and the target object. Here, t and R are both 3-dimensional vectors, and R is expressed in the form of an axis angle. In the shape optimization step, the optimization target is the value of the key point rotation parameter and the value of the shape parameter.

In the optimization process, changing the position of the camera or changing the position of the 3D key point on the human body both causes a change in the 2D projection of the 3D key point, which causes instability in the optimization process. Therefore, in the camera optimization step, the body shape is fixed, and in the shape optimization step, the camera position is fixed to improve the stability of the optimization process. That is, in a situation where the initial values of the figure parameters and the initial values of the key point rotation parameters do not change, the current values of the displacement parameters and the global rotation parameters of the image acquisition device are based on the supervision information and the initial values of the displacement parameters. optimize the initial value of to obtain an optimized value of the displacement parameter and an optimized value of the global rotation parameter; The optimized value of the next displacement parameter and the optimized value of the global rotation parameter are kept unchanged, and the initial value of the key point rotation parameter and the configuration are based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter. The initial values of the parameters are optimized to obtain the optimized values of the key point rotation parameters and the optimized values of the shape parameters.

Furthermore, among the 2D projection key points corresponding to the 3D key points of the target object, a target 2D projection key point belonging to a preset portion of the target object may be acquired; Here, the three-dimensional key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the shape parameter; The 2D projection key point is obtained by performing projection on the 3D key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter. A first loss between the target 2D projection key point and the initial 2D key point is obtained. A second loss between an initial value of the displacement parameter and a current value of the displacement parameter is obtained. A current value of the displacement parameter and an initial value of the global rotation parameter are optimized based on the first loss and the second loss.

Here, the preset part may be a body part, and the target 2D projection key point may include key points such as left and right shoulder points, left and right hip joint points, and a central point of the spine. Since the influence on the key points of the torso of different motions is small, by setting the first loss using the key points of the torso, the influence on the position of the key points of the different motions can be mitigated and the accuracy of the optimization result can be improved. can The first loss can also be referred to as the torso key point projection loss, the second loss can also be referred to as the camera displacement normalization loss, the first loss can be obtained through Equation (1), and the second loss is It can be obtained through Equation (2):

와

는 각각 제1 손실과 제2 손실을 표시하고,

와

는 각각 목표 2차원 투영 키 포인트와 초기 2차원 키 포인트를 표시하며,

와

는 각각 상기 이미지 수집 장치와 상기 목표 오브젝트 사이의 변위 파라미터의 현재 값과 상기 변위 파라미터의 초기값을 표시한다. 제1 손실과 제2 손실에 기반하여 제1 목표 손실을 결정할 수 있다. 예를 들어, 상기 제1 목표 손실은 상기 제1 손실과 상기 제2 손실의 합으로 결정할 수 있고 다음 수학식(3)을 통해 결정할 수 있다:here

and

Represents the first loss and the second loss, respectively,

and

denotes a target 2D projection key point and an initial 2D key point, respectively,

and

Indicates a current value of a displacement parameter between the image acquisition device and the target object and an initial value of the displacement parameter, respectively. A first target loss may be determined based on the first loss and the second loss. For example, the first target loss may be determined as the sum of the first loss and the second loss and may be determined through Equation (3):

A third loss between an optimized 2D projection key point of the target object and the initial 2D key point may be obtained, wherein the optimized 2D projection key point is a ratio between an optimized value of the displacement parameter and a global rotation parameter. It is obtained by performing projection on an optimized 3D key point of the target object based on the optimized value, and the optimized 3D key point is obtained by performing a projection on the optimized value of the global rotation parameter, an initial value of a key point rotation parameter, and a shape parameter. Get it based on the initial value. A fourth loss used for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter is obtained. An initial value of the key point rotation parameter and an initial value of the shape parameter are optimized based on the third loss and the fourth loss.

The third loss may also be referred to as the two-dimensional key point projection loss, the fourth loss may also be referred to as the self-rationality loss, and the third loss may be determined through Equation (4):

는 제3 손실이고,

와

는 각각 상기 최적화 2차원 투영 키 포인트 및 상기 초기 2차원 키 포인트를 표시한다. 제3 손실과 제4 손실에 기반하여 제2 목표 손실을 결정할 수 있는바, 예를 들어, 상기 제2 목표 손실은 상기 제3 손실과 상기 제4 손실의 합으로 결정될 수 있고, 다음 수학식(5)을 통해 결정된다:here,

is the third loss,

and

denote the optimization two-dimensional projection key point and the initial two-dimensional key point, respectively. A second target loss may be determined based on the third loss and the fourth loss. For example, the second target loss may be determined as the sum of the third loss and the fourth loss, and the following equation ( 5) is determined via:

는 제2 목표 손실이고,

는 제4 손실이고, 가우스 혼합 모델(Gaussian Mixture Model, GMM)을 사용하여 얻을 수 있으며, 전역 회전 파라미터의 최적화된 값, 키 포인트 회전 파라미터의 초기값 및 자태 파라미터의 초기값에 대응하는 자태가 합리한지 여부를 판단하고, 불합리한 자태에 대해 큰 손실을 수출하는 데에 사용된다.here,

is the second target loss,

is the fourth loss, and can be obtained using a Gaussian Mixture Model (GMM), and the shape corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the shape parameter is reasonable It is used to determine whether or not to do so, and to export large losses for unreasonable behavior.

After optimizing the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the optimized value of the global rotation parameter, the key point Combined optimization is performed on the optimized values of the rotation parameters, the optimized values of the shape parameters, and the optimized values of the displacement parameters, that is, a three-step optimization method may be used. When the supervision information includes information on the 3D point cloud of the surface of the target object, the three-step optimization method including a camera optimization step, a self-optimization step, and a point cloud optimization step may be used.

In the camera optimization step, among the 2D projection key points corresponding to the 3D key points of the target object, a target 2D projection key point belonging to a preset part of the target object is obtained; Here, the 3D key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the shape parameter, and the 2D projection key point is the current value of the displacement parameter. It is obtained by performing projection on the 3D key point of the target object based on the initial values of and global rotation parameters. A first loss between the target 2D projection key point and the initial 2D key point is obtained. A second loss between an initial value of the displacement parameter and a current value of the displacement parameter is obtained. obtain a fifth loss between the first 3-dimensional point cloud of the surface of the target object and the initial 3-dimensional point cloud; Here, the first 3D point cloud is obtained based on the initial values of the global rotation parameters, the initial values of the key point rotation parameters, and the initial values of the shape parameters. A current value of the displacement parameter and an initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss. The fifth loss may also be referred to as an Iterative Closest Point (ICP) point cloud registration loss, and may be determined through Equation (6):

는 상기 제5 손실이고, 상기 초기 3차원 포인트 클라우드를 포인트 클라우드 P로 간주하며, 상기 제1 3차원 포인트 클라우드를 포인트 클라우드 Q로 간주하고,

는 포인트 클라우드 P 중 각 포인트로부터 포인트 클라우드 Q까지의 거리가 가장 가까운 포인트들로 구성된 포인트 쌍(point pair) 집합이며,

는 포인트 클라우드 Q 중 매개 포인트로부터 포인트 클라우드 P까지의 거리가 가장 가까운 포인트들로 구성된 포인트 쌍 집합이다. 제1 손실과 제2 손실은 각각 다음 수학식(7)과 수학식(8)으로 표시된다:in the equation

is the fifth loss, the initial 3-dimensional point cloud is regarded as a point cloud P, the first 3-dimensional point cloud is regarded as a point cloud Q,

Is a set of point pairs consisting of points with the closest distance from each point in the point cloud P to the point cloud Q,

is a set of point pairs consisting of points with the closest distance from each point in the point cloud Q to the point cloud P. The first loss and the second loss are represented by the following equations (7) and (8), respectively:

와

는 각각 제1 손실과 제2 손실을 표시하고,

와

는 각각 목표 2차원 투영 키 포인트와 초기 2차원 키 포인트를 표시하며,

와

는 각각 상기 변위 파라미터의 현재 값과 상기 변위 파라미터의 초기값을 표시한다. 제1 손실, 제2 손실 및 제5 손실의 합에 기반하여 제1 목표 손실을 결정하고, 다시 제1 목표 손실에 기반하여 상기 변위 파라미터의 현재 값과 전역 회전 파라미터의 초기값을 최적화하는 바, 즉 다음 수학식(9)이다:here

and

Represents the first loss and the second loss, respectively,

and

denotes a target 2D projection key point and an initial 2D key point, respectively,

and

denotes the current value of the displacement parameter and the initial value of the displacement parameter, respectively. Determine a first target loss based on the sum of the first loss, the second loss and the fifth loss, and optimize the current value of the displacement parameter and the initial value of the global rotation parameter based on the first target loss again, That is, the following equation (9):

Since the optimization method of the phase optimization step of the 3-step optimization process and the phase optimization step of the 2-step optimization process are the same, they are not further described here.

In the point cloud optimization step, a sixth loss between an optimized 2D projection key point of the target object and the initial 2D key point is obtained, wherein the optimized 2D projection key point is equal to the optimized value of the displacement parameter. Obtained by performing projection on an optimized 3D key point of the target object based on an optimized value of a global rotation parameter, wherein the optimized 3D key point is obtained by performing a projection on an optimized value of the global rotation parameter and an optimized key point rotation parameter. It is obtained based on the optimized value of the value and aspect parameters. A seventh loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter is obtained. obtain an eighth loss between a second 3-dimensional point cloud of the target object surface and the initial 3-dimensional point cloud; Here, the second 3D point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the shape parameter. For the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter based on the sixth loss, the seventh loss and the eighth loss. Perform federated optimization, and optimize through the following equations (10) and (11):

는 제6 손실이고,

는 최적화 2차원 투영 키 포인트이며,

는 초기 2차원 키 포인트다. 제7 손실은 가우스 혼합 모델을 사용하여 획득할 수 있고, 이는 전역 회전 파라미터의 최적화된 값, 키 포인트 회전 파라미터의 최적화된 값 및 자태 파라미터의 최적화된 값에 대응하는 자태가 합리한지 여부를 판단하는 데에 사용되며, 불합리한 자태에 대해 큰 손실을 출력한다.

는 제8 손실이고, 상기 초기 3차원 포인트 클라우드를 포인트 클라우드 P로 간주하며,

는 상기 제2 3차원 포인트 클라우드이고,

는 포인트 클라우드 P 중 각 포인트로부터 포인트 클라우드

까지의 거리가 가장 가까운 포인트들로 구성된 포인트 쌍 집합이며,

는 포인트 클라우드

중 각 포인트로부터 포인트 클라우드 P까지의 거리가 가장 가까운 포인트들로 구성된 포인트 쌍 집합이다. 나아가, 제6 손실, 제7 손실 및 제8 손실의 합을 제3 목표 손실

로 결정할 수 있고, 제3 목표 손실에 기반하여 상기 전역 회전 파라미터의 최적화된 값, 상기 키 포인트 회전 파라미터의 최적화된 값, 자태 파라미터의 최적화된 값 및 상기 변위 파라미터의 최적화된 값에 대해 연합 최적화를 수행하며, 다음 수학식(12)을 통해 연합 최적화를 수행할 수 있다:in the equation

is the sixth loss,

is the optimization two-dimensional projection key point,

is an initial two-dimensional key point. The seventh loss can be obtained using a Gaussian mixture model, which determines whether the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the shape corresponding to the optimized value of the shape parameter are reasonable. It is used for, and outputs a large loss for unreasonable self.

is the eighth loss, and regards the initial three-dimensional point cloud as a point cloud P,

Is the second 3-dimensional point cloud,

is the point cloud from each point in the point cloud P

is a set of point pairs consisting of points with the closest distance to

is the point cloud

It is a set of point pairs consisting of points with the closest distance from each point to the point cloud P. Further, the third target loss is the sum of the sixth loss, the seventh loss, and the eighth loss.

, and based on the third target loss, joint optimization is performed for the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter, and the optimized value of the displacement parameter. , and federation optimization can be performed through the following equation (12):

If the image of the target object is an RGB image, parameter optimization may be performed based on the above-mentioned two-step optimization method including the camera optimization step and the shape optimization step; When the image of the target object is an RGBD image, parameter optimization may be performed based on the above-mentioned three-step optimization method including the camera optimization step, the self-optimization step, and the point cloud optimization step.

This method can be used in various scenarios, and can provide a natural, reasonable, and accurate human body reconstruction model in scenarios such as virtual dressing rooms, virtual anchors, and video motion migration.

As shown in Fig. 4A, this is a schematic diagram of a virtual dressing room application scenario in an embodiment of the present invention. A human body reconstruction model 404 corresponding to the user 401 may be obtained by collecting images of the user 401 through the camera 403 and transmitting the collected images to a processor (not shown) to perform 3D human body reconstruction. is acquired, and the human body reconstruction model 404 is displayed on the display interface 402 so that the user 401 can see it. At the same time, the user 401 can select desired clothing 405, and the clothing 405 includes, but is not limited to, the clothing 4051 and a hat 4052. The clothing 405 is displayed on the display interface 402 based on the human body reconstruction model 404 so that the user 401 can see the wearing effect of the clothing 405 .

As shown in Fig. 4B, this is a schematic diagram of a virtual live room application scenario of an embodiment of the present invention. In the process of live broadcasting, the anchor user 406's image can be collected through the anchor client 407, the anchor user's 406's image is transmitted to the server 408 to perform 3D reconstruction, and the anchor user's human body is performed. Obtain a reconstruction model, i.e. a virtual anchor. The server 408 can return the anchor user's human body reconstruction model to the anchor client 407 for display, as shown in the model 4071 in the drawing. In addition, the anchor client 407 may collect voice information of the anchor user and transmit the voice information to the server 408 so that the server 408 performs convergence on the human body reconstruction model and the voice information. The server 408 transmits the human body reconstruction model and voice information after fusion to the viewer client 409 watching the live broadcast program so that they can be displayed and reproduced. Here, the displayed human body reconstruction model corresponds to the model 4091 in the drawing. As shown. Through the above method, a live broadcasting screen of the virtual anchor can be displayed on the viewer client 409 .

A person skilled in the art does not imply that, in the described method of a particular implementation, the order in which the drafting of each step is strictly executed forms any limitation to the implementation process, and the order of drafting each step It can be appreciated that the specific order of execution should be determined by its functionality and possibly its own logic.

As shown in FIG. 5, the present invention further provides a 3D reconstruction device, which includes a first 3D reconstruction module 501, an optimization module 502, and a second 3D reconstruction module 503. include

The first 3D reconstruction module 501 is used to obtain initial values of parameters of the target object by performing 3D reconstruction on a target object in an image through a 3D reconstruction network, and the initial values of the parameters are used to build a three-dimensional model of a target object;

the optimization module 502 is used to optimize the initial value of the parameter according to previously obtained supervision information representing the characteristics of the target object to obtain an optimized value of the parameter;

The second 3D reconstruction module 503 is used to construct a 3D model of the target object by performing skeletal skinning processing based on the optimized values of the parameters.

In some embodiments, the supervisory information includes first supervisory information, or the supervisory information includes first supervisory information and second supervisory information; the first supervision information includes at least one of an initial two-dimensional key point of the target object and semantic information of a plurality of pixel points of the target object in the image; The second supervision information includes an initial 3D point cloud of the surface of the target object. In an embodiment of the present invention, only semantic information of an initial two-dimensional key point or pixel point of a target object is used as supervision information to optimize the initial values of the parameters so that optimization efficiency is high and optimization complexity is low; Alternatively, the initial 3D point cloud of the target object surface and the aforementioned initial 2D key point or pixel point semantic information are used together as supervision information to improve the accuracy of the optimized value of the acquired parameter.

In some embodiments, the device further includes a 2D key point extraction module, which is used to extract information of an initial 2D key point of the target object from the image via a key point extraction network. A natural and reasonable motion can be created for a 3D model by using the initial 2D key point information extracted by the key point extraction network as supervision information.

In some embodiments, the image includes a depth map of the target object; The device further includes a depth information extraction module and a back-projection module. the depth information extraction module is used to extract depth information of a plurality of pixel points of the target object from the depth map; The back-projection module is used to back-project a plurality of pixel points of the target object in the depth map into a 3-dimensional space based on the depth information to obtain an initial 3-dimensional point cloud of a surface of the target object. Depth information is extracted, and based on the depth information, pixel points in the 2D image are back-projected onto a 3D space to obtain an initial 3D point cloud of the surface of the target object, and the initial 3D point cloud is used as supervision information. Optimize the initial values of parameters, and further improve the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; The depth information extraction module includes an image segmentation unit, an image area determination unit, and a depth information acquisition unit. the image segmentation unit is used to perform image segmentation on the RGB image; The image area determination unit determines an image area where a target object in the RGB image is located based on a result of image segmentation, and a target object in the depth map is determined based on the image area where the target object in the RGB image is located. used to determine the region of the image to be located; The depth information acquisition unit is used to acquire depth information of a plurality of pixel points of an image area where the target object in the depth map is located. By performing image segmentation on the RGB image to accurately determine the location of the target object, depth information of the extraction target object can be accurately extracted.

In some embodiments, the device further includes a filtering module used for filtering outliers in the initial 3D point cloud and using the filtered initial 3D point cloud as the second supervisory information. Through outlier filtering, the interference of outliers is mitigated, and furthermore, the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is collected and obtained by an image collecting device, and the parameters include a global rotation parameter of the target object, a key point rotation parameter of each key point of the target object, and a shape parameter of the target object. and a displacement parameter of the image acquisition device; The optimization module includes a first optimization unit and a second optimization unit. The first optimization unit determines the current value of the displacement parameter of the image capture device and used to optimize an initial value of the global rotation parameter to obtain an optimized value of a displacement parameter and an optimized value of a global rotation parameter; The second optimization unit optimizes the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter to optimize the value of the key point rotation parameter. It is used to obtain the optimized values of the parameters of the state and the state. In the optimization process, changing the position of the image acquisition device or changing the position of the 3D key point may cause a change in the 2D projection of the 3D key point, and thus the optimization process becomes unstable. By using a two-step optimization method, the initial values of the key point rotation parameters and the initial values of the shape parameters are fixed in advance to optimize the initial values of the displacement parameters and the initial values of the global rotation parameters of the image acquisition device. The initial values of the initial values and the global rotation parameters are fixed, and the initial values of the key point rotation parameters and the initial values of the shape parameters are optimized to improve the stability of the optimization process.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object; the first optimization unit is used to obtain a target 2D projection key point belonging to a preset part of the target object from among the 2D projection key points corresponding to the 3D key points of the target object; Here, the 3D key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the shape parameter, and the 2D projection key point is the current value of the displacement parameter. Obtain by performing projection on the three-dimensional key points of the target object based on initial values of and global rotation parameters; obtain a first loss between the target two-dimensional projection key point and the initial two-dimensional key point; obtain a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; A current value of the displacement parameter and an initial value of the global rotation parameter are optimized based on the first loss and the second loss. The preset part may be a torso or the like, and since different motions have a small effect on key points of the torso, the first loss is determined using the key points of the torso to reduce the influence of different motions on key point positions. and improve the accuracy of optimization results. Since the two-dimensional key point is the supervision information of the two-dimensional plane, and the displacement parameter of the image acquisition device is the parameter of the three-dimensional plane, by obtaining the second loss, the optimization result falls to the local optimization point of the two-dimensional plane at the actual point. You can reduce the chances of getting out of the way.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object; The second optimization unit is used to obtain a third loss between an optimized 2D projection key point of the target object and the initial 2D key point, and the optimized 2D projection key point has an optimized value of the displacement parameter and Obtained by performing projection on an optimized 3D key point of the target object based on an optimized value of a global rotation parameter, wherein the optimized 3D key point is the optimized value of the global rotation parameter and the initial value of the key point rotation parameter. and based on the initial value of the shape parameter; obtain a fourth loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the posture parameter; An initial value of the key point rotation parameter and an initial value of the shape parameter are optimized based on the third loss and the fourth loss. This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter to improve the stability of the optimization process, and at the same time reduce the fourth loss. Through this, the rationality of the shape corresponding to the parameters after optimization is secured.

In some embodiments, the device optimizes the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and then the global rotation parameter. and a federated optimization module for performing federated optimization on the optimized value, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter. This embodiment further improves the accuracy of the optimization result by performing coalition optimization on each parameter after optimization based on the above optimization.

In some embodiments, the supervision information includes an initial two-dimensional key point of the target object and an initial three-dimensional point cloud of a surface of the target object; The first optimization unit is used to obtain a target 2D projection key point belonging to a preset part of the target object among 2D projection key points corresponding to the 3D key point of the target object; A 3D key point of an object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the shape parameter, and the 2D projection key point is the current value of the displacement parameter and the global rotation parameter. obtained by performing projection on the 3D key point of the target object based on an initial value of ; obtain a first loss between the target two-dimensional projection key point and the initial two-dimensional key point; obtain a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; obtain a fifth loss between the first 3-dimensional point cloud of the surface of the target object and the initial 3-dimensional point cloud; the first three-dimensional point cloud is obtained based on the initial values of the global rotation parameters, the initial values of the key point rotation parameters and the initial values of the shape parameters; A current value of the displacement parameter and an initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss. This embodiment improves the accuracy of the optimization result by optimizing each initial parameter by adding a 3D point cloud to the supervisory information.

In some embodiments, the coalitional optimization module includes a first acquisition unit, a second acquisition unit, a third acquisition unit, and a coalitional optimization unit. the first obtaining unit is used to obtain a sixth loss between an optimization two-dimensional projection key point of the target object and the initial two-dimensional key point; The optimized 2D projection key point is obtained by performing projection on the optimized 3D key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized 3D key point is obtain based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the shape parameter; The second obtaining unit is used to obtain a seventh loss, the seventh loss corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the aspect parameter. used to indicate the rationality of; the third obtaining unit is used to acquire an eighth loss between a second 3-dimensional point cloud of the target object surface and the initial 3-dimensional point cloud; the second three-dimensional point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the shape parameter; The joint optimization unit determines the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the displacement parameter based on the sixth loss, the seventh loss and the eighth loss. Used to perform federated optimization on optimized values. This embodiment improves the accuracy of the optimization result by optimizing each initial parameter by adding a 3D point cloud to the supervisory information.

In some embodiments, functions provided by devices provided by embodiments of the present invention or modules included may be used to execute the methods described in the above method embodiments, and for specific implementations thereof, refer to the description of the above method embodiments. As far as I can, for the sake of brevity, I won't explain it further here.

As shown in FIG. 6 , the present invention further provides a 3D reconstruction system, which includes an image acquisition device 601 and a processing unit 602 .

The image collecting device 601 is used to collect an image of a target object.

A processing unit 602 communicatively connected to the image acquisition device 601 is used to obtain initial values of parameters of the target object by performing 3D reconstruction on a target object in the image through a 3D reconstruction network, the initial values of the parameters are used to build a three-dimensional model of the target object; optimize an initial value of the parameter according to previously obtained supervision information representing characteristics of a target object to obtain an optimized value of the parameter; A skeletal skinning process is performed based on the optimized values of the parameters to build a 3D model of the target object.

The image collection device 601 of the embodiment of the present invention may be a device having an image collection function, such as a camera or a camera, and the image collected by the image collection device 601 is transmitted to the processing unit 602 in real time, or After being stored, it can be transmitted from the storage space to the processing unit 602 when needed. The processing unit 602 may be one server or a server cluster composed of a plurality of servers. Details of the method performed by the processing unit 602 may refer to the embodiment of the 3D reconstruction method described above and are not described herein any further.

Embodiments of the present specification further provide a computer device, wherein the computer device includes at least a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein, when the program is executed by the processor, any one of the above A method according to an embodiment is implemented.

7 is a schematic diagram of a hardware structure of a more specific computing device provided by an embodiment of the present specification, which includes a processor 701, a memory 702, an input/output interface 703, a communication interface 704, and bus 705. Here, the processor 701, the memory 702, the input/output interface 703, and the communication interface 704 implement a communication connection inside the device through a bus 705.

The processor 701 may be implemented through a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, which is described herein by implementing a related program. The technical solution provided by the embodiment of is implemented. The processor 701 may further include a graphic card, and the graphic card may be an Nvidia titan X graphic card or a 1080Ti graphic card.

The memory 702 may be implemented in the form of a read only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, and the like. The memory 902 may store an operating system and other application programs, and when implementing the technical solutions provided by the embodiments of the present specification through software or firmware, related program codes are stored in the memory 702, It is called and executed by the processor 701.

The input/output interface 703 is used to connect input/output modules to implement input/output of information. The input/output/module may be installed in the device as a component (not shown in the drawing) or may be externally connected to the device to provide corresponding functions. An input device may include a keyboard, a mouse, a touch screen, a microphone, and various sensors, and an output device may include a display, a speaker, a vibrator, and indicators.

The communication interface 704 is used to connect communication modules (not shown) to realize communication interaction between this device and other devices. The communication module may implement communication through wired means (eg, USB, network cable, etc.) or through wireless means (eg, mobile network, WIFI, Bluetooth, etc.).

Bus 705 includes pathways for transferring information between the various components of the device (e.g., processor 701, memory 702, input/output interface 703, and communication interface 704). .

It should be noted that the device mentioned above only shows the processor 701, memory 702, input/output interface 703, communication interface 704 and bus 705, but in the specific implementation process, the device is normal It may also contain other components required for operation. In addition, those skilled in the art will understand that the above-described device may include only components necessary for implementing the schemes of the embodiments herein without needing to include all components shown in the drawings.

Embodiments of the present invention further provide a computer readable storage medium storing a computer program, and the program implements the method according to any of the above embodiments when executed by a processor.

Computer-readable storage media includes permanent and non-persistent, removable and non-removable media, and storage of information may be realized by any method or technology. The information may be computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory ( RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices or any other non-transmitting medium, including but not limited to, may be configured to store information that can be accessed by computing devices. According to the definition in this paper, computer readable storage media does not include transitory media such as modulated data signals and carrier waves.

From the above description of implementations, it can be seen that those skilled in the art can clearly understand that the embodiments herein can be implemented by software and a necessary general-purpose hardware platform. Based on this understanding, for the technical solutions of the embodiments of this description, their essential part, that is, the part contributing to the prior art, may be embodied in the form of a software product. A computer software product may be stored on a storage medium. For example, a computer software product, such as a ROM/RAM, a magnetic disk, an optical disk, or the like, may be a computer device (which may be a personal computer, a server, a network device, or the like) in each embodiment of this description or a method described in some portion of the embodiment. can contain several commands that allow you to run

The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having specific functions. Typical implementations are computers, some types of computers being personal computers, laptop computers, cellular phones, camera phones, smart phones, personal digital assistants, media players, navigation devices, e-mail transceiver devices, game consoles, It may be a tablet computer, a wearable device, or a combination of any of these devices.

Various embodiments herein have been described in a progressive manner, and reference may be made to like parts relative to each other. The description of each embodiment differs from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is simplified, and reference can be made to corresponding parts of the description of the method embodiments. In the foregoing device embodiments, modules described as separate components may or may not be physically separated, and functions of the modules may be implemented in one or more software and/or hardware when embodiments of the present description are implemented. These are just sketchy things. Some or all of the modules may be selected according to actual requirements to implement the objectives of the solutions in the embodiments. Those skilled in the art can understand and implement the present disclosure without creative work.

Claims (16)

As a three-dimensional reconstruction method,
Obtaining initial values of parameters of the target object by performing 3D reconstruction on a target object in the image through a 3D reconstruction network, wherein the initial values of the parameters are used to build a 3D model of the target object. ;
obtaining an optimized value of the parameter by optimizing an initial value of the parameter based on previously obtained supervision information representing characteristics of the target object;
and constructing a 3D model of the target object by performing skeletal skinning based on the optimized values of the parameters. According to claim 1,
the supervisory information includes first supervisory information or the supervisory information includes first supervisory information and second supervisory information;
the first supervision information includes at least one of an initial two-dimensional key point of the target object and semantic information of a plurality of pixel points of the target object in the image;
The 3D reconstruction method, wherein the second supervision information includes an initial 3D point cloud of the surface of the target object. According to claim 2,
The 3D reconstruction method further comprising extracting information of an initial 2D key point of the target object from the image through a key point extraction network. According to claim 2 or 3,
the image includes a depth map of the target object;
The 3D reconstruction method,
extract depth information of the plurality of pixel points of the target object from the depth map;
and back-projecting the plurality of pixel points of the target object in the depth map into a 3-dimensional space based on the depth information to obtain the initial 3-dimensional point cloud of the surface of the target object. Dimensional reconstruction method. According to claim 4,
the image further includes an RGB image of the target object; Extracting depth information of the plurality of pixel points of the target object from the depth map includes:
performing image segmentation on the RGB image;
determining an image area in which the target object is located in the RGB image based on a result of image segmentation;
determining an image area where the target object is located in the depth map based on an image area where the target object is located in the RGB image;
and obtaining depth information of the plurality of pixel points of an image area where the target object in the depth map is located. According to any one of claims 2 to 5,
The 3D reconstruction method is:
The 3D reconstruction method further comprising filtering outliers among the initial 3D point clouds and using the filtered initial 3D point clouds as the second supervisory information. According to any one of claims 1 to 6,
An image of the target object is collected and obtained through an image collecting device, and the parameters include a global rotation parameter of the target object, a key point rotation parameter of each key point of the target object, a shape parameter of the target object, and the image collecting device. includes a displacement parameter of;
The step of optimizing the initial value of the parameter based on previously obtained supervision information representing the characteristics of the target object,
In a situation where the initial value of the posture parameter and the initial value of the key point rotation parameter do not change, the current value of the displacement parameter and the global rotation parameter of the image acquisition device based on the supervision information and the initial value of the displacement parameter optimizing an initial value of to obtain an optimized value of the displacement parameter and an optimized value of the global rotation parameter;
An initial value of the key point rotation parameter and an initial value of the shape parameter are optimized based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, so that the optimized value of the key point rotation parameter and the shape parameter A three-dimensional reconstruction method comprising the step of obtaining an optimized value of According to claim 7,
the supervision information includes an initial two-dimensional key point of the target object;
The step of optimizing the current value of the displacement parameter and the initial value of the global rotation parameter of the image acquisition device based on the supervision information and the initial value of the displacement parameter,
obtaining a target 2D projection key point belonging to a preset portion of the target object among 2D projection key points corresponding to the 3D key point of the target object, wherein the 3D key point of the target object corresponds to the global rotation parameter Obtained based on the initial value of , the initial value of the key point rotation parameter and the initial value of the shape parameter, and the 2D projection key point is obtained based on the current value of the displacement parameter and the initial value of the global rotation parameter. Obtained by performing projection on the three-dimensional key points of -;
obtaining a first loss between the target 2D projection key point and the initial 2D key point;
obtaining a second loss between the initial value of the displacement parameter and the current value of the displacement parameter;
and optimizing a current value of the displacement parameter and an initial value of the global rotation parameter based on the first loss and the second loss. According to claim 7 or 8,
the supervision information includes an initial two-dimensional key point of the target object; Optimizing the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter comprises:
obtaining a third loss between an optimized 2D projection key point of the target object and the initial 2D key point, wherein the optimized 2D projection key point is an optimized value of the displacement parameter and an optimized value of the global rotation parameter. obtained by performing projection on an optimized 3D key point of the target object based on a value, wherein the optimized 3D key point is an optimized value of the global rotation parameter, an initial value of the key point rotation parameter, and the shape parameter. Obtained based on initial value -;
obtaining a fourth loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter;
and optimizing an initial value of the key point rotation parameter and an initial value of the shape parameter based on the third loss and the fourth loss. According to any one of claims 7 to 9,
After optimizing the initial value of the key point rotation parameter and the initial value of the shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the 3D reconstruction method comprises:
Performing joint optimization on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter. 3D reconstruction method. According to claim 10,
the supervision information includes an initial two-dimensional key point of the target object and an initial three-dimensional point cloud of a surface of the target object; The step of optimizing the current value of the displacement parameter and the initial value of the global rotation parameter of the image acquisition device based on the supervision information and the initial value of the displacement parameter,
obtaining a target 2D projection key point belonging to a preset portion of the target object among 2D projection key points corresponding to the 3D key point of the target object, wherein the 3D key point of the target object corresponds to the global rotation parameter is obtained based on an initial value of , an initial value of the key point rotation parameter and an initial value of the shape parameter, and the two-dimensional projection key point is obtained based on the current value of the displacement parameter and the initial value of the global rotation parameter. Obtained by performing projection on the object's three-dimensional key points -;
obtaining a first loss between the target 2D projection key point and the initial 2D key point;
obtaining a second loss between the initial value of the displacement parameter and the current value of the displacement parameter;
obtaining a fifth loss between a first 3D point cloud of the target object surface and the initial 3D point cloud, wherein the first 3D point cloud is the initial value of the global rotation parameter, the key point rotation parameter Obtained based on the initial value and the initial value of the state parameter -;
and optimizing a current value of the displacement parameter and an initial value of the global rotation parameter based on the first loss, the second loss, and the fifth loss. According to claim 10 or 11,
Performing joint optimization on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the shape parameter and the optimized value of the displacement parameter:
obtaining a sixth loss between an optimized 2D projection key point of the target object and the initial 2D key point, wherein the optimized 2D projection key point is an optimized value of the displacement parameter and an optimized value of the global rotation parameter. obtained by performing projection on an optimized 3D key point of the target object based on a value, wherein the optimized 3D key point includes an optimized value of the global rotation parameter, an optimized value of the key point rotation parameter, and the shape parameter. Obtained based on the optimized value of -;
obtaining a seventh loss for representing the rationality of a posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter;
obtaining an eighth loss between a second 3D point cloud of the target object surface and the initial 3D point cloud, wherein the second 3D point cloud is an optimized value of the global rotation parameter, the key point rotation parameter Obtained based on the optimized value of and the optimized value of the state parameter -;
An optimized value of the global rotation parameter, an optimized value of the key point rotation parameter, an optimized value of the shape parameter, and an optimized value of the displacement parameter based on the sixth loss, the seventh loss, and the eighth loss. A three-dimensional reconstruction method comprising the step of performing federated optimization on the values. As a three-dimensional reconstruction device,
A first 3D reconstruction module for performing 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain initial values of parameters of the target object, wherein the initial values of the parameters form a 3D model of the target object used to build -;
an optimization module for obtaining an optimized value of the parameter by optimizing an initial value of the parameter based on previously obtained supervision information representing characteristics of the target object;
and a second 3D reconstruction module for constructing a 3D model of the target object by performing skeletal skinning based on the optimized values of the parameters. As a three-dimensional reconstruction system,
an image collecting device for collecting an image of a target object; and
a processing unit communicatively connected with the image collection device, which is used to obtain initial values of parameters of the target object by performing 3D reconstruction on the target object in the image through a 3D reconstruction network, wherein the initial values of the parameters are used to build a three-dimensional model of the target object; optimize an initial value of the parameter according to previously obtained supervision information representing characteristics of the target object to obtain an optimized value of the parameter; and constructing a 3D model of the target object by performing skeletal skinning based on the optimized values of the parameters. As a computer readable storage medium,
A computer readable storage medium in which a computer program is stored and the method according to any one of claims 1 to 12 is implemented when the computer program is executed by a processor. As a computer device,
A computer comprising a memory, a processor and a computer program stored in the memory and executable by the processor, wherein the method according to any one of claims 1 to 12 is implemented when the program is executed by the processor. device.

Images