CN115862149A - Method and system for generating 3D human skeleton key point data set - Google Patents
Method and system for generating 3D human skeleton key point data set Download PDFInfo
- Publication number
- CN115862149A CN115862149A CN202211721504.3A CN202211721504A CN115862149A CN 115862149 A CN115862149 A CN 115862149A CN 202211721504 A CN202211721504 A CN 202211721504A CN 115862149 A CN115862149 A CN 115862149A
- Authority
- CN
- China
- Prior art keywords
- model
- smpl
- human body
- parameters
- key point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 71
- 238000005457 optimization Methods 0.000 claims abstract description 49
- 238000002372 labelling Methods 0.000 claims abstract description 27
- 210000000988 bone and bone Anatomy 0.000 claims abstract description 15
- 238000009877 rendering Methods 0.000 claims description 21
- 230000000694 effects Effects 0.000 claims description 17
- 230000009471 action Effects 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 8
- 230000000877 morphologic effect Effects 0.000 claims description 6
- 230000036544 posture Effects 0.000 abstract description 36
- 239000000203 mixture Substances 0.000 description 20
- 238000004590 computer program Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 6
- 238000013507 mapping Methods 0.000 description 5
- 230000037237 body shape Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- WYBXZVJGRKCXPJ-UHFFFAOYSA-N 4-chloro-n-[3-(2,3-dihydro-1,4-benzodioxin-6-ylamino)quinoxalin-2-yl]benzenesulfonamide Chemical compound C1=CC(Cl)=CC=C1S(=O)(=O)NC1=NC2=CC=CC=C2N=C1NC1=CC=C(OCCO2)C2=C1 WYBXZVJGRKCXPJ-UHFFFAOYSA-N 0.000 description 2
- 101000742346 Crotalus durissus collilineatus Zinc metalloproteinase/disintegrin Proteins 0.000 description 2
- 101000872559 Hediste diversicolor Hemerythrin Proteins 0.000 description 2
- 230000001276 controlling effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 210000003127 knee Anatomy 0.000 description 2
- 238000013178 mathematical model Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 231100000673 dose–response relationship Toxicity 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
Images
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention provides a method and a system for generating a 3D human skeleton key point data set, S1, obtaining 2D posture data containing labeling information of depth sequence between skeleton edges; and S2, obtaining an optimal model of each labeled object in the 2D attitude data by utilizing an SMPLfy model and an iterative optimization algorithm, and obtaining 3D bone key point data and json file labeled files of the reconstructed human body model. According to the method, by using the depth sequence marking information of the joint edge and the depth sequence punishment optimization item, the ambiguity that the same 2D key point marking information corresponds to a plurality of 3D body postures can be reduced to the maximum extent, the optimization precision is greatly improved, and the optimization loss is greatly reduced.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a method and a system for generating 3D human skeleton key points.
Background
Human skeletal keypoint detection is the basis for many computer vision tasks, such as motion classification, behavior recognition, and unmanned driving, among others. Meanwhile, under the background of rapid development of the current metauniverse related technology, the technologies such as the human-computer interaction related technology, the virtual digital human drive and the like are applied more and more widely in practical engineering. As a human-computer interaction body, the recognition accuracy of the action posture of a human directly influences the interactive user experience. Compared with a 2D skeleton key point algorithm, the human body 3D skeleton key point detection algorithm has more one-dimensional depth information, can improve the accuracy of gesture recognition, and is a main method for recognizing human body gestures at present. Therefore, it is important to improve the accuracy of the human body 3D bone key point detection algorithm. And the data is used as a base stone of the algorithm, and the accuracy and generalization capability of the algorithm are directly influenced.
The key challenge of constructing a human body 3D skeleton key point algorithm model is to obtain vivid 3D key point data in a natural environment, and compared with 2D key point data which can be obtained through artificial marking, accurate manual 3D key point marking becomes a unique challenging task.
The existing human 3D key point data is mainly synthetic data and data obtained in a laboratory environment (the representative data sets are CMU Panoptic Dataset and human3.6 m). For pure synthetic data, data synthesized by a 3D rendering engine introduces another challenge of adapting to real-world pictures due to a large difference between human skin and wearing fashion and a real human body, resulting in a poor detection effect of an algorithm trained based on a synthetic data set on a real human body. For data to be obtained in a laboratory environment, a laboratory setup and dedicated hardware with depth sensors for 3D scanning are required-this puts other additional constraints on the data: for example, the data set does not maintain a good level of human and environmental diversity. Therefore, the method and the system for quickly generating the 3D human skeleton key point data with various environments and people have important practical value and practical significance for the algorithm research based on the 3D key point data.
The background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims in this application and is not admitted to be prior art by inclusion in this section.
Disclosure of Invention
In view of the above technical problems in the related art, the present invention provides a method for generating a 3D human skeleton key point data set,
s1, acquiring 2D attitude data containing labeling information of depth sequence between skeleton edges;
s2, obtaining an optimal model of each labeled object in the 2D attitude data by utilizing an SMPLfy model and an iterative optimization algorithm, and obtaining 3D bone key point data and json file labeled files of the reconstructed human body model; the optimization items of the SMPLfy iterative optimization algorithm are as follows:
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
wherein E J Representing 2D and 3D keypoint reprojection errors, J depth Labeling information for the depth sequence of the 2D joint edges; e a As attitude penalty term, E θ As an attitude prior term, E sp As collision term, E β For the identity constraint terms, the corresponding beta and theta in each term respectively represent the human body morphological parameter and the human body joint angle parameter.
Specifically, the method further comprises a step S3 of finely adjusting key parameters of the 3D model of the reconstructed human body model to meet the preset precision requirement, and acquiring corresponding joint point coordinate data to generate 3D bone key point data.
Specifically, the step S3 further includes:
s31, importing the SMPL three-dimensional model into a Unity scene;
s32, rendering an effect SMPL three-dimensional model in real time according to the beta, theta and (S, R, t) parameters of the SMPL obtained in the step S2, wherein the beta and the theta respectively represent human body form parameters and human body joint angle parameters, and the S, R and t represent zooming, rotating and translating parameters of the weak perspective projection;
s33, adjusting the parameters of beta, theta and (S, R, t) to enable the precision of the SMPL model to fit the action of the figure of the 2D picture to meet the preset requirement;
and S34, acquiring joint point coordinate data of the SMPL model with the standard precision, and automatically generating a standard json file annotation file.
Specifically, the step S31 is:
importing the SMPL three-dimensional model into the Unity scene, and exposing the adjustable beta, theta and (s, R, t) parameters of the SMPL model on an operation panel.
Specifically, the step S32 is: and (3) transmitting the parameters of beta, theta and (S, R, t) of the SMPL obtained in the step (S2) to the Unity in real time through Socket communication, receiving the parameters by the Unity, rendering the SMPL three-dimensional model with the effect in real time, and updating the SMPL model display result in the scene.
In a second aspect, another embodiment of the present invention discloses a system for generating a 3D human bone key point data set, comprising the following units:
the 2D data labeling unit is used for acquiring 2D attitude data containing labeling information of depth sequences among skeleton edges;
the 3D human body reconstruction and skeleton key point data acquisition unit is used for acquiring an optimal model of each labeled object in the 2D attitude data by utilizing an SMPLfy model and an iterative optimization algorithm and acquiring reconstructed human body model 3D skeleton key point data; the optimization items of the SMPLfy iterative optimization algorithm are as follows:
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
wherein E J Representing 2D and 3D keypoint reprojection errors, J depth Labeling information for the depth sequence of the 2D joint edges; e a As a pose penalty term, E θ As an attitude prior term, E sp As a collision term, E β For the identity constraint terms, the corresponding beta and theta in each term respectively represent the human body morphological parameter and the human body joint angle parameter.
Specifically, the system further includes: and the 3D model fine-tuning unit is used for fine-tuning the key parameters of the 3D model of the reconstructed human body model to enable the key parameters to meet the preset precision requirement and acquiring the corresponding joint point coordinate data to generate 3D skeleton key point data.
Specifically, the 3D model fine-tuning unit further includes:
the three-dimensional model importing unit is used for importing the SMPL three-dimensional model into a Unity scene;
the three-dimensional model rendering unit is used for rendering an SMPL three-dimensional model with an effect in real time according to the beta, theta and (s, R, t) parameters of the SMPL acquired in the 3D human body reconstruction and skeleton key point data acquisition unit;
the parameter fine-tuning unit is used for adjusting the beta, theta and (s, R, t) parameters to enable the precision of the SMPL model to fit the action of the figure of the 2D picture to reach the preset requirement;
and the joint point coordinate data acquisition unit is used for acquiring joint point coordinate data of the SMPL model with the standard precision and automatically generating a standard json file marking file.
Specifically, the three-dimensional model importing unit is as follows:
the SMPL three-dimensional model is imported into the Unity scene and the SMPL model adjustable β, θ and (s, R, t) parameters are exposed on the operation panel.
Specifically, the three-dimensional model rendering unit is: and transmitting the beta, theta and (s, R, t) parameters of the SMPL obtained in the 3D human body reconstruction and skeleton key point data acquisition unit to the Unity in real time through Socket communication, receiving the parameters by the Unity, rendering an effect SMPL three-dimensional model in real time, and updating an SMPL model display result in the scene.
According to the method, the ambiguity that the same 2D key point marking information corresponds to a plurality of 3D body postures can be reduced to the maximum extent by using the depth sequence marking information of the joint edge and the depth sequence punishment optimization item, the optimization precision is greatly improved, the optimization loss is greatly reduced, and the reconstruction error is reduced from 25% to 3% after the depth sequence marking information is added. Furthermore, the invention also provides a data fine-tuning labeling unit, so that the human body with larger reconstruction error can be subjected to manual fine tuning, the diversity of data is increased, and the accuracy and the reliability of the data are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a flowchart of a method for generating a 3D human skeleton key point data set according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of original labeling and depth order increasing labeling of 2D key points according to an embodiment of the present invention;
FIG. 3 is a diagram of the relationship between the key points of the hand in space and time according to an embodiment of the present invention;
fig. 4 is a schematic flow chart of acquiring a 3D key point coordinate by combining SMPLify according to the embodiment of the present invention;
FIG. 5 is a schematic workflow diagram of a conditioning system provided by an embodiment of the present invention;
FIG. 6 is a schematic diagram of a real-time rendering screen and a parameter adjustment panel of the trimming system according to an embodiment of the present invention;
FIG. 7 is a graphical illustration of the pre-and post-fine tuning variation provided by an embodiment of the present invention;
FIG. 8 is a schematic diagram of a system for generating a 3D human skeleton key point data set according to an embodiment of the present invention;
fig. 9 is a schematic diagram of an apparatus for generating a 3D human skeleton key point data set according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
Example one
Referring to fig. 1, the present embodiment discloses a method of generating a 3D human bone key point data set,
s1, acquiring 2D attitude data containing labeling information of depth sequence between skeleton edges;
the presently disclosed 2D keypoint data sets, such as COCO, MPII, contain rich background and scene information, and train high quality 2D keypoint detection algorithms. However, in the human body 3D key point algorithm task, the data set thereof is mostly data of a laboratory scene with a single background scene at present. Therefore, if the 2D keypoint data set can be used for a 3D keypoint algorithm task, the accuracy and robustness of the 3D keypoint algorithm can be directly improved. The current SMPLify model can reconstruct a human body model of an object directly from RGB images and corresponding human body 2D keypoint information. But this method is not ideal for reconstruction with slightly more complex motion.
For this case, the present embodiment adds the depth order between the determined gesture skeleton edges on the basis of the 2D keypoint labeling. As shown in fig. 3, the original 2D keypoint labeled information graph is shown on the left, and the skeleton edge depth-added sequential labeled information graph is shown on the right. The wider edge angles in the right figure represent joints closer to the camera (in the example, the person's knee is closer to the camera than the hip). Compared with the marking of real depth information, the marking task is simpler, and the precision of a human body reconstruction algorithm can be directly and effectively improved.
S2, obtaining an optimal model of each labeled object in the 2D attitude data by utilizing an SMPLfy model and an iterative optimization algorithm, and obtaining reconstructed 3D skeleton key point data of the human body model and json file labeled files; the optimization items of the SMPLfy iterative optimization algorithm are as follows:
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
wherein E J Representing 2D and 3D keypoint reprojection errors, J depth Is 2D joint marginMarking information in the depth sequence; e a As a pose penalty term, E θ As an attitude prior term, E sp As collision term, E β For the identity constraint terms, the corresponding beta and theta in each term respectively represent the human body morphological parameter and the human body joint angle parameter.
The SMPL Model is called Skinned Multi-Person Linear (SMP) Model. The human body three-dimensional model is a naked (skeletal) human body, and based on a vertex-based human body three-dimensional model, different shapes (shape) and postures (position) of the human body can be accurately represented. Meanwhile, the SMPL is a model which can be learned, and the shape of the human body and the deformation under different postures can be better fitted through training.
SMPL divides body shape into identity-dependent shape and non-region pos-dependent shape. The human body can be understood as a basic model and the sum of deformation is carried out on the basis of the model, PCA is carried out on the basis of the deformation, and low-dimensional shape (shape) parameters of the depicted shape are obtained; meanwhile, a motion tree is used for representing the posture of a human body, namely the rotation relation of each joint point of the motion tree and a parent node, the relation can be represented as a three-dimensional vector, and finally the local rotation vector of each joint point forms the posture (position) parameter of the SPML model.
The overall steps for acquiring 3D data using SMPL are therefore:
a) First, an average human body (mean template) is defined as shown in FIG. 3 (a), which only includes
And ω;
b) The shape transformer parameters (shape blend) are introduced as shown in fig. 3 (b), and the vertices of the human body 3D model are recalculated using the parameters
Coordinate of key point of joint->
The position of (a);
c) Introducing a human body motion deformer (close blend shape) on the basis of the step b), and calculating the position of the vertex of the human body 3D model again by combining the calculation result in the step 2 as shown in a figure 3 (c), wherein the formula is shown in a formula (1);
d) And inputting the calculation results of the steps 1 to 3 into a blend skinning function W to realize the deformation of the human body model, which is shown in a figure 3 (d). The coordinates of all vertexes of the 3D model can be obtained by utilizing the deformed model
e) Obtaining dense landmark coordinates of the human body surface according to the finally deformed vertex distribution of the human body model;
in the above formula, each parameter represents the following meaning:
representing three-dimensional vertex parameters
ω denotes that the hybrid skinning weight function contains 6890 × 24 parameters, defining the weight of the vertex affected by each rotation matrix
Beta represents a body type parameter, i.e., a parameter such as height, weight, etc. corresponding to each person
●B s Representing the mapping relation from the body type parameters to the shape blend shape;
j represents the mapping relationship of body type parameters to joint points;
θ represents the pose parameter, i.e. the rotation angle of each joint;
·B P representing the mapping relation of the gesture parameters to the position blend shape;
w represents the blend skinning function (which may be LBS or DQBS)
In the process of acquiring human 3D key points by SMPL, it is known that a human model can be reconstructed by human body shape deformer parameters (shape blend shape), human body action deformer parameters (position blend shape) and camera model parameters (camera parameter). Therefore, if the algorithm end can obtain the shape deformer parameter (shape blend shape), the action deformer parameter (position blend shape) and the camera parameter (camera parameter) corresponding to each human body in the 2D posture data set, the 3D coordinate corresponding to the human body in the current 2D posture data set can be obtained.
The existing human body reconstruction algorithm has two main implementation schemes: the first method is reconstruction by means of iterative optimization for controlling iteration times and iteration precision, and represents that an algorithm has SMPLfy. The other method is to directly use the regression parameter of the deep learning algorithm to carry out reconstruction, and typical algorithms include SPIN, VIBE, HMR and the like.
In the embodiment, a mode of an iterative optimization algorithm based on SMPLify is selected, and a reconstruction result with a smaller error than an original algorithm can be obtained by improving an iterative optimization target.
The iterative optimization algorithm of the original SMPLfy optimizes the terms as
E total ＝E J (β,θ；K,J est )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
Wherein E J Representing the reprojection errors of the 2D and 3D key points, the overall optimization term expression after adding the depth sequence annotation information optimization penalty term of the 2D joint edge in the EJ term is
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
In this embodiment, by using the depth order annotation information of the joint edge and the depth order punishment optimization item, the ambiguity that the same 2D key point annotation information corresponds to a plurality of 3D body postures can be reduced to the maximum extent, the optimization precision is greatly improved, and the optimization loss is greatly reduced, and the experimental data shows that the reconstruction error is reduced from 25% to 3% after the depth order annotation information is added.
Fig. 4 shows a flow of how SMPL model-related parameters are calculated using a human reconstruction algorithm and 3D key points are obtained. It can be seen from the figure that the human body reconstruction algorithm and the human body 3D mathematical model task are separated in cascade, so this embodiment can integrate any advanced human body reconstruction-based algorithm without much modification and optimization.
According to the method for generating the 3D human skeleton key point data set, on the basis of 2D key point labeling, the depth sequence between the determined posture skeleton edges is increased, the ambiguity that the same 2D key point labeling information corresponds to a plurality of 3D body postures can be reduced to the greatest extent, the optimization precision is greatly improved, and the optimization loss is greatly reduced; further, compared with real depth annotation, a mode of adding the depth sequence between the determined posture skeleton edges in the 2D posture data is simpler, and the accuracy of a human body reconstruction algorithm can be effectively improved. The depth sequence between the edges of the posture skeleton is added in the human body reconstruction algorithm to be used as a penalty loss item, so that the algorithm precision of human body 3D reconstruction based on the 2D picture is effectively improved, and the 3D key point data precision is improved.
In the embodiment, the 2D posture data set added with the skeleton edge depth sequence is used, the human body is reconstructed by using an optimized fitting mode, and 3D skeleton key point data corresponding to the 2D posture data set can be generated quickly and accurately. The algorithm gap problem of data set environment, human body and domain migration is solved.
Further, the embodiment further includes step S3, fine-tuning the key parameters of the 3D model of the reconstructed human body model to meet the preset precision requirement, and obtaining the corresponding joint point coordinate data to generate 3D bone key point data.
Due to the nature of 3D to 2D projection, multiple points in 3D may have the same projection in 2D (i.e., X and Y are the same but Z is different). Therefore, for a given 2D pose data, the 3D model generated by the human reconstruction algorithm may have errors that affect the final accuracy of the model. Therefore, in addition to the algorithm reconstructing the 2D body posture, in order to improve the quality of data and reduce the error of the reconstruction algorithm, we require fine tuning of several parameters of β, θ and (s, R, t) of SMPL to meet the final data accuracy requirement.
Therefore, in the embodiment, after the 3D model corresponding to the 2D posture data set is generated by using the algorithm, a parameter fine-tuning system is further designed to perform real-time visual adjustment on the model reconstruction result, so that the human body reconstruction accuracy is increased, and the operability of the system is improved to reduce the labor cost. The whole work flow of the regulating system is shown in figure 5. And after the fine adjustment meets the precision requirement, the final accurate 3D human skeleton key point data can be obtained.
The specific process of the fine adjustment system is mainly divided into 4 steps:
s31, importing the SMPL three-dimensional model into a Unity scene;
specifically, in this step, the SMPL three-dimensional model is imported into the Unity scene, and adjustable β, θ and (s, R, t) parameters of the SMPL model are exposed on the operation panel, and the model in the scene can be rendered and visualized by adjusting the parameters, and the real-time rendering effect is shown in fig. 6.
S32, rendering an effect SMPL three-dimensional model in real time according to the beta, theta and (S, R, t) parameters of the SMPL obtained in the step S2;
specifically, in this embodiment, the β, θ and (S, R, t) parameters of the SMPL obtained in step S2 are transmitted to the Unity in real time through Socket communication, and the Unity receives the parameters, renders the effect SMPL three-dimensional model in real time, and updates the SMPL model display result in the scene.
S33, adjusting the parameters of beta, theta and (S, R, t) to enable the precision of the SMPL model to fit the action of the figure of the 2D picture to meet the preset requirement;
specifically, in this step, if the Unity real-time rendering visualization is adopted, and the accuracy of the parameters obtained by the algorithm at this time cannot meet the requirement, the corresponding parameters are manually adjusted by using the operation panel described in step S31, and the reconstruction effect is finely adjusted, so that the SMPL model of the operation panel completely fits the actions of the 2D picture character. As shown in fig. 7.
And S34, acquiring joint point coordinate data of the SMPL model with the standard precision, and automatically generating a standard json file marking file.
The embodiment also provides a data fine-tuning labeling system, so that manual fine-tuning can be performed on a human body with a large reconstruction error, and the accuracy and reliability of data are improved while the diversity of the data is increased. Since only a small number of reconstruction parameters need to be manually fine-tuned, the embodiment greatly reduces the labor cost for labeling the human body 3D key point data from zero.
Example two
Referring to fig. 8, the present embodiment discloses a system for generating a 3D human bone key point data set, which includes the following units:
the 2D data labeling unit is used for acquiring 2D attitude data containing labeling information of depth sequences among skeleton edges;
the presently disclosed 2D keypoint data sets, such as COCO, MPII, contain rich background and scene information, and train high quality 2D keypoint detection algorithms. However, in the human body 3D key point algorithm task, the data set thereof is mostly data of a laboratory scene with a single background scene at present. Therefore, if the 2D keypoint data set can be used for a 3D keypoint algorithm task, the accuracy and robustness of the 3D keypoint algorithm can be directly improved. The current SMPLify model can reconstruct a human body model of an object directly from RGB images and corresponding human body 2D keypoint information. But this method is not ideal for reconstruction with slightly complex motion.
For this case, the present embodiment adds the depth order between the determined gesture skeleton edges on the basis of the 2D keypoint labeling. As shown in fig. 3, the original 2D keypoint labeled information graph is shown on the left, and the skeleton edge depth-added sequential labeled information graph is shown on the right. The wider edge angles in the right figure represent joints closer to the camera (in the example, the person's knee is closer to the camera than the hip). Compared with the marking of real depth information, the marking task is simpler, and the precision of a human body reconstruction algorithm can be directly and effectively improved.
The 3D human body reconstruction and skeleton key point data acquisition unit is used for acquiring an optimal model of each labeled object in the 2D attitude data by utilizing an SMPLfy model and an iterative optimization algorithm, and acquiring reconstructed human body model 3D skeleton key point data and json file labeled files; the optimization items of the SMPLfy iterative optimization algorithm are as follows:
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
wherein E J Representing 2D and 3D keypoint reprojection errors, J depth Labeling information for the depth sequence of the 2D joint edges; e a As a pose penalty term, E θ As an attitude prior term, E sp As a collision term, E β For the identity constraint terms, the corresponding beta and theta in each term respectively represent the human body morphological parameter and the human body joint angle parameter.
The SMPL Model is called Skinned Multi-Person Linear (SMP) Model. The human body three-dimensional model is a naked (skinned) human body, and can accurately represent different shapes (shape) and postures (position) of the human body based on a vertex-based human body three-dimensional model. Meanwhile, the SMPL is a model which can be learned, and the shape of the human body and the deformation under different postures can be better fitted through training.
SMPL divides body shape into identity-dependent shape and non-rigid dose-dependent shape. The human body can be understood as a basic model and the sum of deformation on the basis of the model, PCA is carried out on the basis of the deformation, and low-dimensional shape (shape) parameters of the depicted shape are obtained; meanwhile, a motion tree is used for representing the posture of a human body, namely the rotation relation of each joint point of the motion tree and a parent node, the relation can be represented as a three-dimensional vector, and finally the local rotation vector of each joint point forms the posture (position) parameter of the SPML model.
The overall steps for acquiring 3D data using SMPL are therefore:
a) First, an average human body (mean template) is defined as shown in FIG. 3 (a), which only includes
And ω;
b) The shape shaper parameter (shape blend shape) is introduced as shown in FIG. 3 (b), and the vertex of the human body 3D model is recalculated by using the parameter
Coordinate of key point of joint->
The position of (a);
c) Introducing a human body motion deformer (close blend shape) on the basis of the step b), and calculating the position of the vertex of the human body 3D model again by combining the calculation result in the step 2 as shown in a figure 3 (c), wherein the formula is shown in a formula (1);
d) And inputting the calculation results of the steps 1 to 3 into a blend skinning function W to realize the deformation of the human body model, which is shown in a figure 3 (d). The coordinates of all vertexes of the 3D model can be obtained by utilizing the deformed model
/>
e) Obtaining dense landmark coordinates of the human body surface according to the finally deformed vertex distribution of the human body model;
in the above formula, the parameters represent the meanings given below:
representing three-dimensional vertex parameters
ω denotes that the hybrid skinning weight function contains 6890 × 24 parameters, defining the weight of the vertex affected by each rotation matrix
Beta represents a body type parameter, i.e., a parameter such as height, weight, etc. corresponding to each person
●B s Representing the mapping relation from the body type parameters to the shape blend shape;
● J represents the mapping relation from body type parameters to joint points;
θ represents the pose parameter, i.e. the rotation angle of each joint;
●B P representing a gesture parameter to a posMapping relation of blend shape;
● W represents the blend skinning function (which may be LBS or DQBS)
In the process of acquiring human 3D key points by SMPL, it is known that a human model can be reconstructed by human body shape deformer parameters (shape blend shape), human body action deformer parameters (position blend shape) and camera model parameters (camera parameter). Therefore, if the algorithm end can obtain the shape deformer parameter (shape blend shape), the action deformer parameter (position blend shape) and the camera parameter (camera parameter) corresponding to each human body in the 2D posture data set, the 3D coordinates corresponding to the human body in the current 2D posture data set can be obtained.
The main implementation schemes of the existing human body reconstruction algorithm are two: the first method is reconstruction by means of iterative optimization for controlling iteration times and iteration precision, and represents that an algorithm has SMPLfy. The other method is to directly use the regression parameters of the deep learning algorithm for reconstruction, and typical algorithms include SPIN, VIBE, HMR and the like.
In the embodiment, a mode of an iterative optimization algorithm based on SMPLify is selected, and a reconstruction result with a smaller error than an original algorithm can be obtained by improving an iterative optimization target.
The iterative optimization algorithm of the original SMPLfy optimizes the terms as
E total ＝E J (β,θ；K,J est )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
Wherein E J Representing the reprojection errors of the 2D and 3D key points, the overall optimization term expression after adding the depth sequence annotation information optimization penalty term of the 2D joint edge in the EJ term is
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
In this embodiment, by using the depth order annotation information of the joint edge and the depth order punishment optimization item, the ambiguity that the same 2D key point annotation information corresponds to a plurality of 3D body postures can be reduced to the maximum extent, the optimization precision is greatly improved, and the optimization loss is greatly reduced, and the experimental data shows that the reconstruction error is reduced from 25% to 3% after the depth order annotation information is added.
Fig. 4 shows a flow of how SMPL model-related parameters are calculated using a human reconstruction algorithm and 3D key points are obtained. It can be seen from the figure that the human body reconstruction algorithm and the human body 3D mathematical model task are separated in cascade, so this embodiment can integrate any advanced human body reconstruction-based algorithm without much modification and optimization.
According to the method for generating the 3D human skeleton key point data set, on the basis of 2D key point labeling, the depth sequence between the determined posture skeleton edges is increased, the ambiguity that the same 2D key point labeling information corresponds to a plurality of 3D body postures can be reduced to the greatest extent, the optimization precision is greatly improved, and the optimization loss is greatly reduced; further, compared with real depth annotation, a mode of adding the depth sequence between the determined posture skeleton edges in the 2D posture data is simpler, and the accuracy of a human body reconstruction algorithm can be effectively improved. The depth sequence between the edges of the posture skeleton is added in the human body reconstruction algorithm to be used as a penalty loss item, so that the algorithm precision of human body 3D reconstruction based on the 2D picture is effectively improved, and the 3D key point data precision is improved.
In the embodiment, the 2D posture data set added with the skeleton edge depth sequence is used, the human body is reconstructed in an optimized fitting mode, and 3D skeleton key point data corresponding to the 2D posture data set can be rapidly and accurately generated. The algorithm gap problem of data set environment, human body and domain migration is solved.
Further, the embodiment further includes a 3D model fine tuning unit, configured to fine tune a key parameter of the 3D model of the reconstructed human body model to meet a preset precision requirement, and obtain corresponding joint point coordinate data to generate 3D bone key point data.
Due to the nature of 3D to 2D projection, multiple points in 3D may have the same projection in 2D (i.e., X and Y are the same but Z is different). Therefore, for a given 2D pose data, the 3D model generated by the human reconstruction algorithm may have errors that affect the final accuracy of the model. Therefore, in addition to the algorithm reconstructing the 2D body posture, in order to improve the quality of data and reduce the error of the reconstruction algorithm, we require fine tuning of several parameters of β, θ and (s, R, t) of SMPL to meet the final data accuracy requirement.
Therefore, in the embodiment, after the 3D model corresponding to the 2D posture data set is generated by using the algorithm, a 3D model fine tuning unit is further designed to perform real-time visual adjustment on the model reconstruction result, so that the human body reconstruction accuracy is increased, and the operability of the system is improved to reduce the labor cost. The whole work flow of the 3D model fine tuning unit is shown in figure 5. And after the fine adjustment meets the precision requirement, the final accurate 3D human skeleton key point data can be obtained.
The parameter fine-tuning unit comprises:
the three-dimensional model importing unit is used for importing the SMPL three-dimensional model into a Unity scene;
specifically, in this step, the SMPL three-dimensional model is imported into the Unity scene, and adjustable β, θ and (s, R, t) parameters of the SMPL model are exposed on the operation panel, and the model in the scene can be rendered and visualized by adjusting the parameters, and the real-time rendering effect is shown in fig. 6.
The three-dimensional model rendering unit is used for rendering an SMPL three-dimensional model with an effect in real time according to the beta, theta and (s, R, t) parameters of the SMPL obtained by the 3D human body reconstruction and skeleton key point data obtaining unit;
specifically, in this embodiment, the β, θ and (S, R, t) parameters of the SMPL obtained in step S2 are transmitted to the Unity in real time through Socket communication, and the Unity receives the parameters, renders the effect SMPL three-dimensional model in real time, and updates the SMPL model display result in the scene.
The parameter fine-tuning unit is used for adjusting the beta, theta and (s, R, t) parameters to enable the precision of the SMPL model to fit the action of the figure of the 2D picture to meet the preset requirement;
specifically, in this step, if the Unity real-time rendering visualization is adopted, and the accuracy of the parameters obtained by the algorithm at this time cannot meet the requirement, the corresponding parameters are manually adjusted by using the operation panel described in step S31, and the reconstruction effect is finely adjusted, so that the SMPL model of the operation panel completely fits the actions of the 2D picture character. As shown in fig. 7.
And the joint point coordinate data acquisition unit is used for acquiring joint point coordinate data of the SMPL model with the standard precision and automatically generating a standard json file marking file.
This embodiment still through data fine setting mark unit for can also improve the accuracy and the reliability of data when increasing data variety to rebuilding the great human body of error and carrying out artifical fine setting. Since only a small number of reconstruction parameters need to be manually fine-tuned, the embodiment greatly reduces the labor cost for labeling the human body 3D key point data from zero.
EXAMPLE III
Referring to fig. 9, fig. 9 is a schematic structural diagram of an apparatus for generating a 3D human bone key point data set according to the present embodiment. The apparatus 20 for generating a 3D human skeletal keypoint data set of this embodiment comprises a processor 21, a memory 22 and a computer program stored in said memory 22 and executable on said processor 21. The processor 21 realizes the steps in the above-described method embodiments when executing the computer program. Alternatively, the processor 21 implements the functions of the modules/units in the above device embodiments when executing the computer program.
Illustratively, the computer program may be divided into one or more modules/units, which are stored in the memory 22 and executed by the processor 21 to accomplish the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions for describing the execution of the computer program in the apparatus 20 for generating a 3D human skeletal keypoint data set. For example, the computer program may be divided into the modules in the second embodiment, and for the specific functions of the modules, reference is made to the working process of the apparatus in the foregoing embodiment, which is not described herein again.
The apparatus 20 for generating a 3D human skeletal key point data set may include, but is not limited to, a processor 21, a memory 22. It will be appreciated by those skilled in the art that the schematic diagram is merely an example of the apparatus 20 for generating a 3D human skeletal keypoint data set and does not constitute a limitation of the apparatus 20 for generating a 3D human skeletal keypoint data set and may comprise more or fewer components than shown, or some components in combination, or different components, e.g. the apparatus 20 for generating a 3D human skeletal keypoint data set may further comprise an input-output device, a network access device, a bus, etc.
The Processor 21 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, and the processor 21 is the control center of the apparatus for generating 3D human bone key data set 20, and various interfaces and lines are used to connect the various parts of the apparatus for generating 3D human bone key data set 20 as a whole.
The memory 22 may be used to store the computer programs and/or modules, and the processor 21 implements various functions of the apparatus 20 for generating a 3D human skeletal key point data set by running or executing the computer programs and/or modules stored in the memory 22 and calling up the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory 22 may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
Wherein, the modules/units integrated by the apparatus 20 for generating the 3D human skeleton key point data set can be stored in a computer readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer readable storage medium, and when the computer program is executed by the processor 21, the steps of the method embodiments described above may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that the above-described device embodiments are merely illustrative, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement without inventive effort.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for generating a 3D human skeleton key point data set,
s1, acquiring 2D attitude data containing labeling information of depth sequence between skeleton edges;
s2, obtaining an optimal model of each labeled object in the 2D attitude data by utilizing an SMPLfy model and an iterative optimization algorithm, and obtaining 3D bone key point data and json file labeled files of the reconstructed human body model; the optimization items of the SMPLfy iterative optimization algorithm are as follows:
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
wherein E J Representing 2D and 3D keypoint reprojection errors, J depth Labeling information for the depth sequence of the 2D joint edges; e a As a pose penalty term, E θ As an attitude prior term, E sp As a collision term, E β For the identity constraint terms, the corresponding beta and theta in each term respectively represent the human body morphological parameter and the human body joint angle parameter.
2. The method according to claim 1, further comprising step S3 of fine-tuning key parameters of the 3D model of the reconstructed human body model to meet a preset accuracy requirement, and obtaining corresponding joint point coordinate data to generate 3D bone key point data.
3. The method of claim 2, the step S3 further comprising:
s31, importing the SMPL three-dimensional model into a Unity scene;
s32, rendering an effect SMPL three-dimensional model in real time according to the beta, theta and (S, R, t) parameters of the SMPL obtained in the step S2, wherein the beta and the theta respectively represent human body form parameters and human body joint angle parameters, and the S, R and t represent zooming, rotating and translating parameters of the weak perspective projection;
s33, adjusting the parameters of beta, theta and (S, R, t) to enable the precision of the motion of the SMPL model fitting the 2D picture character to reach a preset requirement;
and S34, acquiring joint point coordinate data of the SMPL model with the standard precision, and automatically generating a standard json file marking file.
4. The method of claim 3, wherein the step S31 is:
the SMPL three-dimensional model is imported into the Unity scene and the SMPL model adjustable β, θ and (s, R, t) parameters are exposed on the operation panel.
5. The method of claim 3, wherein the step S32 is: and (3) transmitting the parameters of beta, theta and (S, R, t) of the SMPL obtained in the step (S2) to the Unity in real time through Socket communication, receiving the parameters by the Unity, rendering the SMPL three-dimensional model with the effect in real time, and updating the SMPL model display result in the scene.
6. A system for generating a 3D human bone key-point data set, comprising the following units:
the 2D data labeling unit is used for acquiring 2D attitude data containing labeling information of depth sequences among skeleton edges;
the 3D human body reconstruction and skeleton key point data acquisition unit is used for acquiring an optimal model of each marked object in the 2D posture data by utilizing an SMPLfy model and an iterative optimization algorithm and acquiring reconstructed human body model 3D skeleton key point data; the optimization items of the SMPLfy iterative optimization algorithm are as follows:
E total ＝E J (β,θ；K,J est ,J depth )+λ θ E θ (θ)+λ a E a (θ)+λ sp E sp (θ；β)+λ β E β (β)
wherein E J Representing 2D and 3D keypoint reprojection errors, J depth Labeling information for the depth sequence of the 2D joint edges; e a As a pose penalty term, E θ As an attitude prior term, E sp As a collision term, E β For the identity constraint terms, the corresponding beta and theta in each term respectively represent the human body morphological parameter and the human body joint angle parameter.
7. The system of claim 6, further comprising: and the 3D model fine-tuning unit is used for fine-tuning the key parameters of the 3D model of the reconstructed human body model to enable the key parameters to meet the preset precision requirement and acquiring the corresponding joint point coordinate data to generate 3D skeleton key point data.
8. The system of claim 7, the 3D model fine tuning unit further comprising:
the three-dimensional model importing unit is used for importing the SMPL three-dimensional model into a Unity scene;
the three-dimensional model rendering unit is used for rendering an SMPL three-dimensional model with an effect in real time according to the beta, theta and (s, R, t) parameters of the SMPL obtained by the 3D human body reconstruction and skeleton key point data obtaining unit;
the parameter fine-tuning unit is used for adjusting the beta, theta and (s, R, t) parameters to enable the precision of the SMPL model to fit the action of the figure of the 2D picture to reach the preset requirement;
and the joint point coordinate data acquisition unit is used for acquiring joint point coordinate data of the SMPL model with the standard precision and automatically generating a standard json file marking file.
9. The system of claim 8, the three-dimensional model importing unit being:
the SMPL three-dimensional model is imported into the Unity scene and the SMPL model adjustable β, θ and (s, R, t) parameters are exposed on the operation panel.
10. The system of claim 8, the three-dimensional model rendering unit being: and transmitting the beta, theta and (s, R, t) parameters of the SMPL obtained in the 3D human body reconstruction and skeleton key point data acquisition unit to the Unity in real time through Socket communication, receiving the parameters by the Unity, rendering an effect SMPL three-dimensional model in real time, and updating an SMPL model display result in the scene.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211721504.3A CN115862149B (en) | 2022-12-30 | 2022-12-30 | Method and system for generating 3D human skeleton key point data set |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211721504.3A CN115862149B (en) | 2022-12-30 | 2022-12-30 | Method and system for generating 3D human skeleton key point data set |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115862149A true CN115862149A (en) | 2023-03-28 |
| CN115862149B CN115862149B (en) | 2024-03-22 |
Family
ID=85656275
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211721504.3A Active CN115862149B (en) | 2022-12-30 | 2022-12-30 | Method and system for generating 3D human skeleton key point data set |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115862149B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117095131A (en) * | 2023-10-16 | 2023-11-21 | 广州紫为云科技有限公司 | Three-dimensional reconstruction method, equipment and storage medium for object motion key points |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109636831A (en) * | 2018-12-19 | 2019-04-16 | 安徽大学 | A method of estimation 3 D human body posture and hand information |
| CN111339903A (en) * | 2020-02-21 | 2020-06-26 | 河北工业大学 | Multi-person human body posture estimation method |
| CN111582036A (en) * | 2020-04-09 | 2020-08-25 | 天津大学 | Cross-view-angle person identification method based on shape and posture under wearable device |
| CN111738220A (en) * | 2020-07-27 | 2020-10-02 | 腾讯科技（深圳）有限公司 | Three-dimensional human body posture estimation method, device, equipment and medium |
| US20210049811A1 (en) * | 2019-08-13 | 2021-02-18 | Texel Llc | Method and System for Remote Clothing Selection |
| CN112750199A (en) * | 2021-01-13 | 2021-05-04 | 电子科技大学 | Energy optimization-based panda three-dimensional model reconstruction method |
| CN114119911A (en) * | 2020-08-27 | 2022-03-01 | 北京陌陌信息技术有限公司 | Human body model neural network training method, device and storage medium |
| CN114119907A (en) * | 2020-08-27 | 2022-03-01 | 北京陌陌信息技术有限公司 | Fitting method and device of human body model and storage medium |
| CN114202629A (en) * | 2020-08-27 | 2022-03-18 | 北京陌陌信息技术有限公司 | Human body model establishing method, system, equipment and storage medium |
| CN114359445A (en) * | 2021-12-29 | 2022-04-15 | 中央财经大学 | Training data generation method and device for human body bone joint point extraction model |
| CN114581613A (en) * | 2022-04-29 | 2022-06-03 | 杭州倚澜科技有限公司 | Trajectory constraint-based human body model posture and shape optimization method and system |
| WO2022166412A1 (en) * | 2021-02-05 | 2022-08-11 | 中国科学院深圳先进技术研究院 | Self-supervised three-dimensional reconstruction method and system based on collaborative segmentation and data enhancement |
| CN115083015A (en) * | 2022-06-09 | 2022-09-20 | 广州紫为云科技有限公司 | 3D human body posture estimation data labeling mode and corresponding model construction method |
| CN115115776A (en) * | 2022-06-15 | 2022-09-27 | 中国科学院软件研究所 | Single image three-dimensional human body reconstruction method and device based on shadow |
| WO2022222091A1 (en) * | 2021-04-22 | 2022-10-27 | 浙江大学 | Method for generating character bas-relief model on basis of single photo |
-
2022
- 2022-12-30 CN CN202211721504.3A patent/CN115862149B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109636831A (en) * | 2018-12-19 | 2019-04-16 | 安徽大学 | A method of estimation 3 D human body posture and hand information |
| US20210049811A1 (en) * | 2019-08-13 | 2021-02-18 | Texel Llc | Method and System for Remote Clothing Selection |
| CN111339903A (en) * | 2020-02-21 | 2020-06-26 | 河北工业大学 | Multi-person human body posture estimation method |
| CN111582036A (en) * | 2020-04-09 | 2020-08-25 | 天津大学 | Cross-view-angle person identification method based on shape and posture under wearable device |
| CN111738220A (en) * | 2020-07-27 | 2020-10-02 | 腾讯科技（深圳）有限公司 | Three-dimensional human body posture estimation method, device, equipment and medium |
| CN114119911A (en) * | 2020-08-27 | 2022-03-01 | 北京陌陌信息技术有限公司 | Human body model neural network training method, device and storage medium |
| CN114119907A (en) * | 2020-08-27 | 2022-03-01 | 北京陌陌信息技术有限公司 | Fitting method and device of human body model and storage medium |
| CN114202629A (en) * | 2020-08-27 | 2022-03-18 | 北京陌陌信息技术有限公司 | Human body model establishing method, system, equipment and storage medium |
| CN112750199A (en) * | 2021-01-13 | 2021-05-04 | 电子科技大学 | Energy optimization-based panda three-dimensional model reconstruction method |
| WO2022166412A1 (en) * | 2021-02-05 | 2022-08-11 | 中国科学院深圳先进技术研究院 | Self-supervised three-dimensional reconstruction method and system based on collaborative segmentation and data enhancement |
| WO2022222091A1 (en) * | 2021-04-22 | 2022-10-27 | 浙江大学 | Method for generating character bas-relief model on basis of single photo |
| CN114359445A (en) * | 2021-12-29 | 2022-04-15 | 中央财经大学 | Training data generation method and device for human body bone joint point extraction model |
| CN114581613A (en) * | 2022-04-29 | 2022-06-03 | 杭州倚澜科技有限公司 | Trajectory constraint-based human body model posture and shape optimization method and system |
| CN115083015A (en) * | 2022-06-09 | 2022-09-20 | 广州紫为云科技有限公司 | 3D human body posture estimation data labeling mode and corresponding model construction method |
| CN115115776A (en) * | 2022-06-15 | 2022-09-27 | 中国科学院软件研究所 | Single image three-dimensional human body reconstruction method and device based on shadow |
Non-Patent Citations (7)
| Title |
|---|
| LI, Z.等: ""CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation"", 《COMPUTER VISION – ECCV 2022》, vol. 13665, pages 590 * |
| N. RÜEGG等: ""BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information"", 《2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》, pages 3866 - 3874 * |
| 彭睿孜等: ""基于空洞转置卷积神经网络的人体骨骼关键点检测算法研究"", 《智能物联技术》, vol. 5, no. 1, pages 8 - 13 * |
| 徐土豆: ""人体动作捕捉与SMPL模型 (mocap and SMPL model)"", pages 1 - 12, Retrieved from the Internet <URL:《https://zhuanlan.zhihu.com/p/158700893》> * |
| 杨起航: ""基于深度学习的单目视频人体动作捕捉研究"", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 3, pages 138 - 1715 * |
| 林之钊等: ""从单幅图像生成人物三维动画"", 《计算机辅助设计与图形学学报》, vol. 34, no. 9, pages 1341 - 1350 * |
| 陈秋敏: ""基于深度学习的多视图物体三维重建研究"", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 7, pages 138 - 1237 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117095131A (en) * | 2023-10-16 | 2023-11-21 | 广州紫为云科技有限公司 | Three-dimensional reconstruction method, equipment and storage medium for object motion key points |
| CN117095131B (en) * | 2023-10-16 | 2024-02-06 | 广州紫为云科技有限公司 | Three-dimensional reconstruction method, equipment and storage medium for object motion key points |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115862149B (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6532011B1 (en) | Method of creating 3-D facial models starting from face images | |
| Zhou et al. | Parametric reshaping of human bodies in images | |
| US7570264B2 (en) | Rig baking | |
| JP2023526566A (en) | fast and deep facial deformation | |
| CN109712223B (en) | Three-dimensional model automatic coloring method based on texture synthesis | |
| US11587288B2 (en) | Methods and systems for constructing facial position map | |
| WO2022197431A1 (en) | Methods and systems for personalized 3d head model deformation | |
| Han et al. | Exemplar-based 3d portrait stylization | |
| CN110751733A (en) | Method and apparatus for converting 3D scanned object into avatar | |
| WO2022197429A1 (en) | Methods and systems for extracting color from facial image | |
| CN115862149B (en) | Method and system for generating 3D human skeleton key point data set | |
| CN113593001A (en) | Target object three-dimensional reconstruction method and device, computer equipment and storage medium | |
| Ye et al. | 3d morphable face model for face animation | |
| KR20230110787A (en) | Methods and systems for forming personalized 3D head and face models | |
| Milosevic et al. | A SmartPen for 3D interaction and sketch-based surface modeling | |
| CN115661367B (en) | Dynamic hybrid deformation modeling method and system based on photo collection | |
| CN116452715A (en) | Dynamic human hand rendering method, device and storage medium | |
| CN110751710A (en) | Method and apparatus for fitting an accessory object to an avatar | |
| CN116912433B (en) | Three-dimensional model skeleton binding method, device, equipment and storage medium | |
| Zhang et al. | CreatureShop: Interactive 3D Character Modeling and Texturing From a Single Color Drawing | |
| US20230306686A1 (en) | Three-dimensional mesh generator based on two-dimensional image | |
| Kazmi | Efficient sketch-based 3D character modelling. | |
| Hennessey | Assistive visual content creation tools via multimodal correlation analysis | |
| CN116766596A (en) | Character model printing method, character model printing device, electronic equipment and storage medium | |
| EP4111420A1 (en) | Face mesh deformation with detailed wrinkles |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |