WO2016207311A1  Skinned multiperson linear model  Google Patents
Skinned multiperson linear model Download PDFInfo
 Publication number
 WO2016207311A1 WO2016207311A1 PCT/EP2016/064610 EP2016064610W WO2016207311A1 WO 2016207311 A1 WO2016207311 A1 WO 2016207311A1 EP 2016064610 W EP2016064610 W EP 2016064610W WO 2016207311 A1 WO2016207311 A1 WO 2016207311A1
 Authority
 WO
 WIPO (PCT)
 Prior art keywords
 pose
 shape
 blend
 method
 model
 Prior art date
Links
Classifications

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T13/00—Animation
 G06T13/20—3D [Three Dimensional] animation
 G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T19/00—Manipulating 3D models or images for computer graphics
 G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T7/00—Image analysis
 G06T7/70—Determining position or orientation of objects or cameras
 G06T7/73—Determining position or orientation of objects or cameras using featurebased methods
 G06T7/75—Determining position or orientation of objects or cameras using featurebased methods involving models

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/30—Subject of image; Context of image processing
 G06T2207/30196—Human being; Person

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
 G06T2219/20—Indexing scheme for editing of 3D models
 G06T2219/2021—Shape modification
Abstract
Description
Skinned MultiPerson Linear Model
The present invention relates to realistic digital models of animated human bodies that can represent different body shapes, deform naturally with pose, and exhibit soft tissue motions like those of real humans.
INTRODUCTION AND RELATED WORK
It is desirable that such models are fast to render, easy to deploy, and compatible with existing rendering engines.
Linear blend skinning and blend shapes are widely used throughout the animation industry. The commercial approach commonly involves hand rigging a mesh and manually sculpting blend shapes to correct problems with traditional skinning methods. Many blend shapes are typically needed and the manual effort required to build them is large.
As an alternative, the research community has focused on learning statistical body models from example scans of different bodies in a varied set of poses. While promising, these approaches are not compatible with existing graphics software and rendering engines that use standard skinning methods.
Many authors have tried to bring these worlds together with varying degrees of suc cess.
Traditional methods model how vertices are related to an underlying skeleton structure. Basic linear blend skinning (LBS) models are the most widely used, are supported by all game engines, and are efficient to render. Unfortunately, they produce unre alistic deformations at joints including the wellknown taffy and bowtie effects . Work has gone into skinning methods that ameliorate these effects [Lewis et al. 2000; Wang and Phillips 2002; Kavan and Zara 2005; Merry et al. 2006; Kavan et al. 2008]. There has also been a lot of work on learning realistic body models from data [Allen et al. 2006; Anguelov et al. 2005; Freifeld and Black 2012; Hasler et al. 2010; Chang and Zwicker 2009; Chen et al. 2013]. These methods can capture the body shape of many people as well as nonrigid deformations due to pose. The most successful approaches are so far based on triangle deformations [Anguelov et al. 2005; Chen et al. 2013; Has ler et al. 2010; PonsMoll et al. 2015]. Despite the above research, existing models either lack realism, do not work with existing packages, do not represent a wide variety of body shapes, are not compatible with standard graphics pipelines, or require significant manual labor.
Blend Skinning. Skeleton subspace deformation methods also known as blend skinning, attach the surface of a mesh to an underlying skeletal structure. Each vertex in the mesh surface is transformed using a weighted influence of its neighboring bones. This influence can be defined linearly as in Linear Blend Skinning (LBS). The problems of LBS have been widely published and the literature is dense with generic methods that attempt to fix these, such as quaternion or dualquaternion skinning, spherical skinning, etc. (e.g. [Wang and Phillips 2002; Kavan and Zara 2005; Kavan et al. 2008; Le and Deng 2012; Wang et al. 2007]). Generic methods, however, often produce unnatural results.
Autorigging. There is a great deal of work on automatically rigging LBS models (e.g. [De Aguiar et al. 2008; Baran and Popovic 2007; Corazza and Gambaretto 2014; Schaefer and Yuksel 2007]) and commercial solutions exist. Most relevant here are methods that take a collection of meshes and infer the bones as well as the joints and blend weights (e.g. [Le and Deng 2014]). Such methods do not address the common problems of LBS models because they do not learn corrective blend shapes. Models created from sequences of meshes (e.g. [De Aguiar et al. 2008]) may not generalize well to new poses and motions
The key limitation of the above methods is that the models do not span a space of body shapes. Miller et al. [2010] partially address this by autorigging using a database of prerigged models. They formulate rigging and skinning as the process of transferring and adapting skinning weights from known models to a new model. Their method does not generate blend shapes, produces standard LBS artifacts, and does not minimize a clear objective function.
Blend shapes. To address the shortcomings of basic blend skinning, the pose space deformation model (PSD) [Lewis et al. 2000] defines deformations (as vertex dis placements) relative to a base shape, where these deformations are a function of articulated pose. This is largely followed by later approaches and is referred to as scattered data interpolation and corrective enveloping [Rouet and Lewis 1999] . Another approach is weighted posespace deformation (WPSD) [Kurihara and Miyata 2004; Rhee et al. 2006], which defines the corrections in a rest pose and then applies a standard skinning equation (e.g. LBS). The idea is to define corrective shapes (sculpts) for specific key poses, so that when added to the base shape and transformed by blend skinning, produce the right shape. Typically, one finds the distance (in pose space) to the exemplar poses and uses a function, e.g. a Radial Basis (RBF) kernel [Lewis et al. 2000], to weight the exemplars nonlinearly based on distance. The sculpted blend shapes are then weighted and linearly combined. In practice however, a large number of poses might be needed to cover the pose space well. This makes animation slow since the closest key poses have to be found at run time. These approaches are all based on computing weighted distances to exemplar shapes. Consequently, these methods require computation of the distances and weights at runtime to obtain the corrective blend shape. For a given animation (e.g. in a video game) these weights are often defined in advance based on the poses and baked into the model. Game engines apply the bakedin weights to the blend shapes. The sculpt ing process is typically done by an artist and then only for poses that will be used in the animation.
Learning pose models. Allen et al. [2002] use this PSD approach but rather than handsculpt the corrections, learn them from registered 3D scans. Their work focuses primarily on modeling the torso and arms of individuals, rather than whole bodies of a population. They store deformations of key poses and interpolate between them. When at, or close to, a stored shape, these methods are effectively perfect. They do not tend to generalize well to new poses, requiring dense training data. It is not clear how many such shapes would be necessary to model the full range of articulated human pose. As the complexity of the model increases, so does the complexity of controlling all these shapes and how they interact.
To address this, Kry et al. [2002] learn a lowdimensional PCA basis for each joints deformations. Posedependent deformations are described in terms of the coefficients of the basis vectors. Kavan et al. [2009] use example meshes generated using a non linear skinning method to construct linear approximations. James and Twigg [2005] combine the idea of learning the bones (nonrigid, affine bones) and skinning weights directly from registered meshes. For blend shapes, they use an approach similar to [Kry et al. 2002] .
Another way to address the limitations of blend skinning is through multiweight enveloping (MWE) [Wang and Phillips 2002]. Rather than weight each vertex by a weighted combination of the bone transformation matrices, MWE learns weights for the elements of these matrices. This increases the capacity of the model (more param eters). Like [James and Twigg 2005] they overparameterize the bone transformations to give more expressive power and then use PCA to remove unneeded degrees of freedom. Their experiments typically involve user interaction and current game engines do not support the MWE approach. Merry et al. [2006] find MWE to be overparameterized, because it allows vertices to deform differently depending on rotation in the global coordinate system. Their Animation Space model reduces the parameterization at minimal loss of representational power, while also showing computational efficiency on par with LBS. Mohr and Gleicher [2003] who learn an efficient linear and realistic model from example meshes propose another alternative. To deal with the problems of LBS, however, they introduce extra bones to capture effects like muscle bulging. These extra bones increase complexity, are nonphysical, and are nonintuitive for artists. Our blend shapes are simpler, more intuitive, more practical, and offer greater realism. Similarly, Wang et al. [2007] introduce joints related to surface deformation. Their rotational regression approach uses deformation gradients, which then must be converted to a vertex representation.
Learning pose and shape models. The above methods focus on learning poseable singleshape models. What is needed, however, are realistic poseable models that cover the space of human shape variation. Early methods use PCA to characterize a space of human body shapes [Allen et al. 2003; Seo et al. 2003] but do not model how body shape changes with pose. The most successful class of models are based on SCAPE [Anguelov et al. 2005] and represent body shape and posedependent shape in terms of triangle deformations rather than vertex displacements [Chen et al. 2013; Freifeld and Black 2012; Hasler et al. 2009; Hirshberg et al. 2012; PonsMoll et al. 2015] . These methods learn statistical models of shape variation from training scans containing different body shapes and poses. Triangle deformations provide allow the composition of different transformations such as body shape variation, rigid part rotation, and pose dependent deformation. Weber et al. [2007] present an approach that has properties of SCAPE but blends this with example shapes. These models are not consistent with existing animation software.
Hasler et al. [2010] learn two linear blend rigs: one for pose and one for body shape. To represent shape change, they introduce abstract bones that control the shape change of the vertices. Animating a character of a particular shape involves manipulating the shape and pose bones. They learn a base mesh and blend weights but not blend shapes. Consequently, the model lacks realism. Allen et al. [2006] formulate a vertexbased model that has the expressive power of the triangle deformation models so that it can capture a whole range of natural shapes and poses. For a given base body shape, they define a standard LBS model with scattered/ exemplar PSD to model pose deformations, using radial basis functions for scattered data interpolation, shapedependent pose deformations, and a fixed set of carry ing angles. Consequently training it is also complex and requires a good initialization.. They greedily define key angles at which to represent corrective blend shapes and they hold these fixed across all body shapes. A given body shape is parameterized by the vertices of the rest pose, corrective blend shapes (at the key angles), and bone lengths; these comprise a character vector. Given different character vectors for different bod ies, they learn a lowdimensional latent space that lets them generalize character vectors to new body shapes; they learn these parameters from data. However, they had limited data and difficulty with overfitting so they restricted their body shape PCA space. As a result, the model did not generalize well to new shapes and poses. Their model is complex, has few parameters, and is learned from much less data.
OBJECT OF THE INVENTION
It is therefore an object of the invention, to provide a method and a device for learning a model of the body automatically that is both realistic and compatible with existing graphics software. It is a further object of the invention to make the body model as standard as possible so that it can be widely used, while, at the same time, keeping the realism of deformationbased models learned from data.
A BRIEF DESCRIPTION OF THE INVENTION
These objects are achieved by a method and a device according to the independent claims. Advantageous embodiments are defined in the dependent claims. In particular, the invention comprises a Skinned MultiPerson Linear (SMPL) model of the human body that can realistically represent a wide range of human body shapes, can be posed with natural posedependent deformations, exhibits softtissue dynamics, is efficient to animate, and is compatible with existing rendering engines.
The invention provides a human body model that captures body shape and pose variation as well as, or better than, the best previous models while being compatible with existing graphics pipelines and software. To that end, the invention uses standard skinning equations, defines body shape and pose blend shapes that modify the base mesh. The pose blend shapes depend o the elements of body part rotation matrices. The model may be trained on thousands of aligned scans of different people in different poses. The form of the model makes it possible to learn the parameters from large amounts of data while directly minimizing vertex reconstruction error. In one embod iment of the invention, the rest template, joint regressor, body shape model, pose blend shapes, and dynamic blend shapes are learned. Using 4D registered meshes, SMPL may also be extended to model dynamic softtissue deformations as a function of poses over time using an autoregressive model. SMPL can be exported as an FBX file
According to another embodiment of the invention, blend shapes may be learned to correct for the limitations of standard skinning. Different blend shapes for identity, pose, and softtissue dynamics may be additively combined with a rest template before being transformed by blend skinning. The pose blend shapes may be formulated as a linear function of the pose, in particular as linear function of elements of the part rotation matrices. This formulation is different from previous methods [Allen et al.2006; Merry et al.2006; Wang and Phillips 2002] and makes training and animating with the blend shapes simple. Because the elements of rotation matrices are bounded, so are the resulting deformations, helping the invention model to generalize better. The model admits an objective function that penalizes the per vertex disparities between registered meshes and our model, enabling training from data. To learn how people deform with pose, 1786 highresolution 3D scans of different subjects may be used in a wide variety of poses. The template mesh is aligned to each scan to create a training set. The blend weights, posedependent blend shapes, the mean template shape (rest pose), and a regressor from shape to joint locations are optimized to minimize the vertex error of the model on the training set. This joint regressor predicts the location of the joints as a function of the body shape and is critical to animating realistic posedependent deformations for any body shape. All parameters are estimated automatically from the aligned scans.
Linear models of male and female body shape may be learned from the CAESAR da taset [Robinette et al. 2002] (approximately 2000 scans per gender) using principal component analysis (PCA). First, a template mesh is registered to each scan and the data is pose normalized, which is critical when learning a vertexbased shape model. The resulting principal components become body shape blend shapes.
The SMPL model may be extended to capture softtissue dynamics by adapting the Dyna model [PonsMoll et al. 2015]. The resulting DynamicSMPL, or DMPL model, is trained from the same dataset of 4D meshes as Dyna. DMPL, however, is based on vertices instead of triangle deformations. Vertex errors are computed between SMPL and Dyna training meshes, transformed into the rest pose, and use PCA to reduce the dimensionality, producing dynamic blend shapes. A softtissue model is then trained based on angular velocities and accelerations of the parts and the history of dynamic deformations as in [PonsMoll et al. 2015]. Since softtissue dynamics strongly depend on body shape, DMPL may be trained using bodies of varying body mass index and a model of dynamic deformations may be learned that depends of body shape. The surprising result is that, when BlendSCAPE and the inventive model are trained on exactly the same data, the vertexbased model is more accurate and significantly more effi cient to render than the deformation based model. Also surprising is that a relatively small set of learned blend shapes do as good a job of correcting the errors of LBS as they do for DQBS.
Animating softtissue dynamics in a standard rendering engine only requires compu ting the dynamic linear blend shape coefficients from the sequence of poses. Sideby side animations of Dyna and DMPL reveal that DMPL is more realistic. This extension of SMPL illustrates the generality of the inventive additive blend shape approach, shows how deformations can depend on body shape, and demonstrates how the approach provides a extensible foundation for modeling body shape.
SMPL models can be animated significantly faster than real time on a CPU using standard rendering engines. Consequently, the invention addresses an open problem in the field; it makes a realistic learned model accessible to animators. The inventive base template is designed with animation in mind; it has a lowpolygon count, a sim pie vertex topology, clean quad structure, a standard rig, and reasonable face and hand detail (though the hands or face are not rigged here). Models according to the invention can be represented as an Autodesk Filmbox (FBX) file that can be imported into animation systems.
BRIEF DESCRIPTION OF THE FIGURES
Different embodiments of the invention are described in more detail, in relation to the drawing in which
(left) SMPL model (orange) fit to ground truth 3D meshes (gray), (right) Unity 5.0 game engine screenshot showing bodies from the CAESAR dataset animated in real time.
Figure 2 : shows models compared with ground truth. The far right (light gray) mesh is a 3D scan. Next to it (dark gray) is a registered mesh with the same topology as our model. We ask how well different models can approximate this registration. From left to right: (light green) Linear blend skinning (LBS), (dark green) Dualquaternion blend skinning (DQBS), (blue) BlendSCAPE, (red) SMPLLBS, (orange) SMPLDQBS. The zoomed regions highlight differences between the models at the subjects right elbow and hip. LBS and DQBS produce serious artifacts at the knees, elbows, shoulders and hips. BlendSCAPE and both SMPL models do similarly well at fitting the data.
Figure 3 shows a SMPL model according to an embodiment of the invention, (a)
Template mesh with blend weights indicated by color and joints shown in white, (b) With identitydriven blendshape contribution only; vertex and joint locations are linear in shape vector β . (c) With the addition of of pose blend shapes in preparation for the split pose; note the expansion of the hips, (d) Deformed vertices reposed by dual quaternion skinning for the split pose. shows sample registrations from the multipose dataset. shows sample registrations from the multishape dataset. shows initialization of joints and blend weights. Discrete part segmentation in (a) is diffused to obtain initial blend weights, Wi, in (b). Initial joint centers are shown as white dots. shows a joint regression, (left) Initialization. Joint locations can be influenced by locations on the surface, indicated by the colored lines. It is assumed that these influences are somewhat local, (right) Optimized. After optimization a sparse set of vertices and associated weights influencing each joint are found. shows shape blend shapes. The first three shape principal components of body shape are shown. PCi and PC2 vary from 2 to +2 standard deviations from left to right, while PC3 varies from 5 to +5 standard deviations to make the shape variation more visible. Joint locations (red dots) vary as a function of body shape and are predicted using the learned regressor, J. shows a cumulative relative variance of the CAESAR dataset explained as a function of the number of shape coefficients. For SMPL the variance is in vertex locations, while for BlendSCAPE it is in triangle deformations.
Figure 10 : shows a model fitting with intermediate stages. We fit both BlendSCAPE (blue) and SMPLLBS, ( ?, ø), (red) to registered meshes by optimizing pose and shape. T + B_{S} [β] shows the estimated body shape and Τ_{ρ}(β,θ) shows the effects of posedependent blend shapes. Here we show SMPLLBS, because TP shows more variation due to pose than SMPLDQBS.
Figure li : Model generalization indicates how well one can fit an independent registration. Mean absolute vertex error versus the number of shape coefficients used. Figure 12 : Pose generalization error indicates how well a fitted shape generalizes to new poses.
Figure 13 : shows animating SMPL. Decomposition of SMPL parameters into pose and shape: Shape parameters, β , vary across different subjects from left to right, while pose parameters, Θ vary from top to bottom for each subject.
Figure 14 : shows how performance of SMPL and BlendSCAPE vary with the number of body shape coefficients used. Performance shown here is from a 2014 Macbook Pro.
Figure 15: shows DMPL model of softtissue motion according to an embodiment of the invention. Above, two frames of a "running" sequence of a male subject from the Dyna dataset, below two frames of a jumping jacks se quence of a female subject from the Dyna dataset. From left to right:
SMPL, DMPL, and the dynamic blend shapes added to the base body shape. While SMPL models deformations due to pose well it does not model dynamic deformations. DMPL predicts dynamic deformations from motion and body shape, resulting in more life like animations.
Figure 16 : shows parameterizing pose blend shapes, (a) Pose blend shapes parameterized by Euler angles cause significant problems, (b) the proposed parameterization allows the head to rotate in either direction with natural deformations. DETAILED EMBODIMENTS
Figure 1 shows a realistic learned model of human body shape and pose according to a first embodiment of the invention that is compatible with existing rendering engines and allows animator control using standard graphics tools, (left) SMPL model (orange) fit to ground truth 3D meshes (gray), (right) Unity 5.0 game engine screenshot showing bodies from the CAESAR dataset animated in real time. Figure 2 compares models with ground truth. More particularly, the inventive model is trained in various forms and compared quantitatively to a BlendSCAPE model [Hirshberg et al. 2012], trained with exactly the same data. The models are both evaluated qualitatively with animations and quantitatively using meshes that were not used for training. SMPL and BlendSCAPE are fit to these meshes and then the vertex errors are compared. Two main variants of SMPL are explored, one using linear blend skinning (LBS) and the other with DualQuaternion blend skinning (DQBS).
The far right (light gray) mesh is a 3D scan. Next to it (dark gray) is a registered mesh with the same topology as the inventive model. The comparison shows how well dif ferent models can approximate this registration. From left to right: (light green) Linear blend skinning (LBS), (dark green) DualQuaternion blend skinning (DQBS), (blue) BlendSCAPE, (red) SMPLLBS, (orange) SMPLDQBS. The zoomed regions highlight differences between the models at the subject's right elbow and hip. LBS and DQBS produce serious artifacts at the knees, elbows, shoulders and hips. BlendSCAPE and both SMPL models do similarly well at fitting the data.
Surprisingly, the vertexbased, skinned, model according to the invention is actually more accurate than a deformationbased model like BlendSCAPE trained on the same data.
Figure 3 shows an illustration of the skinned multiperson linear model according to the invention. Like SCAPE, the SMPL model decomposes body shape into identity dependent shape and nonrigid posedependent shape; unlike SCAPE, the invention takes a vertexbased skinning approach that uses corrective blend shapes. A single blend shape is represented as a vector of concatenated vertex offsets. The method be gins with an artist created mesh with N =6890 vertices and K =23 joints. The mesh has the same topology for men and women, spatially varying resolution, a clean quad structure, a segmentation into parts, initial blend weights, and a skeletal rig. The part segmentation and initial blend weights are shown in figure 6.
Following standard skinning practice, the model is defined by a mean template shape represented by a vector of N concatenated vertices T e 9 ^{3Ai} in the zero pose, Θ^{*} ; a set of blend weights, W G <R^{NxK} , (Fig. 3(a)); a blend shape function, Β_{5} β): ^→¾^{M} , that takes as input a vector of shape parameters, β , (Fig. 3(b)) and outputs a blend shape sculpting the subject identity; a function to predict K joint locations (white dots in Fig. 3(b)), /( ?): — > 'R^{3K} as a function of shape parameters, β ; and a pose dependent blend shape function, Β_{ρ} (θ) : 91^— > 9ί^{3Αί} , that takes as input a vector of pose parameters, θ , and accounts for the effects of posedependent deformations (Fig. 3(c)). The corrective blend shapes of these functions are added together in the rest pose as illustrated in (Fig. 3(c)). Finally, a standard blend skinning function W (·)
(linear or dualquaternion) is applied to rotate the vertices around the estimated joint centers with smoothing defined by the blend weights. The result is a model,
Μ(β, θ; ): %^→%^{3N} , that maps shape and pose parameters to vertices (Fig. 3(d)). Here φ represents the learned model parameters described below.
Both LBS and DQBS skinning methods will be used below. In general the skinning method can be thought of as a generic black box. Given a particular skinning method our goal is to learn φ to correct for limitations of the method so as to model training meshes. Note that the learned pose blend shapes both correct errors caused by the blend skinning function and static softtissue deformations caused by changes in pose.
Blend skinning. To fix ideas and define notation, the LBS version is presented as it makes exposition clear (the DQBS version of SMPL only requires changing the skinning equation). Meshes and blend shapes are vectors of vertices represented by bold capital letters (e.g. X) and lowercase bold letters (e.g. x_{t} e 9 ^{3} ) are vectors representing a particular vertex. The vertices are sometimes represented in homogeneous coor dinates. The same notation is used for a vertex whether it is in standard or homogeneous coordinates as it should always be clear from the context which form is needed.
The pose of the body is defined by a standard skeletal rig, where w_{k} e 9 ^{3} denotes the axisangle representation of the relative rotation of part k with respect to its parent in the kinematic tree. In the present embodiment, the rig has K = 23 joints, hence a pose θ ^{=} [^o _{>}·■· ¾ f is defined by Θ = 3x23 + 3 = 72 parameters; i.e. 3 for each part plus w
3 for the root orientation. Let w =  denote the unit norm axis of rotation. Then the axis angle for every joint j is transformed to a rotation matrix using the Rodrigues formula. e p (l . )= I + \W ; )+ w' 1  cosl \\w (1) where w is the skew symmetric matrix of the 3vector w and / is the 3 x 3 identity matrix. Usi e standard linear blend skinning function
:
N takes vertices in the rest pose, T , joint locations, J, a pose, Θ ; and the blend weights, ω, and returns the posed vertices. Each vertex t_{t} in T is transformed into t (both column vectors in homogeneous coordinates) as= G_{k} (e, j]G_{k} (e jY (3)
where w_{k i} is an element of the blend weight matrix W, representing how much the rotation of part k effects the vertex i, exp(^_{;} ) is the local 3 x 3 rotation matrix corresponding to joint j, G k (e, j) is the world transformation of joint k, and G k (θ, j) is the same transformation after removing the transformation due to the rest pose, Θ . Each 3element vector in J corresponding to a single joint center, j, is denoted jj. Finally, A(k) denotes the ordered set of joint ancestors of joint k. Note, for compatibility with existing rendering engines, it is assumed that W is sparse and at most four parts are allowed to influence a vertex.Many methods have modified equation (2) to make skinning more expressive. For example MWE [Wang and Phillips 2002] replaces G_{k} (&, j) with a more general affine transformation matrix and replaces the scalar weight with a separate weight for every element of the transformation matrix. Such changes are expressive but are not compatible with existing animation systems.
To maintain compatibility, the basic skinning function may be kept and instead the template may be modified in an additive way and a function is learned to predict joint locations. The model,
where B_{s} [β j and B_{p} [Θ J are vectors of vertices representing offsets from the template. These are referred to as shape and pose blend shapes respectively.
Given this definition, a vertex t_{i} is transformed according to
where b_{S i} fil), b_{P i} (&) are vertices in Β_{8} (β) and Β_{ρ} (θ) respectively and represent the shape and pose blend shape offsets for the vertex t_{i} . Hence, the joint centers are now a function of body shape and the template mesh that is deformed by blend skinning is now a function of both pose and shape. Shape blend shapes. The body shapes of different people are represented by a linear function Bsfl_{s} (¾ s) =∑A5, (8)
where β = β^{η}· · the number of linear shape coefficients, and the
represent orthonormal principal components of shape displacements. Let
S = ^1 > · · · ·> e 9t ^{1 1} be the matrix of all such shape displacements. Then the lin ear function B_{s} [J3;s) is fully defined by the matrix S, which is learned from registered training meshes
Notationally, the values to the right of a semicolon represent learned parameters, while those on the left are parameters set by an animator. For notational convenience, the learned parameters are often omitted when they are not explicitly being optimized in training.
Figure 3(b) illustrates the application of these shape blend shapes to the template T to produce a new body shape.
Pose blend shapes. R denotes: R ' — > 9ΐ^{9ί} a function that maps a pose vector Θ to a vector of concatenated part relative rotation matrices, exp(w>). Given that the rig has 23 joints, R(#) is a vector of length (23 x 9 = 207). Elements of R(#) are functions of sines and cosines (Eq. (19)) of joint angles and therefore R(#) is nonlinear with Θ .
This formulation differs from previous work in that the effect of the pose blend shapes is defined to be linear in R^{*} {§)= (R(^) R(^^{*} )), where Θ^{*} denotes the rest pose. Let
R_{n} (e) denote the n_{t}h element of R(#), then the vertex deviations from the rest template are
where the blend shapes, P_{N} e 91 , are again vectors of vertex displacements. Here P = [P_{L} , . . . , P_{9K} ] e y{^{3Nx9K} i_{s a} matrix of all 207 pose blend shapes. In this way, the pose blend shape function B_{p} (θ; p) is fully defined by the matrix P.Subtracting the rest pose rotation vector R(#) guarantees that the contribution of the pose blend shapes is zero in the rest pose, which is important for animation.
Joint locations. Different body shapes have different joint locations. Each joint is represented by its 3D location in the rest pose. It is critical that these are accurate, otherwise there will be artifacts when the model is posed using the skinning equation.
For that reason, the joints are defined as a function of the body shape, β , j(]3; Z,T, s) = z(T + B_{s} (j3; s)) (10) where 3 is a matrix that transforms rest vertices into rest joints. The regression matrix, 3 , is learned from examples of different people in many poses. This matrix mod els which mesh vertices are important and how to combine them to estimate the joint locations.
SMPL model. One can now specify the full set of model parameters of the SMPL model as = ί ,W, S, J, P}. Once learned they are held fixed and new body shapes and poses are created and animated by varying β and Θ respectively. Then the SMPL model i
and hence each vertex is transformed as where ΐ_{Ρ} β, θ;Τ , S,p)=t, +∑ fi_{m} s_{mJ} +∑ k (*) *„ (F ))p_{nJ} (13) m=l =1 represents the vertex i after applying the blend shapes and where s_{m i} , p_{n i} e 9t^{3} are the elements of the shape and pose blend shapes corresponding to template vertex t_{i} .
Below, experiments with both LBS and DQBS are described, wherein the parameters are trained for each. These models are referred to as SMPLLBS and SMPLDQBS; SMPLDQBS is the default model, and SMPL is used as shorthand to mean SMPL DQBS.
TRAINING
The SMPL model parameters are trained to minimize reconstruction error on two da tasets. Each dataset contains meshes with the same topology as our template that have been aligned to highresolution 3D scans using [Bogo et al. 2014]; these aligned mesh es are called "registrations"; The multipose dataset consists of 1786 registrations of 40 individuals (891 registrations spanning 20 females, and 895 registrations spanning 20 males); a sampling is shown in figure 4. The multishape dataset consists of registrations to the CAESAR dataset [Robinette et al. 2002], totaling 1700 registrations for males and 2100 for females; a few examples are shown in figure 5. The j^{th} mesh in the multipose dataset is denoted as Vf and the j^{th} mesh in the multishape dataset as
According to the invention, the parameters = τ ,W , S, J, P} are trained to minimize a vertex reconstruction error on the datasets. Because the model decomposes shape and pose, these are trained separately, simplifying optimization. First, { 3 ,ω,Ρ^ is trained using a multipose dataset and then trained using our multishape dataset. Separate models are trained for men and women (i.e. 0_{m} and _{{} )
First, the multipose dataset is used to train {3 ,i*),P}. To this end, one needs to com pute the rest templates, T^{p} , and joint locations, J^{p} , for each subject, i, as well as the pose parameters, θ. , for each registration, j, in the dataset. The alternates framing method between optimizing registration specific parameters , subjectspecific pa
Ϊ ^{Λ} 1
T_{t} , J_{;} j, and global parameters {W,P}. Then the matrix, 3 , is learned to regress from subjectspecific vertex locations, T_{t} , to subjectspecific joint locations, J _{l} : . To achieve all this, one minimizes an objective function consisting of a data term, E_{D}, and several regularization terms {Ej, E_{Y}, E_{P} , E_{w} } defined below.
The data term penalizes the squared Euclidean distance between registration vertices and model vertices
where Θ = s(j) is the subject index corresponding to registration j, P_{reg}
are the number of meshes in the pose trainings set, f^{p} = ^{sub}} , J^{P} = {j^{P} } ^{subj} are i = l i = l the sets of all rest poses and joints, and P_{su}bj is the number of subjects in the pose training set. The method estimates 207 x 3 x 6890 = 4,278,690 parameters for the pose blend shapes, P, 4 x 3 x 6890 = 82,680 parameters for the skinning weights, W, and 3 x 6890 x 23 x 3 = 1,426,230 for the joint regressor matrix, 3 . To make the estimation well behaved, we regularize by making several assumptions. A symmetry regulariza tion term, Εγ, penalizes leftright asymmetry for J^{p} and f^{p} . where A_{y} = 100 , and where U(T) finds a mirror image of vertices T, by flipping across the sagittal plane and swapping symmetric vertices. This term encourages symmetric template meshes and, more importantly, symmetric joint locations. Joints are unobserved variables and along the spine they are particularly difficult to localize. While models trained without the symmetry term produce reasonable results, enforcing symmetry produces models that are visually more intuitive for animation.The model is hand segmented into 24 parts (Fig. 6). This segmentation is used to compute an initial estimate of the joint centers and a regressor Ji from vertices to these centers. This regressor computes the initial joints by taking the average of the ring of vertices connecting two parts. When estimating the joints for each subject, they are regularized to be close to this initial prediction:
To help prevent overfitting of the posedependent blend shapes, they are regularized towards zero
E_{P} (p) = p where ^{■} denotes the Frobenius norm. Replacing the quadratic penalty with an Li penalty would encourage greater sparsity of the blend shapes.
The blend weights are also regularized towards the initial weights, Wi, shown in figure 6: 2
Ev iw^ fv w,
The initial weights are computed by simply diffusing the segmentation.
Altogether, the energy for training {ω , P } is as follows:
E_{D} + Λ_{γ}Ε_{γ} + A_{J} E_{J} + λ_{ρ}Ε_{ρ} + Ε
where Λ_{γ} = 100, λ_{}} = 100 and λ_{ρ} = 25 . These weights were set empirically. The model has a large number of parameters and the regularization helps prevent overfit ting. As the size of the training set grows, so does the strength of the data term, effectively reducing the influence of the regularization terms. The experiments below with heldout test data suggest that the learned models are not overfit to the data and generalize well.
Joint regressor. Optimizing the above gives a template mesh and joint locations for each subject, but one wants to predict joint locations for new subjects with new body shapes. To that end, the regressor matrix 3 is learned to predict the training joints from the training bodies. Several regression strategies were tried; what was found to work best, was to compute J using nonnegative least squares [Lawson and Hanson 1995] with the inclusion of a term that encourages the weights to add to one. This approach encourages sparsity of the vertices used to predict the joints. Making weights positive and add to one discourages predicting joints outside the surface. These constraints enforce the predictions to be in the convex hull of surface points. Figure 7 shows the nonzero elements of the regression matrix, illustrating that a sparse set of surface vertices are linearly combined to estimate the joint centers. According to the invention, the shape space is defined by a mean and principal shape directions {τ , ^}. It is computed by running PCA on shape registrations from our multishape database after pose normalization. Pose normalization transforms a raw reg istration V ^{s} into a registration, Γ^{5} , in the rest pose, Θ^{*} . This normalization is critical to ensure that pose and shape are modeled separately.
To posenormalize a registration, V . s , first its pose is estimated. ^{Λ} P ^{Λ} P
T and J _{μ} denote the mean shape and mean joint locations from the multipose database respectively. Let
V_{j e}€ 9ί^{3} denote an edge of the model and of the registration respectively. An edge is obtained by subtracting a pair of neighboring vertices. To es timate the pose using an average generic shape T , the following sum of squared edge differences is minimized so that 0. =+ B_{P}(e, _{M} ^{p},e,w)v[_{e} (15)
where the sum is taken over all edges in the mesh. This allows us to get a good estimate of the pose without knowing the subject specific shape.Once the pose 0. is known we solve for T_{j} by minimizing
This computes the shape that, when posed, matches the training registration. This shape is the posenormalized shape.
We then run PCA on { ^ } ^{subj}^ to obtain {τ , s}. This procedure is designed to maximize the explained variance of vertex offsets in the rest pose, given a limited number of shape directions.
The optimization of pose is critically important when building a shape basis from vertices. Without this step, pose variations of the subjects in the shape training dataset would be captured in the shape blend shapes. The resulting model would not be de composed properly into shape and pose. Note also that this approach contrasts with SCAPE or BlendSCAPE, where PCA is performed in the space of pertriangle deformations. Unlike vertices, triangle deformations do not live in a Euclidean space [Freifeld and Black 2012] . Hence PCA on vertices is more principled and is consistent with the registration data term, which consists of squared vertex disparities.
Figure 8 visualizes the first three shape components. The figure also shows how the joint locations change with the changes in body shape. The joint positions are shown by the spheres and are computed from the surface meshes using the learned joint re gression function. The lines connecting the joints across the standard deviations illustrate how the joint positions vary linearly with shape.
Figure 9 shows the relative cumulative variance of SMPL and BlendSCAPE. While SMPL requires many fewer PCs to account for the same percentage of overall variance, the variance is different in the two cases: one is variance in vertex locations and the other is variance in triangle deformations. Explained variance in deformations does not directly translate into explained variance in vertex locations. While this makes the two models difficult to compare precisely, triangle deformations have many more degrees of freedom and it is likely that there are many deformations that produce visual ly similar shapes. A model requiring fewer components is generally preferable.
Pose parameters 0. in Eq. (14) are first initialized by minimizing the difference between the model and the registration edges, similar to Eq. (15) using an average tern are estimated in an alternating manner to mini
J ,T J. We then run PCA on pose nor malized subjects The final model is defined by
{j,W, P,T ,s}. Note that all training parameters except for τ ,s} are found with gra dientbased dogleg minimization [Nocedal and Wright 2006] . Gradients are computed with automatic differentiation using the the Chumpy framework [Loper and Black 2014] .
EVALUATION TTwwoo ttyyppeess ooff eerrrroorr aarree eevvaalluuaatteedd.. MMooddeell ggeenneerraalliizzaattiioonn iiss tthhee aabbiilliittyy ooff tthhee mmooddeell ttoo ffiitt ttoo mmeesshheess ooff nneeww ppeeooppllee aanndd ppoosseess;; tthhiiss tteessttss bbootthh sshhaappee aanndd ppoossee bblleenndd sshhaappeess.. PPoossee ggeenneerraalliizzaattiioonn iiss tthhee aabbiilliittyy ttoo ggeenneerraalliizzee aa sshhaappee ooff aann iinnddiivviidduuaall ttoo nneeww ppoosseess ooff tthhee ssaammee ppeerrssoonn;; tthhiiss pprriimmaaririllyy tteessttss hhooww wweellll ppoossee bblleenndd sshhaappeess ccoorrrreecctt sskkiinnnniinngg aarrttiiffaaccttss 55 aanndd ppoosseeddeeppeennddeenntt ddeeffoorrmmaattiioonnss.. BBootthh aarree mmeeaassuurreedd bbyy mmeeaann aabbssoolluuttee vveerrtteexxttoo vveerrtteexx ddiissttaannccee bbeettwweeeenn tthhee mmooddeell aanndd tteesstt rreeggiissttrraattiioonnss.. FFoorr tthhiiss eevvaalluuaattiioonn wwee uussee 112200 rreeggiisstteerreedd mmeesshheess ooff ffoouurr wwoommeenn aanndd ttwwoo mmeenn ffrroomm tthhee ppuubblliicc DDyynnaa ddaattaasseett [[DDyynn 22001155]].. TThheessee mmeesshheess ccoonnttaaiinn aa vvaarriieettyy ooff bbooddyy sshhaappeess aanndd ppoosseess.. A Allll mmeesshheess aarree iinn aalliiggnnmmeenntt wwiitthh oouurr tteemmppllaattee aanndd nnoonnee wweerree uusseedd ttoo ttrraaiinn oouurr mmooddeellss.. FFiigguurree 1100 ((ggrraayy)) 1100 sshhoowwss ffoouurr eexxaammpplleess ooff tthheessee rreeggiisstteerreedd mmeesshheess..
SSMMPPLLLLBBSS aanndd SSMMPPLLDDQQBBSS aarree eevvaalluuaatteedd aanndd ccoommppaarreedd wwiitthh aa BBlleennddSSCCAAPPEE mmooddeell [[HHiirrsshhbbeerrgg eett aall.. 22001122]] ttrraaiinneedd ffrroomm eexxaaccttllyy tthhee ssaammee ddaattaa aass tthhee SSMMPPLL mmooddeellss.. TThhee kkiinneemmaattiicc ttrreeee ssttrruuccttuurree ffoorr SSMMPPLL aanndd tthhee BBlleennddSSCCAAPPEE mmooddeell aarree tthhee ssaammee:: tthheerreeffoorree 1155 tthhee ssaammee nnuummbbeerr ooff ppoossee ppaarraammeetteerrss iiss uusseedd.. TThhee mmooddeellss aarree aallssoo ccoommppaarreedd uussiinngg tthhee ssaammee nnuummbbeerr ooff sshhaappee ppaarraammeetteerrss..
TToo mmeeaassuurree mmooddeell ggeenneerraalliizzaattiioonn eeaacchh mmooddeell iiss ffiirrsstt ffiitt ttoo eeaacchh rreeggiisstteerreedd mmeesshh,, ooppttii¬mmiizziinngg oovveerr sshhaappee ββ aanndd ppoossee ΘΘ ttoo ffiinndd tthhee bbeesstt ffiitt iinn tteerrmmss ooff ssqquuaarreedd vveerrtteexx ddiiss
2200 ttaanncceess.. FFiigguurree 1100 sshhoowwss ffiittss ooff tthhee SSMMPPLLLLBBSS ((rreedd)) aanndd BBlleennddSSCCAAPPEE ((bblluuee)) mmooddeellss ttoo tthhee rreeggiisstteerreedd mmeesshheess.. BBootthh ddoo aa ggoooodd jjoobb ooff ffiittttiinngg tthhee ddaattaa.. TThhee ffiigguurree aallssoo sshhoowwss hhooww tthhee mmooddeell wwoorrkkss.. IIlllluussttrraatteedd aarree tthhee eessttiimmaatteedd bbooddyy sshhaappee,, TT ++ BB_{SS} [[ββ]] aanndd tthhee eeffffeecctt ooff aappppllyyiinngg tthhee ppoossee bblleenndd sshhaappeess,,
25 For pose generalization, for each individual, one of the estimated body shapes from the generalization task is selected, and then the pose, Θ is optimal for each of the other meshes of that subject, keeping the body shape fixed. The assumption behind pose generalization is that, if a model is properly decomposed into pose and shape, then the model should be able to fit the same subject in a different pose, without readjusting 30 shape parameters. The pose blend shapes are trained to fit observed registrations. As such, they correct for problems of blend skinning and try to capture posedependent deformations. Since the pose blend shapes are not dependent on body shape, they capture something about the average deformations in the training set. Figures 11 and 12 show the error of the SMPL models and BlendSCAPE as a function of the number of body shape coefficients used. The differences between SMPL and BlendSCAPE are small (on the order of 0.5mm) but SMPL is more accurate on average. Remarkably, SMPLLBS and SMPLDQBS are essentially identical in model gen eralization and SMPLLBS is actually slightly better at pose generalization. This is surprising because the pose blend shapes have much more to correct with LBS. Possibly the simplicity of LBS helps with generalization to new poses.
This analysis is important because it says that users can choose the simpler and faster LBS model over the DQBS model.
The plots also show how well standard LBS fits the test data. This corresponds to the SMPLLBS model with no pose blend shapes. Not surprisingly, LBS produces much higher error than either BlendSCAPE or SMPL. LBS is not as bad in Fig. 11 because here the model can vary body shape parameters, effectively using changes in identity to try to explain deformations due to pose. Figure 12 uses a fixed body shape and consequently illustrates how LBS does not model posedependent deformations realistically. Note that here one does not retrain a model specifically for LBS and expect such a model would be marginally more accurate.
The pose blend shapes in SMPL are not sparse in the sense that a rotation of a part can influence any vertex of the mesh. With sufficient training data sparsity may emerge from data; e.g. the shoulder corrections will not be influenced by ankle motions. To make hand animation more intuitive, and to regularize the model to prevent longrange influences of joints, one can manually enforce sparsity. To this end, one may train a sparse version of SMPL by using the same sparsity pattern used for blend weights. In particular, a vertex deviation is allowed to be influenced by at most 4 joints. Since every joint corresponds to a rotation matrix, the pose blend shape corresponding to any given vertex will be driven by 9 x 4 numbers as opposed to 9 x 23.
This model is referred to as SMPLLBSSparse in Figs. 11 and 12. It is consistently less accurate than the regular SMPLLBS model but may still be useful to animators. This suggests that SMPLLBS is not overfit to the training data and that sparseness reduces the capacity of the model. The sparse model sometimes produces slight discontinuities at boundaries were vertices are influenced by different joint angles. Other strategies to enforce sparsity could be adopted, such as using an Li prior or enforcing smoothness in the pose blend shapes. These approaches, however, complicate the training process.
Figure 13 illustrates the decomposition of shape parameters β and pose parameters Θ in SMPL. Pose is held constant from left to right across each row while varying the shape. Similarly, the shape of each person is held constant while varying the pose from top to bottom in each column. The bodies are reposed using poses from the CMU mocap database [CMU 2000]. The posedependent deformations look natural through a wide range of poses, despite very different body shapes. This illustrates that the joint regression works well and that the blend shapes are general.
The runtime cost of SMPL is dominated by skinning and blendshape multiplication. Performance of a CPU based implementation of the invention model, and a comparison against BlendSCAPE, is shown in Fig. 14. The plot shows the time needed to gen erate the vertices. The BlendSCAPE rendering makes use of multiple cores, while the SMPL rendering does not; this is why the system time for BlendSCAPE is higher than the wallclock time. Shown is the cost of changing body shape. For most applications, this is done once and the shape is then held fixed. The cost of animating the mesh then comes from only the pose blend shapes; this cost corresponds to o shape coefficients.
For meshes with the same number of vertices, SCAPE will always be slower. In SMPL each blend shape is of size 3 N, requiring that many multiplications per shape. SCAPE uses triangle deformations with 9 elements per triangle and there are roughly twice as many triangles as vertices. This results in roughly a factor of 6 difference between SMPL and SCAPE in terms of basic multiplications.
COMPATIBILITY WITH RENDERING ENGINES
Because SMPL is built on standard skinning, it is compatible with existing 3D animation software. In particular, for a given body shape, one may generate the subject specific restpose template mesh and skeleton (estimated joint locations) and export SMPL as a rigged model with pose blend shapes in Autodesk's Filmbox (FBX) file format, giving crossplatform compatibility. The model loads as a typical rigged mesh and can be animated as usual in standard 3D animation software. Pose blend weights can be precomputed, baked into the model, and exported as an animated FBX file. This kind of file can be loaded into animation packages and played directly. The animated FBX files were tested in Maya, Unity, and Blender. Pose blend weights can also be computed on the fly given the pose, θ_{{} , at time f. To enable this, scripts may be provided that take the joint angles and compute the pose blend weights. Loading and animating SMPL was tested in Maya 2013, 2014 and 2015. The animator can animate the model using any of the conventional animation methods typically used in Maya. The pose blend shape values can be viewed and/or edited manually within Maya if desired. SMPL was also tested in Unity. In SMPL, the blend weights range from 1 to +1 while in Unity they range form o to 1. Consequently, the weights are scaled and recentered for compatibility. For Unity, each blend shapes can be split into two  one positive and one negative. If the SMPL blend shape should be positive, then a script tells unity that the negative blend shape has zero weight (and vice versa for negative values). To speed computation for realtime graphics, blend shape weights that are close to zero can be set to zero and not used.
DMPL: DYNAMIC SMPL
While SMPL models static softtissue deformations with pose it does not model dy namic deformations that occur due to body movement and impact forces with the ground. Given 4D registrations that contain softtissue dynamics, we fit them by optimizing only the pose of a SMPL model with a personalized template shape. Displacements between SMPL and the observed meshes correspond to dynamic soft tissue motions. To model these, a further embodiment of the invention introduces a new set of additive blend shapes called dynamic blend shapes. These additional displacements are correlated with velocities and accelerations of the body and limbs rather than with pose.
Let (j>_{t} = [θ_{(} ,θ_{(} ,v_{t} ,a_{t} , 2 ] denote the dynamic control vector at time f. It is com posed of pose velocities and accelerations #_{f} ,#_{f} e , root joint velocities and accelerations v_{f} ,a_{f} e ¾^{3} and a history of two vectors of predicted dynamic coefficients
4i ' 2 ε 91^ , describes below. The previous linear formulation is extended by adding the dynamic blend shape function, B_{D} (φ β), to the other blend shapes in the rest pose before applying the skinning function. The shape in the zero pose becomes Τ_{Β} {^ , = Τ + Β β)+ Β_{ρ}{θ)+ Β_{Β} { β) (16) as illustrated in figures 15. Here, Β_{Β} (φ β) takes as input the dynamic control vector at time f, and shape coefficients β , and predicts vertex offsets in the rest pose.
Whereas in [PonsMoll et al. 2015] dynamic deformations are modeled using triangle deformations, DMPL models deformations in vertex space. The method according to the present embodiment of the invention build male and female models using roughly 40, 000 registered male and female meshes from [Dyn 2015]. The pose in each frame and the displacements between SMPL and the registration are computed. Using PCA, one obtains a mean and the dynamic blend shapes, μ_{Β} e 9t^{3Ai} and De ¾^{3A¾}^ respectively. We take δ = 300 principal components as in Dyna. Dynamic deformations vary significantly between subjects based on their body shape and fat distribution. To capture this, we train a model that depends on the body shape parameters β as in Dyna. namic blend shapes are then predicted using
Β_{Β} (φ β; Ό) = μ_{Β} + Ό (φ β) analogous to Eq. (22) in [PonsMoll et al. 2015] where /(^{■}) is a function that takes as input a dynamic control vector, (j>_{t} , and predicts the vector of dynamic shape coefficients, S_{t} . This formulation of softtissue displacements in terms of dynamic blend shapes means that, unlike Dyna, this inventive model remains compatible with current graphics software. To animate the model, one only needs a script to compute the coefficients, S_{r} = /(φ ,β), from the pose sequence and body shape. The DMPL model produces softtissue dynamics that appear more realistic than those of Dyna. DISCUSSION
Importantly, the pose training data spans a range of body shapes enabling to learn a good predictor of joint locations. Second, training all the parameters (template shape, blend weights, joint regressor, shape/pose/dynamic blend shapes) to minimize vertex reconstruction error is important to obtain a good model. Here the simplicity of the model is an advantage as it enables training everything with large amounts of data.
In contrast to the scattereddata interpolation methods, the blend shapes are learned from a large set of training meshes covering the space of possible poses and learn a simpler function relating pose to blendshape weights. In particular, the inventive function is linear in the elements of the part rotation matrices. The larger support of the learned linear functions as opposed to radial basis functions allows the model to generalize to arbitrary poses; in addition the simple linear form makes it fast to ani mate in a game engine without baking in the weights. Because elements of a rotation matrix are constrained, the model cannot "blow up"; when generalizing outside the training set.
SMPL is an additive model in vertex space. In contrast, while SCAPE also factors de formations into shape and pose deformations, SCAPE multiplies the triangle deformations. With SCAPE a bigger person will have bigger posedependent deformations even though these deformations are not learned for different body shapes. Despite this, the experiments show that, the SCAPE approach is less accurate at generalizing to new shapes. Ideally one would have enough pose data from enough different people to learn a true bodyshape dependent pose deformation space. DMPL, where deformations depend on body shape, shows that this is possible.
Models based on the statistics of triangle deformations have dominated the recent literature [Anguelov et al. 2005; Chen et al. 2013; Freifeld and Black 2012; Hasler et al. 2009] . Such models are not trained to reproduce their training registrations directly. Instead, they are trained to reproduce the local deformations that produced those registrations. Part of the tractability of training these models comes from the ability to train deformations independently across triangles. As a result, long range distances and relationships are not preserved as well as local relationships between vertices. An advantage of vertex based models (such as SMPL and [Allen et al. 2006]) is that they can be trained to minimize the mean squared error between the model and training vertices. One could train a SCAPE model to minimize vertex error in global coordinates, but the inner loop of the optimization would involve solving a leastsquares problem to reconstruct vertices from the deformations. This would significantly in crease the cost of optimization and make it difficult to train the model with large amounts of data.
The key to SMPL's performance is to make the blend shapes a linear function of the elements of R^{*} (#). This formulation, sufficient training data, and a good optimization strategy make it possible to learn the model.
In a further embodiment of the invention, pose blend shapes may be driven linearly from other features, such as raw θ , simple polynomials of θ , and trigonometric functions (sin, cos) of Θ . Using raw Θ has limitations because the values vary between  π and π . Imagine a twist of the neck (Fig. 16), which produces negative and positive angles about the vertical axis. Standard LBS will cause the neck to shrink as it rotates in either direction. To counteract this, a blend shape is required that increases the neck volume no matter which direction it rotates. In general the raw rotations may be replaced with any functions of rotations and used to weight the blend shapes; for example, normalized quaternions.
The posedependent offsets of the basic SMPL model are not dependent on body shape. It is surprising how well SMPL works without this, but the general approach would likely not work if a space of nonrealistic animated characters were modeled, in which body part scales vary widely, or a space of humans that includes infants and adults. However, this limitation may be addressed by training a more general function that takes elements of R^{*} (#) together with β to predict the blend shape coefficients.
The dynamic blend shape coefficients of DMPL already depend on body shape and therefore the same approach can be used to make the pose blend shapes depend on body shape. This does not significantly complicate the model or runtime behavior, but may only require more training data. As described, the basic SMPL model is a function of joint angles and shape parameters only: it does not model breathing, facial motion, muscle tension, or any changes independent of skeletal joint angles and overall shape. These can be learned as additional additive blend shapes (as with DMPL) if the appropriate factored data is available (cf. [Tsoli et al. 2014]).
While the segmentation of the template into parts, the topology of the mesh, and the zero pose are normally defined in the previous embodiments, these can also be learned.
SMPL uses 207 pose blend shapes. In same cases, this may be reduced by performing PCA on the blend shapes, reducing the number of multiplications and consequently increasing rendering speed. Also, the dynamic model uses PCA to learn the dynamic blend shapes but one may also learn the elements of these blend shapes directly as done for the pose blend shapes. Finally, instead of fitting the model to registered meshes one may also fit it to mocap marker data (cf. MoSh [Loper et al. 2014]), depth data, or video data.
References
ALLEN, B., CURLESS, B., AND POPOVI'C , Z. 2002. Articulated body defor mation from range scan data. ACM Trans. Graph. (Proc. SIGGRAPH) 21, 3 (July), 612619.
ALLEN, B., CURLESS, B., AND POPOVI'C, Z. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. (Proc. SIGGRAPH) 22, 3, 587594.
ALLEN, B., CURLESS, B., POPOVI'C, Z., AND HERTZMANN, A. 2006. Learning a correlated model of identity and posedependent body shape variation for realtime synthesis. In Proceedings of the 2006 ACM SIG GRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, AirelaVille, Switzerland, Switzerland, SCA Ό6, 147156.
ANGUELOV, D., SRINIVASAN, P., ROLLER, D., THRUN, S., RODGERS, J., AND DAVIS, J. 2005. SCAPE: Shape Completion and Animation of PEople. ACM Trans. Graph. (Proc. SIGGRAPH 24, 3, 408416.
BARAN, I., AND POPOVI'C , J. 2007. Automatic rigging and animation of 3D characters. ACM Trans. Graph. (Proc. SIGGRAPH) 26, 3 (July).
BOGO, F., ROMERO, J., LOPER, M., AND BLACK, M. J. 2014. FAUST: Da taset and evaluation for 3D mesh registration. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 3794 3801. CHANG, W., AND ZWICKER, M. 2009. Range scan registration using reduced deformable models. Computer Graphics Forum 28, 2, 447456.
CHEN, Y., LIU, Z., AND ZHANG, Z. 2013. Tensorbased human body modeling. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 105 112. 2000. CMU graphics lab motion capture database. http://mocap.cs.cmu.edu. Accessed: 20121211.
CORAZZA, S., AND GAMBARETTO, E., 2014. Automatic generation of 3D character animation from 3D meshes, Aug. 5. US Patent 8,797,328.
DE AGUIAR, E., THEOBALT, C, THRUN, S., AND SEIDEL, H.P. 2008. Automatic conversion of mesh animations into skeletonbased animations. Computer Graphics Forum 27, 2, 389397. 2015. Dyna dataset. http://dyna.is.tue.mpg.de/. Accessed: 20150515. FREIFELD, O., AND BLACK, M. J. 2012. Lie bodies: A manifold representation of 3D human shape. In European Conf. on Computer Vision (ECCV), SpringerVerlag, A. Fitzgibbon et al.
(Eds.), Ed., Part I, LNCS 7572, 114.
HASLER, N., STOLL, C, SUNKEL, M., ROSENHAHN, B., AND SEIDEL, H.
2009. A statistical model of human pose and body shape. Computer Graphics Forum 28, 2, 337346. HASLER, N., THORM^{"}AHLEN, T., ROSENHAHN, B., AND SEIDEL, H.P.
2010. Learning skeletons for shape and pose. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D '10, 2330. HIRSHBERG, D., LOPER, M., RACHLIN, E., AND BLACK, M. 2012. Coregis tration: Simultaneous alignment and modeling of articulated 3D shape. In European Conf. on Computer Vision (ECCV), SpringerVerlag, A. F. et al. (Eds.), Ed., LNCS 7577, Part IV, 242255. JAMES, D. L., AND TWIGG, C. D. 2005. Skinning mesh animations. ACM Trans. Graph. 24, 3 (July), 399407.
KAVAN, L., AND ^{V}Z 'ARA, J. 2005. Spherical blend skinning: Arealtime deformation of articulated models. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D '05, 916.
KAVAN, L., COLLINS, S., ^{V}Z 'ARA, J., AND O'SULLIVAN, C. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics (TOG) 27, 4, 105:1105:23. KAVAN, L., COLLINS, S., AND O'SULLIVAN, C. 2009. Automatic linearization of nonlinear skinning. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D '09, 4956. KRY, P. G., JAMES, D. L., AND PAI, D. K. 2002. EigenSkin: Real time large deformation character skinning in hardware. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, SCA '02, 153159. KURIHARA, T., AND MIYATA, N. 2004. Modeling deformable human hands from medical images. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, AirelaVille, Switzerland, Switzerland, SCA '04, 355363. LAWSON, C. L., AND HANSON, R. J. 1995. Solving least squares problems. Classics in applied mathematics. SIAM, Philadelphia, PA. SIAM : Society of industrial and applied mathematics.
LE, B. H., AND DENG, Z. 2012. Smooth skinning decomposition with rigid bones. ACM Trans. Graph. 31, 6 (Nov.), 199:1199:10.
LE, B. H., AND DENG, Z. 2014. Robust and accurate skeletal rigging from mesh sequences. ACM Trans. Graph. 33, 4 (July), 84:184:10. LEWIS, J. P., CORDNER, M., AND FONG, N. 2000. Pose space deformation: A unified approach to shape interpolation and skeletondriven deformation. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/ Addison Wesley Publishing Co., New York, NY, USA, SIGGRAPH '00, 165172.
LOPER, M. M., AND BLACK, M. J. 2014. OpenDR: An approximate differenti able renderer. In Computer Vision  ECCV 2014, Springer, Heidelberg, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds., vol. 8695 of Lecture Notes in Computer Science, 154169.
LOPER, M.M., MAHMOOD, N., AND BLACK, M. J. 2014. MoSh: Motion and shape capture from sparse markers. ACM Trans. Graph., (Proc. SIGGRAPH Asia) 33, 6 (Nov.), 220:1220:13.
MERRY, B., MARAIS, P., AND GAIN, J. 2006. Animation space: A truly linear framework for character animation. ACM Trans. Graph. 25, 4 (Oct.), 1400 1423.
MILLER, C, ARIKAN, O., AND FUSSELL, D. 2010. Frankenrigs: Building character rigs from multiple sources. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D '10, 3138.
MOHR, A., AND GLEICHER, M. 2003. Building efficient, accurate character skins from examples. ACM Trans. Graph. (Proc. SIGGRAPH), 562568. NOCEDAL, J., AND WRIGHT, S. J. 2006. Numerical Optimization, 2nd ed. Springer, New York.
PONSMOLL, G., ROMERO, J., MAHMOOD, N., AND BLACK, M. J. 2015. Dyna: A model of dynamic human shape in motion. ACM Transactions on Graphics, (Proc. SIGGRAPH) 34, 4
(July), 120:1120:14.
RHEE, T., LEWIS, J., AND NEUMANN, U. 2006. Realtime weighted pose space deformation on the GPU. EUROGRAPHICS 25, 3.
ROBINETTE, K., BLACKWELL, S., DAANEN, H., BOEHMER, M., FLEMING, S., BRILL, T., HOEFERLIN, D., AND BURNSIDES, D. 2002. Civilian American and European Surface Anthropometry Resource (CAESAR) final report. Tech. Rep. AFRLHEWPTR20020169, US Air Force Research Laboratory.
ROUET, C, AND LEWIS, J., 1999. Method and apparatus for creating lifelike digital representations of computer animated objects by providing corrective enveloping, Mar. 16. US Patent 5,883,638.
SCHAEFER, S., AND YUKSEL, C. 2007. Examplebased skeleton extraction. In Proceedings of the Fifth Eurographics Symposium on Geometry Processing, Eurographics Association, AirelaVille, Switzerland, Switzerland, SGP '07, 153 162.
SEO, H., CORDIER, F., AND MAGNENATTHALMANN, N. 2003. Synthesizing animatable body models with parameterized shape modifications. In Pro ceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, AirelaVille, Switzerland, Switzerland, SCA '03, 120125.
TSOLI, A., MAHMOOD, N., AND BLACK, M. J. 2014. Breathing life into shape: Capturing, modeling and animating 3D human 52:152:11.
WANG, X. C, AND PHILLIPS, C. 2002. Multiweight enveloping: Least squares approximation techniques for skin animation. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, SCA '02, 129138.
WANG, R. Y., PULLI, K., AND POPOVI' C , J. 2007. Realtime enveloping with rotational regression. ACM Trans. Graph. (Proc. SIGGRAPH) 26, 3 (July). WEBER, O., SORKINE, O., LIPMAN, Y., AND GOTSMAN, C. 2007. Context aware skeletal shape deformation. Computer Graphics Forum 26, 3 (Sept.), 265274.
Claims
Priority Applications (2)
Application Number  Priority Date  Filing Date  Title 

US201562183853P true  20150624  20150624  
US62/183,853  20150624 
Applications Claiming Priority (4)
Application Number  Priority Date  Filing Date  Title 

EP16731900.3A EP3314577A1 (en)  20150624  20160623  Skinned multiperson linear model 
US15/739,658 US10395411B2 (en)  20150624  20160623  Skinned multiperson linear model 
US16/540,967 US20190392626A1 (en)  20150624  20190814  Skinned multiperson linear model 
US16/550,266 US20200058137A1 (en)  20150624  20190825  Skinned MultiPerson Linear Model 
Related Child Applications (3)
Application Number  Title  Priority Date  Filing Date 

US201715739658A A371OfInternational  20171222  20171222  
US16/540,967 Continuation US20190392626A1 (en)  20150624  20190814  Skinned multiperson linear model 
US16/550,266 ContinuationInPart US20200058137A1 (en)  20150624  20190825  Skinned MultiPerson Linear Model 
Publications (1)
Publication Number  Publication Date 

WO2016207311A1 true WO2016207311A1 (en)  20161229 
Family
ID=56203388
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

PCT/EP2016/064610 WO2016207311A1 (en)  20150624  20160623  Skinned multiperson linear model 
Country Status (3)
Country  Link 

US (2)  US10395411B2 (en) 
EP (1)  EP3314577A1 (en) 
WO (1)  WO2016207311A1 (en) 
Cited By (3)
Publication number  Priority date  Publication date  Assignee  Title 

WO2019095051A1 (en) *  20171114  20190523  Ziva Dynamics Inc.  Method and system for generating an animationready anatomy 
US10410431B2 (en)  20170711  20190910  Nvidia Corporation  Skinning a cluster based simulation with a visual mesh using interpolated orientation and position 
WO2019207176A1 (en) *  20180425  20191031  Seddi, Inc.  Modelling of nonlinear softtissue dynamics for interactive avatars 
Families Citing this family (1)
Publication number  Priority date  Publication date  Assignee  Title 

WO2017040905A1 (en) *  20150903  20170309  Stc.Unm  Accelerated precomputation of reduced deformable models 
Citations (2)
Publication number  Priority date  Publication date  Assignee  Title 

US5883638A (en)  19951201  19990316  Lucas Digital, Ltd.  Method and apparatus for creating lifelike digital representations of computer animated objects by providing corrective enveloping 
US8797328B2 (en)  20100723  20140805  Mixamo, Inc.  Automatic generation of 3D character animation from 3D meshes 
Family Cites Families (1)
Publication number  Priority date  Publication date  Assignee  Title 

GB2515343B (en) *  20130621  20180207  Toshiba Res Europe Limited  Methods and systems for generating a three dimensional representation of a subject 

2016
 20160623 EP EP16731900.3A patent/EP3314577A1/en active Pending
 20160623 US US15/739,658 patent/US10395411B2/en active Active
 20160623 WO PCT/EP2016/064610 patent/WO2016207311A1/en active Application Filing

2019
 20190814 US US16/540,967 patent/US20190392626A1/en active Pending
Patent Citations (2)
Publication number  Priority date  Publication date  Assignee  Title 

US5883638A (en)  19951201  19990316  Lucas Digital, Ltd.  Method and apparatus for creating lifelike digital representations of computer animated objects by providing corrective enveloping 
US8797328B2 (en)  20100723  20140805  Mixamo, Inc.  Automatic generation of 3D character animation from 3D meshes 
NonPatent Citations (42)
Title 

"Lecture Notes in Computer Science", vol. 86, pages: 154  169 
ALLEN, B.; CURLESS, B.; POPOVI'C , Z.: "Articulated body deformation from range scan data", ACM TRANS. GRAPH. (PROC. SIGGRAPH, vol. 21, no. 3, July 2007 (20070701), pages 612  619 
ALLEN, B.; CURLESS, B.; POPOVI'C, Z.: "The space of human body shapes: Reconstruction and parameterization from range scans", ACM TRANS. GRAPH. (PROC. SIGGRAPH, vol. 22, no. 3, 2003, pages 587  594 
ALLEN, B.; CURLESS, B.; POPOVI'C, Z.; HERTZMANN, A: "Learning a correlated model of identity and posedependent body shape variation for realtime synthesis", PROCEEDINGS OF THE 6 ACM SIG GRAPH/EUROGRAPHICS SYMPOSIUM ON COMPUTER ANIMATION, EUROGRAPHICS ASSOCIATION, 2006, pages 147  156, XP055298160, DOI: doi:10.2312/SCA/SCA06/147156 
ANGUELOV ET AL: "SCAPE: shape completion and animation of people", ACM TRANSACTIONS ON GRAPHICS (TOG), ACM, US, vol. 24, no. 3, 2005, pages 408  416, XP007913261, ISSN: 07300301 * 
ANGUELOV, D.; SRINIVASAN, P.; KOLLER, D.; THRUN, S.; RODGERS, J.; DAVIS, J.: "SCAPE: Shape Completion and Animation of PEople. ACM Trans. Graph.", PROC. SIGGRAPH, vol. 24, no. 3, 2005, pages 408  416 
BARAN, I.; POPOVI'C , J: "Automatic rigging and animation of D characters", ACM TRANS. GRAPH. (PROC. SIGGRAPH, vol. 26, no. 3, July 2007 (20070701), XP002700724, DOI: doi:10.1145/1276377.1276467 
BOGO, F.; ROMERO, J.; LOPER, M.; BLACK, M. J.: "FAUST: Dataset and evaluation for D mesh registration", PROC. IEEE CONF. ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2014, pages 3794  3801, XP032649382, DOI: doi:10.1109/CVPR.2014.491 
BRETT ALLEN ET AL: "Learning a Correlated Model of Identity and PoseDependent Body Shape Variation for RealTime Synthesis", EUROGRAPHICS/ ACM SIGGRAPH SYMPOSIUM ON COMPUTER ANIMATION, 2006, pages 147  156, XP055298160, Retrieved from the Internet <URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.5067&rep=rep1&type=pdf> [retrieved on 20160829], DOI: 10.2312/SCA/SCA06/147156 * 
CHANG, W.; ZWICKER, M.: "Range scan registration using reduced deformable models", COMPUTER GRAPHICS FORUM, vol. 28, no. 2, 2009, pages 447  456 
CHEN, Y.; LIU, Z.; ZHANG, Z: "Tensorbased human body modeling", IEEE CONF. ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2000, pages 105  112 
DE AGUIAR, E.; THEOBALT, C.; THRUN, S.; SEIDEL, H.P.: "Automatic conversion of mesh animations into skeletonbased animations", COMPUTER GRAPHICS FORUM, vol. 27, no. 2, 2008, pages 389  397 
FREIFELD, O.; BLACK, M. J. ET AL.: "European Conf. on Computer Vision (ECCV", 2012, SPRINGERVERLAG, article "Lie bodies: A manifold representation of D human shape", pages: 1  14 
HASLER, N.; STOLL, C.; SUNKEL, M.; ROSENHAHN, B.; SEIDEL, H.: "A statistical model of human pose and body shape", COMPUTER GRAPHICS FORUM, vol. 28, no. 2, 2009, pages 337  346, XP055069320, DOI: doi:10.1111/j.14678659.2009.01373.x 
HASLER, N.; THORM''AHLEN, T.; ROSENHAHN, B.; SEIDEL, H.P.: "Proceedings of the ACM SIGGRAPH Symposium on Interactive D Graphics and Games", 2010, ACM, article "Learning skeletons for shape and pose", pages: 23  30 
HIRSHBERG, D.; LOPER, M.; RACHLIN, E.; BLACK, M. ET AL.: "European Conf. on Computer Vision (ECCV", 2012, SPRINGERVERLAG, article "Coregistration: Simultaneous alignment and modeling of articulated D shape", pages: 242  255 
JAMES, D. L.; TWIGG, C. D.: "Skinning mesh animations", ACM TRANS. GRAPH., vol. 24, no. 3, July 2005 (20050701), pages 399  407, XP002468241 
KAVAN, L.; COLLINS, S.; O'SULLIVAN, C.: "Proceedings of the Symposium on Interactive D Graphics and Games", 2009, ACM, article "Automatic linearization of nonlinear skinning", pages: 49  56 
KAVAN, L.; COLLINS, S.; Z 'ARA, J.; O'SULLIVAN, C.: "Geometric skinning with approximate dual quaternion blending", ACM TRANSACTIONS ON GRAPHICS (TOG, vol. 27, no. 4, 2008, XP058098859, DOI: doi:10.1145/1409625.1409627 
KAVAN, L.; Z 'ARA, J.: "Proceedings of the Symposium on Interactive D Graphics and Games", 2005, ACM, article "Spherical blend skinning: Arealtime deformation of articulated models" 
KRY, P. G.; JAMES, D. L.; PAI, D. K: "Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation", 2002, ACM, article "EigenSkin: Real time large deformation character skinning in hardware", pages: 153  159 
KURIHARA, T.; MIYATA, N.: "Modeling deformable human hands from medical images", PROCEEDINGS OF THE ACM SIGGRAPH/EUROGRAPHICS SYMPOSIUM ON COMPUTER ANIMATION, EUROGRAPHICS ASSOCIATION, 2004, pages 355  363 
LAWSON, C. L.; HANSON, R. J.: "Solving least squares problems. Classics in applied mathematics", SIAM, 1995 
LE, B. H.; DENG, Z.: "Robust and accurate skeletal rigging from mesh sequences", ACM TRANS. GRAPH., vol. 33, no. 4, July 2014 (20140701), pages 84.1  84.10, XP058051975, DOI: doi:10.1145/2601097.2601161 
LE, B. H.; DENG, Z.: "Smooth skinning decomposition with rigid bones", ACM TRANS. GRAPH., vol. 31, no. 6, November 2012 (20121101), pages 199.1  199.10 
LEWIS J P ET AL: "POSE SPACE DEFORMATION: A UNIFIED APPROACH TO SHAPE INTERPOLATION AND SKELETONDRIVEN DEFORMATION", COMPUTER GRAPHICS. SIGGRAPH 2000 CONFERENCE PROCEEDINGS. NEW ORLEANS, LA, JULY 23  28, 2000; [COMPUTER GRAPHICS PROCEEDINGS. SIGGRAPH], NEW YORK, NY : ACM, US, 23 July 2000 (20000723), pages 165  172, XP001003552, ISBN: 9781581132083, DOI: 10.1145/344779.344862 * 
LEWIS, J. P.; CORDNER, M.; FONG, N.: "Proceedings of the th Annual Conference on Computer Graphics and Interactive Techniques", 2000, ACM PRESS/ADDISONWESLEY PUBLISHING CO., article "Pose space deformation: A unified approach to shape interpolation and skeletondriven deformation", pages: 165  172 
LOPER, M. M.; BLACK, M. J.: "Computer Vision  ECCV", 2014, SPRINGER, article "OpenDR: An approximate differentiable renderer" 
LOPER, M.M.; MAHMOOD, N.; BLACK, M. J.: "MoSh: Motion and shape capture from sparse markers", ACM TRANS. GRAPH., (PROC. SIGGRAPH ASIA, vol. 33, no. 6, November 2014 (20141101), pages 220.1  220.13 
MERRY, B.; MARAIS, P.; GAIN, J.: "Animation space: A truly linear framework for character animation", ACM TRANS. GRAPH., vol. 25, no. 4, October 2006 (20061001), pages 1400  1423 
MILLER, C.; ARIKAN, O.; FUSSELL, D.: "Proceedings of the ACM SIGGRAPH Symposium on Interactive D Graphics and Games", 2010, ACM, article "Frankenrigs: Building character rigs from multiple sources", pages: 31  38 
MOHR, A.; GLEICHER, M.: "Building efficient, accurate character skins from examples", ACM TRANS. GRAPH. (PROC. SIGGRAPH, 2003, pages 562  568 
NOCEDAL, J.; WRIGHT, S. J.: "Numerical Optimization", 2006, SPRINGER 
PONSMOLL, G.; ROMERO, J.; MAHMOOD, N.; BLACK, M. J.: "Dyna: A model of dynamic human shape in motion", ACM TRANSACTIONS ON GRAPHICS, (PROC. SIGGRAPH, vol. 34, no. 4, July 2015 (20150701), pages 120.1  120.14 
RHEE, T.; LEWIS, J.; NEUMANN, U.: "Realtime weighted posespace deformation on the GPU", EUROGRAPHICS, vol. 25, 2006, pages 3 
ROBINETTE, K.; BLACKWELL, S.; DAANEN, H.; BOEHMER, M.; FLEMING, S.; BRILL, T.; HOEFERLIN, D.; BURNSIDES, D.: "Tech. Rep. AFRLHEWPTR  6", 2002, US AIR FORCE RESEARCH LABORATORY, article "Civilian American and European Surface Anthropometry Resource (CAESAR) final report" 
SCHAEFER, S.; YUKSEL, C.: "Examplebased skeleton extraction", PROCEEDINGS OF THE FIFTH EUROGRAPHICS SYMPOSIUM ON GEOMETRY PROCESSING, 2007, pages 153  162 
SEO, H.; CORDIER, F.; MAGNENATTHALMANN, N.: "Synthesizing animatable body models with parameterized shape modifications", PROCEEDINGS OF THE ACM SIGGRAPH/EUROGRAPHICS SYMPOSIUM ON COMPUTER ANIMATION, EUROGRAPHICS ASSOCIATION, 2003, pages 120  125, XP001233114 
TSOLI, A.; MAHMOOD, N.; BLACK, M. J., BREATHING LIFE INTO SHAPE: CAPTURING, MODELING AND ANIMATING D HUMAN, 2014, pages 52.1  52.11 
WANG, R. Y.; PULLI, K.; POPOVI'C , J.: "Realtime enveloping with rotational regression", ACM TRANS. GRAPH. (PROC. SIGGRAPH, vol. 26, no. 3, 2007 
WANG, X. C.; PHILLIPS, C: "Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation", 2002, ACM, article "Multiweight enveloping: Leastsquares approximation techniques for skin animation", pages: 129  138 
WEBER, O.; SORKINE, O.; LIPMAN, Y.; GOTSMAN, C.: "Contextaware skeletal shape deformation", COMPUTER GRAPHICS FORUM, vol. 26, no. 3, September 2007 (20070901), pages 265  274, XP055234401, DOI: doi:10.1111/j.14678659.2007.01048.x 
Cited By (3)
Publication number  Priority date  Publication date  Assignee  Title 

US10410431B2 (en)  20170711  20190910  Nvidia Corporation  Skinning a cluster based simulation with a visual mesh using interpolated orientation and position 
WO2019095051A1 (en) *  20171114  20190523  Ziva Dynamics Inc.  Method and system for generating an animationready anatomy 
WO2019207176A1 (en) *  20180425  20191031  Seddi, Inc.  Modelling of nonlinear softtissue dynamics for interactive avatars 
Also Published As
Publication number  Publication date 

US20180315230A1 (en)  20181101 
EP3314577A1 (en)  20180502 
US10395411B2 (en)  20190827 
US20190392626A1 (en)  20191226 
Similar Documents
Publication  Publication Date  Title 

Cao et al.  Facewarehouse: A 3d facial expression database for visual computing  
Jacobson et al.  Fast automatic skinning transformations  
Jacobson et al.  Bounded biharmonic weights for realtime deformation.  
Ichim et al.  Dynamic 3D avatar creation from handheld video input  
Neumann et al.  Sparse localized deformation components  
US10403404B2 (en)  Physical face cloning  
Le et al.  Robust and accurate skeletal rigging from mesh sequences  
Pighin et al.  Resynthesizing facial animation through 3d modelbased tracking  
Alexa et al.  Asrigidaspossible shape interpolation  
Guan et al.  Drape: Dressing any person  
US9314692B2 (en)  Method of creating avatar from user submitted image  
Igarashi et al.  Asrigidaspossible shape manipulation  
Schaefer et al.  Examplebased skeleton extraction  
Hasler et al.  A statistical model of human pose and body shape  
MagnenatThalmann et al.  Handbook of virtual humans  
Erleben et al.  Physicsbased animation  
Kavan et al.  Geometric skinning with approximate dual quaternion blending  
Xu et al.  Photoinspired modeldriven 3D object modeling  
US7057619B2 (en)  Methods and system for general skinning via hardware accelerators  
DeCarlo et al.  An anthropometric face model using variational techniques  
De Aguiar et al.  Stable spaces for realtime clothing  
Min et al.  Synthesis and editing of personalized stylistic human motion  
Vlasic et al.  Articulated mesh animation from multiview silhouettes  
Wang et al.  Deformation capture and modeling of soft objects.  
Pishchulin et al.  Building statistical shape spaces for 3d human modeling 
Legal Events
Date  Code  Title  Description 

121  Ep: the epo has been informed by wipo that ep was designated in this application 
Ref document number: 16731900 Country of ref document: EP Kind code of ref document: A1 

WWE  Wipo information: entry into national phase 
Ref document number: 15739658 Country of ref document: US 

NENP  Nonentry into the national phase in: 
Ref country code: DE 

WWE  Wipo information: entry into national phase 
Ref document number: 2016731900 Country of ref document: EP 