US20100295854A1  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery  Google Patents
Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery Download PDFInfo
 Publication number
 US20100295854A1 US20100295854A1 US12/635,999 US63599909A US2010295854A1 US 20100295854 A1 US20100295854 A1 US 20100295854A1 US 63599909 A US63599909 A US 63599909A US 2010295854 A1 US2010295854 A1 US 2010295854A1
 Authority
 US
 United States
 Prior art keywords
 3d representation
 source
 reference
 reference 3d
 3d
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Abandoned
Links
Images
Classifications

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06K—RECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
 G06K9/00—Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
 G06K9/62—Methods or arrangements for recognition using electronic means
 G06K9/6217—Design or setup of recognition systems and techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
 G06K9/6255—Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06K—RECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
 G06K9/00—Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
 G06K9/00201—Recognising threedimensional objects, e.g. using range or tactile information
 G06K9/00208—Recognising threedimensional objects, e.g. using range or tactile information by matching twodimensional images to threedimensional objects

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06K—RECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
 G06K9/00—Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
 G06K9/00221—Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
 G06K9/00288—Classification, e.g. identification
Abstract
A method and system for characterizing features in a source multifeatured threedimensional object and for locating a bestmatching threedimensional object from a reference database of such objects by performing a viewpoint invariant search among the reference objects. The invention further includes the creation of a threedimensional representation of the source object by deforming a reference object.
Description
 This application claims priority to and the benefits of U.S. Provisional Applications Ser. Nos. 60/452,429, 60/452,430 and 60/452,431 filed on Mar. 6, 2003 (the entire disclosures of which are hereby incorporated by reference).
 The present invention relates to object modeling and matching systems, and more particularly to the generation of a threedimensional model of a target object from two and threedimensional input.
 In many situations, it is useful to construct a threedimensional (3D) model of an object when only a partial description of the object is available. In a typical situation, one or more twodimensional (2D) images of the 3D object may be available, perhaps photographs taken from different viewpoints. A common method of creating a 3D model of a multifeatured object is to start with a base 3D model which describes a generic or typical example of the type of object being modeled, and then to add texture to the model using one or more 2D images of the object. For example, if the multifeatured object is a human face, a 3D “avatar” (i.e., an electronic 3D graphical or pictorial representation) would be generated by using a preexisting, standard 3D model of a human face, and mapping onto the model a texture from one or more 2D images of the face. See U.S. Pat. No. 6,532,011 B1 to Francini et al., and U.S. Pat. No. 6,434,278 B1 to Hashimoto. The main problem with this approach is that the 3D geometry is not highly defined or tuned for the actual target object which is being generated.
 A common variant of the above approach is to use a set of 3D base models and select the one that most resembles the target object before performing the texture mapping step. Alternatively, a single parameterized base model is used, and the parameters of the model are adjusted to best approximate to the target. See U.S. Pat. No. 6,556,196 B1 to Blanz et al. These methods serve to refine the geometry to make it fit the target, at least to some extent. However, for any target object with a reasonable range of intrinsic variability, the geometry of the model will still not be well tuned to the target. This lack of geometric fit will detract from the verisimilitude of the 3D model to the target object.
 Conventional techniques typically also require that the 2D images being used for texturing the model be acquired from known viewpoints relative to the 3D object being modeled. This usually limits the use of such approaches to situations where the model is being generated in a controlled environment in which the target object can be photographed. Alternatively, resort may be had to human intervention to align 2D images to the 3D model to be generated. See U.S. Patent Publication No. 2002/0012454 to Liu et al. This manual step places a severe limit on the speed with which a 3D model can be generated from 2D imagery.
 Accordingly, a need exists for an automated approach that systematically makes use of available 2D source data for a 3D object to synthesize an optimal 3D model of the object.
 The present invention provides an automated method and system for generating an optimal 3D model of a target multifeatured object when only partial source data describing the object is available. The partial source data often consists of one or more 2D projections of the target object or an obscuration of a single projection, but may also include 3D data, such as from a 3D camera or scanner. The invention uses a set of reference 3D representations that span, to the extent practicable. the variations of the class of objects to which the target object belongs. The invention may automatically identify feature items common to the source data and to the reference representations, and establish correspondences between them. For example, if the target object is a face the system may identify points at the extremities of the eyes and mouth, or the nose profile, and establish correspondences between such features in the source data and in the reference representations. Manual identification and matching of feature items can also be incorporated if desired. Next, all possible positions (i.e., orientations and translations) for each 3D reference representation are searched to identify the position and reference representation combination whose projection most closely matches the source data. The closeness of match is determined by a measure such as the minimum meansquared error (MMSE) between the feature items in the projection of the 3D representation, and the corresponding feature items in the source projection. A comparison is performed in 3D between the estimated deprojected positions of the feature items from the 2D source projection and the corresponding feature items of the 3D representation. The closestfitting 3D reference representation may then be deformed to optimize the correspondence with the source projection. Each point in the mesh which defines the geometry of the 3D representation is free to move during the deformation. The search for the bestfitting position (i.e., orientation and translation) is repeated using the deformed 3D representation, and the deformation and search may be repeated iteratively until convergence occurs or terminated at any time.
 Thus the geometry of the 3D model is tailored to the target object in two ways. First, when more than one reference representation is available, the selection of the bestfitting reference representation from a set of references enables the optimal coarsegrain choice to be made. Second, deformation enables fine scale tuning in which errors introduced by inaccurate choice of viewpoint are progressively reduced by iteration. The invention requires no information about the viewpoint from which the 2D source projection was captured, because a search is performed over all possible viewpoints, and the viewpoint is taken to be that which corresponds to the closest fit between the projected 3D representation and the 2D source data.
 In a first aspect, the invention comprises a method of comparing at least one source 2D projection of a source multifeatured object to a reference library of 3D reference objects. In accordance with the method, a plurality of reference 3D representations of generically similar multifeatured objects is provided, and a viewpointinvariant search of the reference 3D representations is performed to locate the reference 3D representation having a 2D projection most resembling the source projection(s). In some embodiments, resemblance is determined by a degree of alignment between feature items in the 3D representation and corresponding feature items in the source 2D projection(s). Each reference 3D representation may be searched over a range of possible 2D projections of the 3D representation without actually generating any projections. The search over a range of possible 2D projections may comprise computing a rigid motion of the reference 3D representation optimally consistent with a viewpoint of the source multifeatured object in at least one of the 2D projections. The rigid motions may comprise pitch, roll, yaw, and translation in three dimensions. Automatic camera calibration may be performed by estimation of camera parameters, such as aspect ratio and field of view, from image landmarks.
 In some embodiments, the optimum rigid motion may be determined by estimating a conditional mean pose or geometric registration as it relates to feature items comprising points, curves, surfaces, and subvolumes in a 3D coordinate space associated with the reference 3D representation such that the feature items are projectionally consistent with feature items in source 2D projection(s). MMSE estimates between the conditional mean estimate of the projected feature items and corresponding feature items of the reference 3D representation are generated. The rigid motion may be constrained by known 3D position information associated with the source 2D projection(s).
 In some embodiments, the feature items may include curves as well as points which are extracted from the source projection using dynamic programming. Further, areas as well as surfaces and or subvolumes may be used as features generated via isocontouring (such as via the Marching Cubes algorithm) or automated segmentation algorithms. The feature items used in the matching process may be found automatically by using correspondences between the 2D source projection(s) and projected imagery of at least one reference 3D object.
 The invention may further comprise the step of creating a 3D representation of the source 2D projection(s) by deforming the located (i.e., bestfitting) reference 3D representation so as to resemble the source multifeatured object. In one embodiment, the deformation is a large deformation diffeomorphism, which serves to preserve the geometry and topology of the reference 3D representation. The deformation step may deform the located 3D representation so that feature items in the source 2D projection(s) align with corresponding features in the located reference 3D representation. The deformation step may occur with or without rigid motions and may include affine motions. Further, the deformation step may be constrained by at least one of known 3D position information associated with the source 2D projection(s), and 3D data of the source object. The deformation may be performed using a closed form expression.
 In a second aspect, the invention comprises a system for comparing at least one source 2D projection of a source multifeatured object to a reference library of 3D reference objects. The system comprises a database comprising a plurality of reference 3D representations of generically similar multifeatured objects and an analyzer for performing a viewpointinvariant search of the reference 3D representations to locate the reference 3D representation having a 2D projection most resembling the source projection(s). In some embodiments, the analyzer determines resemblance by a degree of alignment between feature items in the 3D representation and corresponding feature items in the source 2D projection(s). The analyzer may search each reference 3D representation over a range of possible 2D projections of the 3D representation without actually generating any projections. In some embodiments, the analyzer searches over a range of possible 2D projections by computing a rigid motion of the reference 3D representation optimally consistent with a viewpoint of the source multifeatured object in at least one of the 2D projections. The rigid motions may comprise pitch, roll, yaw, and translation in three dimensions. The analyzer may be configured to perform automatic camera calibration by estimating camera parameters. such as aspect ratio and field of view, from image landmarks.
 In some embodiments, the analyzer is configured to determine the optimum rigid motion by estimating a conditional mean of feature items comprising points, curves, surfaces, and subvolumes in a 3D coordinate space associated with the reference 3D representation such that the feature items are projectionally consistent with feature items in the source 2D projection(s). The analyzer is further configured to generate MMSE estimates between the conditional mean estimate of the projected feature items and corresponding feature items of the reference 3D representation. The rigid motion may be constrained by known 3D position information associated with the source 2D projection(s).
 In some embodiments, the analyzer is configured to extract feature items from the source projection using dynamic programming. In further embodiments, the analyzer may be configured to find feature items used in the matching process automatically by using correspondences between source imagery and projected imagery of at least one reference 3D object.
 The invention may further comprise a deformation module for creating a 3D representation of the at least one source 2D projection by deforming the located (i.e., bestfitting) reference 3D representation so as to resemble the source multifeatured object. In one embodiment, the deformation module deforms the located reference 3D representation using large deformation diffeomorphism, which serves to preserve the geometry and topology of the reference 3D representation. The deformation module may deform the located 3D representation so that feature items in the source 2D projection(s) align with corresponding features in the located reference 3D representation. The deformation module may or may not use rigid motions and may use affine motions. Further, the deformation module may be constrained by at least one of known 3D position information associated with the source 2D projection(s) and 3D data of the source object. The deformation module may operate in accordance with a closed form expression.
 In a third aspect, the invention comprises a method of comparing a source 3D object to at least one reference 3D object. The method involves creating 2D representations of the source object and the reference object(s) and using projective geometry to characterize a correspondence between the source 3D object and a reference 3D object. For example, the correspondence may be characterized by a particular viewpoint for the 2D representation of the 3D source object.
 In a fourth aspect, the invention comprises a system for comparing a source 3D object to at least one reference 3D object. The system comprises a projection module for creating 2D representations of the source object and the reference object(s) and an analyzer which uses projective geometry to characterize a correspondence between the source 3D object and a reference 3D object.
 In a fifth aspect, the above described methods and systems are used for the case when the 3D object is a face and the reference 3D representations are avatars.
 In a sixth aspect, the invention comprises a method for creating a 3D representation from at least one source 2D projection of a source multifeatured object. In accordance with the method, at least one reference 3D representation of a generically similar object is provided, one of the provided representation(s) is located, and a 3D representation of the source 2D projection(s) is created by deforming the located reference representation in accordance with the source 2D projection(s) so as to resemble the source multifeatured object. In some embodiments, the source 2D projection(s) is used to locate the reference representation. In further embodiments, the set of reference representations includes more than one member, and the reference most resembling the source 2D projection(s) is located by performing a viewpointinvariant search of the set of reference representations, without necessarily actually generating any projections. The search may include computing a rigid motion of the reference representation optimally consistent with a viewpoint of the source multifeatured object in at least one of the source projections.
 In a preferred embodiment, a 3D representation of the source projection(s) is created by deforming the located reference representation so as to resemble the source multifeatured object. The deformation may be a large deformation diffeomorphism. In some embodiments, the deformation deforms the located reference so that feature items in the source projection(s) align with corresponding feature items in the located 3D reference representation. In some embodiments, the deformation is performed in real time.
 In a seventh aspect. the invention comprises a system for creating a 3D representation from at least one source 2D projection of a source multifeatured object. The system includes a database of at least one reference 3D representation of a generically similar object, and an analyzer for locating one of the provided representation(s). The system further includes a deformation module for creating a 3D representation of the source 2D projection(s) by deforming the located reference representation in accordance with the source 2D projection(s) so as to resemble the source multifeatured object. In some embodiments, the analyzer uses the source 2D projection(s) to locate the reference representation. In further embodiments, the set of reference representations includes more than one member, and the analyzer locates the reference most resembling the source 2D projection(s) by performing a viewpointinvariant search of the set of reference representations, without necessarily actually generating any projections. The search may include computing a rigid motion of the reference representation optimally consistent with a viewpoint of the source multi featured object in at least one of the source projections.
 In a preferred embodiment, the deformation module creates a 3D representation of the source projection(s) by deforming the located reference representation so as to resemble the source multifeatured object. The deformation may be a large deformation diffeomorphism. In some embodiments, the deformation module deforms the located reference so that feature items in the source projection(s) align with corresponding feature items in the located 3D reference representation. In some embodiments, the deformation module operates in real time.
 In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which:

FIG. 1 schematically illustrates the various components of the invention, starting with the target object, the reference objects, and yielding an optimal 3D model after performing a search and deformation. 
FIGS. 2A , 2B, and 2C schematically illustrate the components of a 3D avatar. 
FIG. 3 schematically illustrates the matching of feature items in the 2D imagery. 
FIG. 4 is a block diagram showing a representative hardware environment for the present invention. 
FIG. 5 is a block diagram showing components of the analyzer illustrated inFIG. 4 . 
FIG. 6 is a block diagram showing the key functions performed by the analyzer. 
FIG. 1 illustrates the basic operation of the invention in the case where the 3D target multifeatured object is a face and the set of reference 3D representations are avatars. The matching process starts with a set of reference 3D avatars which represent, to the extent practicable, the range of different types of heads to be matched. For example, the avatars may include faces of men and women, faces with varying quantities and types of hair, faces of different ages, and faces representing different races. Typically the reference set includes numerous (e.g., several hundred or more) avatars, though the invention works with as few as one reference object, and as many for which storage space and compute time are available. In some situations, only a single reference avatar will be used. This case may arise, for example, when the bestfitting avatar has been selected manually, or by some other means, or when only one reference avatar is available. InFIG. 1 , the source data of the target face is illustrated as a single 2D photograph of the target 3D face taken from an unknown viewpoint. First, selected features to be used for matching are identified in the source photograph. The features may be points, such as the extremities of the mouth and eyes, or curves representing a profile, an eyebrow, or other distinctive curve, or subareas such as an eyebrow, nose or cheek. The corresponding features are identified in the reference avatars. The selection of features in the target photograph, and the identification of corresponding features in the reference avatars may be done automatically according to the invention. Next, a viewpointinvariant search is conducted in which each 3D avatar is notionally subjected to all possible rigid motions and the features projected into 2D. The positions of the projected avatar features are compared to the feature positions in the target photograph. The avatar for which a particular rigid motion provides the closest fit between projected features and those of the source photograph is selected as the best reference avatar.FIG. 1 illustrates the bestfitting reference avatar to be the middle one.  Next, the best reference avatar is deformed to match the target photograph more closely. First the features of the photograph are reverse projected to the coordinates of the best reference avatar in the orientation and position corresponding to the best match. The mesh points of the avatar are then deformed in 3D to minimize the distances between the reverseprojected features of the photograph and the corresponding avatar features. The avatar resulting from this deformation will be a closer approximation to the target 3D face. The rigid motion search and deformation steps may be repeated iteratively, e.g., until the quality of fit no longer improves appreciably. The resulting 3D model is the optimal match to the target face.
 The invention can be used effectively even when the source imagery includes only a part of the target face, or when the target face is partially obscured, such as, for example, by sunglasses or facial hair. The approach of the invention is suitable for any multifeatured object, such as faces, animals, plants, or buildings. For ease of explanation, however, the ensuing description will focus on faces as an exemplary (and nonlimiting) application.

FIGS. 2A , 2B, and 2C show the components of a representative avatar. In one embodiment of the invention, the geometry of the avatar is represented by a mesh of points in 3D which are the vertices of set of triangular polygons approximating the surface of the avatar.FIG. 2A illustrates a headon view 202 and a side view 204 of the triangular polygon representation. In one representation, each vertex is given a color value, and each triangular face may be colored according to an average of the color values assigned to its vertices. The color values are determined from a 2D texture map 206, illustrated inFIG. 2B , which may be derived from a photograph. InFIG. 2C , the final avatar with texture is illustrated in a headon view 208 and side view 210. The avatar is associated with a coordinate system which is fixed to it, and is indexed by three angular degrees of freedom (pitch, roll, and yaw), and three translational degrees of freedom of the rigid body center in threespace. In addition, individual features of the avatar, such as the chin, teeth and eyes may have their own local coordinates (e.g., chin axis) which form part of the avatar description. The present invention may be equally applied to avatars for which a different data representation is used. For example, texture values may be represented as RGB values, or using other color representations, such as HSL. The data representing the avatar vertices and the relationships among the vertices may vary. For example, the mesh points may be connected to form nontriangular polygons representing the avatar surface.  The invention may include a conventional rendering engine for generating 2D imagery from a 3D avatar. The rendering engine may be implemented in OpenGL, or in any other 3D rendering system, and allows for the rapid projection of a 3D avatar into a 2D image plane representing a camera view of the 3D avatar. The rendering engine may also include the specification of the avatar lighting, allowing for the generation of 2D projections corresponding to varying illumination of the avatar. Lighting corresponding to a varying number of light sources of varying colors, intensities, and positions may be generated.
 The feature items in the 2D source projection which are used for matching are selected by hand or via automated methods. These items may be points or curves. When the source projection includes a front view, suitable points may be inflection points at the lips, points on the eyebrows, points at the extremities of the eyes, or extremities of nostrils, and suitable curves may include an eyebrow or lip. When the source projection includes a side view, the feature points corresponding to the profile are used and may include the tip of the nose or chin. Suitable feature item curves may include distinct parts of the profile, such as nose, forehead, or chin.
 When the feature items are determined manually, a user interface is provided which allows the user to identify feature points individually or to mark groups of points delineated by a spline curve, or to select a set of points forming a line.
 The automated detection of feature items on the 2D source projection is performed by searching for specific features of a face, such as eyeballs, nostrils, and lips. As understood by those of ordinary skill in the art, the approach may use Bayesian classifiers and decision trees in which hierarchical detection probes are built from training data generated from actual avatars. The detection probes are desirably stored at multiple pixel scales so that the specific parameters, such as for orientation of a feature, are only computing on finer scales if the largerscale probes yield a positive detection. The feature detection probes may be generated from image databases representing large numbers of individuals who have had their features demarcated and segregated so that the detection probes become specifically tuned to these features. The automated feature detection approach may use pattern classification, Bayes nets, neural networks, or other known techniques for determining the location of features in facial images.
 The automated detection of curve features in the source projection may use dynamic programming approaches to generate curves from a series of points so as to reduce the amount of computation required to identify an optimal curve and maximize a sequentially additive cost function. Such a cost function represents a sequence of features such as the contrast of the profile against background, or the darkness of an eyebrow, or the crease between lips. A path of N points can be thought of as consisting of a starting node x_{0 }and a set of vectors v_{0}, v_{1}, . . . , v_{N−1 }connecting neighboring nodes. The nodes comprising this path are defined as

${x}_{i}=\sum _{j=0}^{i=1}\ue89e{v}_{j}+{x}_{0}.$  Rather than searching over all paths of length N, dynamic programming may be used to generate maximum (or minimum) cost paths. This reduces the complexity of the algorithm from K^{N }to NK^{2 }where N is the length of a path and K is the total number of nodes, as dynamic programming takes advantage of the fact that the cost is sequentially additive, allowing a host of suboptimal paths to be ignored. Dynamic programming techniques and systems are wellcharacterized in the art and can be applied as discussed herein without undue experimentation.
 Next, the 3D rotation and translation from the avatar coordinates to the source projection is determined. This corresponds to finding the viewpoint from which the source projection was captured. In preferred embodiments, this is achieved by calculating the position of the avatar in 3D space that best matches the set of selected feature items in the 2D source projection. Generally, these feature items will be points, curves, or subareas and the source projection will be a photograph on which the position of these items can be measured, either manually or automatically. The position calculation may be based on the computation of the conditional mean estimate of the reverse projection positions in 3D of the 2D feature items, followed by the computation of MMSE estimates for the rotation and translation parameters in 3D, given the estimates of the 3D positions of the feature items. Since position in 3D space is a vector parameter, the MMSE estimate for translation position is closed form; when substituted back into the squared error function, it gives an explicit function in terms of only the rotations. Since the rotations are not vector parameters, they may be calculated using nonlinear gradient descent through the tangent space of the group or via local representation using the angular velocities of the skewsymmetric matrices.
 In addition to or in lieu of the least squares or weighted least squares techniques described herein, the distance metrics used to measure the quality of fit between the reverse projections of feature items from the source imagery and corresponding items in the 3D avatar may be, for example, Poisson or other distance metrics which may or may not satisfy the triangle inequality.
 If feature items measured in 3D are available, such as from actual 3D source data from 3D cameras or scanners. the feature item matching may be performed directly, without the intermediate step of calculating the conditional mean estimate of the deprojected 2D features. The cost function used for positioning the 3D avatar can be minimized using algorithms such as closed form quadratic optimization, iterative Newton descent or gradient methods.
 The 3D positioning technique is first considered without deformation of the reference avatar. In the following, a 3D reference avatar is referred to as a CAD (computeraided design) model, or by the symbol CAD. The set of x_{j}=(x_{j}, y_{j}, z_{j}), j=1, . . . , N features is defined on the CAD model. The projective geometry mapping is defined as either positive or negative z, i.e., projection occurs along the z axis. In all the projective geometry

${p}_{j}=\left(\frac{{\alpha}_{1}\ue89e{x}_{j}}{{z}_{j}},\frac{{\alpha}_{2}\ue89e{y}_{j}}{{z}_{j}}\right)$  (for negative zaxis projection), or

${p}_{j}=\left(\frac{{\alpha}_{1}\ue89e{x}_{j}}{{z}_{j}},\frac{{\alpha}_{2}\ue89e{y}_{j}}{{z}_{j}}\right)$ 
${p}_{j}=\left(\frac{{\alpha}_{1}\ue89e{x}_{j}}{{z}_{j}},\frac{{\alpha}_{2}\ue89e{y}_{j}}{{z}_{j}}\right),i=1,\dots \ue89e\phantom{\rule{0.8em}{0.8ex}},N,$  where n is the cotangent of the projective angle. The following data structures are defined throughout:

$\begin{array}{cc}{P}_{i}=\left(\begin{array}{c}\frac{{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}}{{\alpha}_{1}}\\ \frac{{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}}{{\alpha}_{2}}\\ 1\end{array}\right),{Q}_{i}=\left(I\frac{{{P}_{i}\ue8a0\left({P}_{i}\right)}^{1}}{{\uf605{P}_{i}\uf606}^{2}}\right),\stackrel{\_}{Q}=\sum _{i=1}^{N}\ue89e{Q}_{i},{X}_{Q}=\sum _{i=1}^{N}\ue89e{Q}_{i}\ue89e{X}_{i}& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e1\right)\\ {X}_{j}=\left(\begin{array}{ccccccccc}{x}_{j}^{1}& {x}_{j}^{2}& {x}_{j}^{3}& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& {x}_{j}^{1}& {x}_{j}^{2}& {x}_{j}^{3}& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& {x}_{j}^{1}& {x}_{j}^{2}& {x}_{j}^{3}\end{array}\right),& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e2\right)\end{array}$  with (•)′ matrix transpose, and the identity matrix

$I=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right).$  For negative (i.e., z<0) mapping

$,{p}_{j}=\left(\frac{{\alpha}_{1}\ue89e{x}_{j}}{{z}_{j}},\frac{{\alpha}_{2}\ue89e{y}_{j}}{{z}_{j}}\right),$  and the change

${P}_{i}={\left(\frac{{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}}{{a}_{1}},\frac{{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}}{{a}_{2}},1\right)}^{\prime}$  is made.
 The basis vectors, Z_{1,}Z_{2,}Z_{3 }at the tangent to the 3×3 rotation element O are defined as:

Z _{1}=1_{1}O^{old} =[o _{21} , o _{22} , o _{23} , −o _{11} , −o _{12} , −o _{13}, 0, 0, 0]′ (Equation 3) 
Z _{2}=1_{2} O ^{old} =[o _{31} , o _{32} , o _{33}, 0, 0, 0, −o _{11} , −o _{12} , −o _{13}]′ (Equation 4) 
Z _{3}=1_{3} O ^{old}=[0, 0, 0, o _{31} , o _{32} , o _{33} , −o _{21} , −o _{22} , −o _{23}]′ (Equation 5) 
$\begin{array}{cc}\mathrm{where}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{1}_{1}=\left(\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& 0\end{array}\right),\text{}\ue89e{1}_{2}=\left(\begin{array}{ccc}0& 0& 1\\ 0& 0& 0\\ 1& 0& 0\end{array}\right),{1}_{3}=\left(\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ 0& 1& 0\end{array}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e6\right)\end{array}$  The reverse projection of feature points from the 2D projection may now be performed. Given the feature points p_{j}=(p_{j1}, p_{j2}), j=1, 2, . . . in the image plane, the minimum normestimates for {circumflex over (z)}_{i }are given by, Ô, {circumflex over (b)} as

${\hat{z}}_{i}=\frac{\u3008{\mathrm{Ox}}_{j}+b,{P}_{i}\u3009}{{\uf605{P}_{i}\uf606}^{2}},$  and the MMSE, Ô, {circumflex over (b)} satisfies

$\begin{array}{cc}\underset{z,O,b}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e{\uf605{\mathrm{Ox}}_{i}+b{z}_{i}\ue89e{P}_{i}\uf606}^{2}=\underset{O,b}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e{\left({\mathrm{Ox}}_{i}+b\right)}^{t}\ue89e{Q}_{i}\ue8a0\left({\mathrm{Ox}}_{i}+b\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e7\right)\end{array}$  During the process of matching a source image to reference avatars, there may be uncertainty in the determined points x, implying that cost matching is performed with a covariance variability structure built in to the formula. In this case, the norm has within it a 3×3 matrix which represents this variability in the norm.


${p}_{j}=\left(\frac{{\alpha}_{1}\ue89e{x}_{j}}{{z}_{j}},\frac{{\alpha}_{2}\ue89e{y}_{j}}{{z}_{j}}\right),$ 
$\begin{array}{cc}\underset{z,O,b}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e{\uf605{\mathrm{Ox}}_{i}+b{z}_{i}\ue89e{P}_{i}\uf606}^{2}=\underset{O,b}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e{\left({\mathrm{Ox}}_{i}+b\right)}^{t}\ue89e{Q}_{i}\ue8a0\left({\mathrm{Ox}}_{i}+b\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e8\right)\end{array}$  The optimum translation rotation solutions are preferably generated as follows. Compute the 3×9 matrix M_{t}=X_{i} −
Q ^{−1 }X_{Q }and evaluate exhaustively the cost function choosing minimum Ô and computing the translation 
$\hat{b}={\left(\sum _{i=1}^{N}\ue89e{Q}_{i}\right)}^{1}\ue89e\sum _{i=1}^{N}\ue89e{Q}_{i}\ue89e\hat{O}\ue89e{x}_{i}$  with minimum Ô attained, for example, via brute force search over the orthogonal group which may for example be parameterized by pitch, roll or yaw or running the gradient search algorithm to convergence as follows.

$\begin{array}{cc}\mathrm{Brute}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{Force}\ue89e\text{:}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\hat{O}=\mathrm{arg}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\underset{o}{\mathrm{max}}\ue89e{O}^{t}\ue8a0\left(\sum _{i=1}^{N}\ue89e{M}_{i}^{t}\ue89e{Q}_{i}\ue89e{M}_{i}\right)\ue89eO;& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e9\right)\\ \mathrm{Gradient}\ue89e\text{:}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{O}^{\mathrm{new}}={\uf74d}^{\sum _{i=1}^{3}\ue89e{a}_{i}^{\mathrm{new}}\ue89e{1}_{i}}\ue89e{O}^{\mathrm{old}},\text{}\ue89e{\alpha}_{j}^{\mathrm{new}}=\u30082\ue89e\left(\sum _{i=1}^{N}\ue89e{M}_{i}^{t}\ue89e{Q}_{i}\ue89e{M}_{i}\right)\ue89e{O}^{\mathrm{old}},{Z}_{j}\u3009,j=1,2,3\ue89e\text{}\ue89e\mathrm{with}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\u3008f,g\u3009=\sum _{i=1}^{3}\ue89e{f}_{i}\ue89e{g}_{i}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e10\right)\end{array}$  In a typical application some information about the position of the object in 3D space is known. For example, in a system which takes a succession of photographs of a moving source object, such as in a tracking system, the position from a previous image may be available. The invention may incorporate this information into the matching process as follows. Given a sequence of points p_{i}, i=1, . . . , N and a rigid transformation of the form A=O, b: xOx+b (centered around x_{e}=0), the MMSE of rotation and translation Ô, {circumflex over (b)} satisfies:

$\begin{array}{cc}\underset{z,O,b}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e{\uf605{\mathrm{Ox}}_{i}+b{z}_{i}\ue89e{P}_{i}\uf606}^{2}+{\left(b\mu \right)}^{t}\ue89e{\sum}^{1}\ue89e\left(b\mu \right)=\underset{O,b}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e{\left({\mathrm{Ox}}_{i}+b\right)}^{t}\ue89e{Q}_{i}\ue8a0\left({\mathrm{Ox}}_{i}+b\right)+{\left(b\mu \right)}^{t}\ue89e{\sum}^{1}\ue89e\left(b\mu \right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e11\right)\end{array}$  The 3×9 matrix M_{i }and a 3×1 column vector N are computed:

M _{i} X _{i} −Q _{Σ} ^{−1} X _{Q} , N=Q _{Σ} ^{−1} X _{Q}Q _{Σ},=(Q +Σ ^{−1}), ψ=Q _{Σ} ^{−1}Σ_{u} , φ=Q _{Σ} ^{−1}Σ_{μ}−μ, Σ_{μ}=Σ^{−1}μ. (Equation 12)  The translation is then determined {circumflex over (b)}=−
Q _{Σ} ^{−1}X_{Q}Ô+Q _{Σ} ^{−1}Σ_{μ} at the minimum Ô obtained by exhaustive search or gradient algorithm run until convergence: 
$\begin{array}{cc}\mathrm{Brute}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{Force}\ue89e\text{:}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\hat{O}=\mathrm{arg}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\underset{o}{\mathrm{max}}\ue89e{O}^{t}\ue8a0\left(\sum _{i=1}^{N}\ue89e{M}_{i}^{t}\ue89e{Q}_{i}\ue89e{M}_{i}+{N}^{t}\ue89e{\sum}^{1}\ue89eN\right)\ue89eO+2\ue89e{O}^{t}\ue8a0\left(\sum _{i=1}^{N}\ue89e{M}_{i}^{t}\ue89e{Q}_{i}\ue89e\psi N\ue89e{\sum}^{1}\ue89e\phi \right)& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e13\right)\\ \mathrm{Gradient}\ue89e\text{:}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{O}^{\mathrm{new}}={\uf74d}^{\sum _{i=1}^{3}\ue89e{a}_{i}^{\mathrm{new}}\ue89e{1}_{i}}\ue89e{O}^{\mathrm{old}},\text{}\ue89e{\alpha}_{j}^{\mathrm{new}}=\u30082\ue89e\left(\sum _{i=1}^{N}\ue89e{M}_{i}^{t}\ue89e{Q}_{i}\ue89e{M}_{i}={N}^{t}\ue89e{\sum}^{1}\ue89eN\right)\ue89e{O}^{\mathrm{old}}+2\ue89e\sum _{i=1}^{N}\ue89e{M}_{i}^{t}\ue89e{Q}_{i}\ue89e\psi {N}^{1}\ue89e{\sum}^{1}\ue89e\phi ,{Z}_{j}\u3009& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.6em}{0.6ex}}\ue89e14\right)\end{array}$  with the projection onto the basis vectors Z_{1},Z_{2},Z_{3 }of equations 35 defined at the tangent to O^{old }in the exponential representation where α^{new }are the directional derivatives of cost.
 The rotation/translation data may be indexed in many different ways. For example, to index according to the rotation around the center of the object, rather than fixed external world coordinates, the coordinates are just reparameterized by defining a {tilde over (x)}←x−x_{c}. All of the techniques described herein remain the same.
 The preferred 3D algorithm for rigid motion efficiently changes states of geometric pose for comparison to the measured imagery. The preferred 3D algorithm for diffeomorphic transformation of geometry matches the geometry to target 2D image features. It should be understood, however, that other methods of performing the comparison of the 3D representation to the source imagery may be used, including those that do not make use of specific image features.
 Once the rigid motion (i.e., rotation and translation) that results in the best fit between 2D source imagery and a selected 3D avatar is determined, the 3D avatar may be deformed in order to improve its correspondence with the source imagery. The allowed deformations are generally limited to diffeomorphisms of the original avatar. This serves to preserve the avatar topology, guaranteeing that the result of the deformation will be a face. The deformations may also enforce topological constraints, such as the symmetry of the geometry. This constraint is especially useful in situations where parts of the source object are obscured, and the full geometry is inferred from partial source information.

FIGS. 3A and 3B illustrate the effect of avatar deformation on the matching of the avatar to the source imagery. InFIG. 3A , feature points are shown as black crosses on the source image 302. An example is the feature point at the left extremity of the left eyebrow 304. The projections of the corresponding feature points belonging to the bestmatching reference avatar with optimal rigid motion prior to deformation are shown as white crosses. It can be seen that the projected point corresponding to the left extremity of the left eyebrow 306 is noticeably displaced from its counterpart 304. InFIG. 3B , the same source image 302 is shown with feature points again indicated by black crosses. This time, the bestfitting avatar feature points shown as white crosses are now projected after deformation. The correspondence between source feature points and avatar feature points is markedly improved, as shown, for example, by the improved proximity of the projected left eyebrow feature point 308 to its source counterpart 304.  The 3D avatar diffeomorphism calculation starts with the initial conditions for placement of the avatar determined by the feature item detection and computation of the bestfitting rigid motion and the original geometry of the avatar. It then proceeds by allowing all of the points on the avatar to move independently according to a predefined formula to minimize the distance between the deformed avatar points in 3D and the conditional mean estimates of the 2D landmark points reverse projected to the 3D coordinates. Once this diffeornorphism is calculated, the 3D landmark rigid motion algorithm is applied again to the source projections and feature items to find the best guess of the camera positions given this newly transformed avatar with its new vertex positions. Subsequently, a new diffeomorphism is generated, and this process is continued until it converges. Alternatively, iteration may not be used, with the rigid motion calculation being performed only a single time, and just one diffeomorphism transformation applied. In the case where camera orientations (i.e., the viewpoint of the measured source projections) are known precisely, these can be used as fixed inputs to the calculation, with no rigid transformation required. When the measured sets of feature items are in 3D, such as from a cyber scan or 3D camera observations of the candidate head, the avatar transformed onto the candidate sets of points without any intermediate generation of the candidate points in 3D space via the conditional mean algorithm for generating 3D points from 2D sets of points.
 The diffeomorphic deformation of an avatar proceeds as follows. Given the set x_{j}=(x_{j}, y_{j}, z_{j}), j=1, . . . , N feature items defined on the CAD model, with the projective geometry mapping with

${\alpha}_{1}=\frac{2\ue89en}{w},{\alpha}_{2}=\frac{2\ue89en}{h},$  where n is cotangent of the angle, and w,h are aspect ratio widths and heights,

$\left(x,y,z\right)\mapsto p\ue8a0\left(x,y,z\right)=\left(\frac{{\alpha}_{1}\ue89ex}{z},\frac{{\alpha}_{2}\ue89ey}{z}\right),$  with observations of the feature items through the projective geometry

${p}_{j}=\left(\frac{{\alpha}_{1}\ue89e{x}_{j}}{{z}_{j}},\frac{{\alpha}_{2}\ue89e{y}_{j}}{{z}_{j}}\right).$  The goal is to construct the deformation of the CAD model x→x+u(x), x ∈ CAD with unknown camera rigid motions corresponding to the measured projective image feature items. The projective points for each orientation v=1, . . . , V, and smoothing matrices

$\begin{array}{cc}{P}_{i}^{\left(v\right)}=\left(\begin{array}{c}\frac{{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}^{\left(v\right)}}{{a}_{1}}\\ \frac{{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}^{\left(v\right)}}{{a}_{2}}\\ 1\end{array}\right),{Q}_{i}^{\left(v\right)}=I\ue89e\frac{{P}_{i}^{\left(v\right)}\ue89e{P}_{i}^{\left(v\right)\ue89et}}{{\uf605{P}_{i}^{\left(v\right)}\uf606}^{2}},{\stackrel{\_}{Q}}_{i}=\sum _{v=1}^{V}\ue89e{O}^{\left(v\right)\ue89et}\ue89e{Q}_{i}^{\left(v\right)}\ue89e{O}^{\left(v\right)},\text{}\ue89e{\stackrel{\_}{Q}}_{{O}_{i}}=\sum _{v=1}^{V}\ue89e{O}^{\left(v\right)\ue89et}\ue89e{Q}_{i}^{\left(v\right)},& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e15\right)\\ \stackrel{\_}{Q}=\underset{\underset{3\ue89eN\times 3\ue89eN}{\uf613}}{\left(\begin{array}{cccc}{\stackrel{\_}{Q}}_{1}& 0& \dots & 0\\ 0& {\stackrel{\_}{Q}}_{2}& 0& \dots \\ \vdots & \vdots & \vdots & \vdots \\ 0& 0& \dots & {\stackrel{\_}{Q}}_{N}\end{array}\right)},{\stackrel{\_}{Q}}_{o}=\underset{\underset{3\ue89eN\times 3\ue89eN}{\uf613}}{\left(\begin{array}{cccc}{\stackrel{\_}{Q}}_{{o}_{1}}& 0& \dots & 0\\ 0& {\stackrel{\_}{Q}}_{{o}_{2}}& 0& \dots \\ \vdots & \vdots & \vdots & \vdots \\ 0& 0& \dots & {\stackrel{\_}{Q}}_{N}\end{array}\right)},& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e16\right)\\ K=\underset{\underset{3\ue89eN\times 3\ue89eN}{\uf613}}{\left(\begin{array}{cccc}{K}_{11}& {K}_{12\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\dots}& {K}_{1\ue89eN}& \phantom{\rule{0.3em}{0.3ex}}\\ {K}_{21}& {K}_{21}& \dots & {K}_{2\ue89eN}\\ \vdots & \vdots & \vdots & \vdots \\ {K}_{N\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}& {K}_{N\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}& \dots & {K}_{\mathrm{NN}}\end{array}\right)},\text{}\ue89e{K}_{\mathrm{ij}}=\left(\begin{array}{ccc}{K}_{\mathrm{ij}}\ue8a0\left(1,1\right)& {K}_{\mathrm{ij}}\ue8a0\left(1,2\right)& {K}_{\mathrm{ij}}\ue8a0\left(1,3\right)\\ {K}_{\mathrm{ij}}\ue8a0\left(2,1\right)& {K}_{\mathrm{ij}}\ue8a0\left(2,2\right)& {K}_{\mathrm{ij}}\ue8a0\left(2,3\right)\\ {K}_{\mathrm{ij}}\ue8a0\left(3,1\right)& {K}_{\mathrm{ij}}\ue8a0\left(3,2\right)& {K}_{\mathrm{ij}}\ue8a0\left(3,3\right)\end{array}\right),& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e17\right)\end{array}$  are constructed, where for example K_{ij}=diag (e^{−a∥x} ^{ i } ^{−x} ^{ j } ^{∥}, e^{−a∥x} ^{ j } ^{−x} ^{ j } ^{∥}, e^{−a∥x} ^{ j } ^{−x} ^{ j } ^{∥}) corresponds to the square root inverse Laplacian operator L=diag (−∇^{2}+c).
 In one embodiment, the avatar may be deformed with small deformations only and no rigid motions. For this embodiment, it is assumed that the measured feature items are all points from a single camera viewing which generated the projected source image in which the feature points were measured. The goal is to construct the deformation of the CAD model constructing the mapping x→x+u(x), x ∈ CAD:

$\begin{array}{cc}\underset{u,{z}_{n}}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\sum _{i=1}^{N}\ue89e{\uf605\left({x}_{n}+u\ue8a0\left({x}_{n}\right)\right){z}_{n}\ue89e{P}_{n}\uf606}^{2}=\underset{u}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\sum _{n=1}^{N}\ue89e{\left({x}_{n}+u\ue8a0\left({x}_{n}\right)\right)}^{t}\ue89e{Q}_{n}\ue8a0\left({x}_{n}+u\ue8a0\left({x}_{n}\right)\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e18\right)\end{array}$  First, the transformation of the model x→x+u(x) with u(x)=Σ_{n−1} ^{N}K(x_{n}, x)β_{n }and where

$\begin{array}{cc}\left(\begin{array}{c}{\beta}_{1}\\ \vdots \\ {\beta}_{N}\end{array}\right)={K}^{1}\ue8a0\left(\frac{1}{{\sigma}^{2}}\ue89e{\left({K}^{1}+\frac{1}{{\sigma}^{2}}\ue89e\stackrel{\_}{Q}\right)}^{1}\ue89e\left(\stackrel{\_}{Q}\ue8a0\left(\begin{array}{c}{x}_{1}\\ \vdots \\ {x}_{N}\end{array}\right)\right)\right)& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e19\right)\end{array}$  is computed. Next, rigid motions are added and the following equation solved for the optimizer:

$\begin{array}{cc}\underset{u,{z}_{n}}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\hspace{1em}\sum _{i=1}^{N}\ue89e\sum _{v=1}^{V}\ue89e{\left(\begin{array}{c}{O}^{\left(v\right)}\\ \left(\begin{array}{c}{x}_{n}+\\ u\ue8a0\left({x}_{n}\right)\end{array}\right)+{b}^{\left(v\right)}\end{array}\right)}^{t}\ue89e\phantom{\rule{0.em}{0.ex}}\ue89e{Q}_{n}^{\left(v\right)}(\phantom{\rule{0.em}{0.ex}}\ue89e{O}^{\left(v\right)}\ue8a0\left({x}_{n}+u(\phantom{\rule{0.em}{0.ex}}\ue89e{x}_{n})\right)+\phantom{\rule{0.em}{0.ex}}\ue89e{b}^{\left(v\right)}).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e20\right)\end{array}$  The transformation of the model using small deformation x→x+u(x) is computed, where u(x)=Σ_{n=1} ^{N}K(x_{n}, x)β_{n }and

$\begin{array}{cc}\left(\begin{array}{c}{\beta}_{1}\ue8a0\left(44\right)\\ \vdots \\ {\beta}_{N}\end{array}\right)={K}^{1}\ue8a0\left(\frac{1}{{\sigma}^{2}}\ue89e{\left({K}^{1}+\frac{1}{{\sigma}^{2}}\ue89e\stackrel{\_}{Q}\right)}^{1}\ue89e\left(\begin{array}{c}\stackrel{\_}{Q}\ue8a0\left(\begin{array}{c}{x}_{1}\\ \vdots \\ {x}_{N}\end{array}\right)+\\ \left(\begin{array}{c}\sum _{v=1}^{V}\ue89e{O}^{\left(v\right)\ue89et}\ue89e{Q}_{1}^{\left(v\right)}\ue8a0\left({b}^{\left(v\right)}\right)\\ \vdots \\ \sum _{v=1}^{V}\ue89e{O}^{\left(v\right)\ue89et}\ue89e{Q}_{N}^{\left(v\right)}({b}^{\left(v\right)}\end{array}\right)\end{array}\right)\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e21\right)\end{array}$  In another embodiment, diffeomorphic deformations with no rigid motions of the avatar are applied. In the case that the change in shape of the face is extensive the large deformation φ: xφ(x) satisfying φ=φ_{1}φ_{t}=∫_{o}V_{s}(φ_{s}(x))ds+x,x ∈ CAD is generated. The deformation of the CAD model constructing the mapping xφ(x), x ∈ CAD is constructed:

$\begin{array}{cc}\underset{{v}_{t},t\in \left[0,1\right],{z}_{n}}{\mathrm{min}}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\uf605\phi \ue8a0\left({x}_{n}\right){z}_{n}\ue89e{P}_{n}\uf606}^{2}={\mathrm{min}}_{{v}_{t},t\in \left[0,1\right]}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e\uf605{\phi \ue8a0\left({x}_{n}\right)}^{t}\ue89e{Q}_{n}\ue89e\phi \ue8a0\left({x}_{n}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e22\right)\end{array}$  Using the initialization v^{new}=0,φ^{new}(x)=x, x ∈ CAD, mappings are repeatedly generated from the new vector field by running an iterative algorithm until convergence:

$\begin{array}{cc}{v}_{1}^{\mathrm{new}}\ue8a0(\xb7)=\sum _{n=1}^{N}\ue89eK\ue8a0\left({\phi}_{1}^{\mathrm{new}}\ue8a0\left({x}_{n}\right),\xb7\right)\ue89e{D}_{{\phi}_{t}^{\mathrm{new}}\ue8a0\left({x}_{n}\right)}\ue89e{\phi}_{t,1}^{\mathrm{new}}\ue89e{Q}_{n}\ue89e{\phi}^{\mathrm{new}}\ue8a0\left({x}_{n}\right),& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e23\right)\\ {\phi}^{\mathrm{new}}\ue8a0\left(x\right)={\int}_{0}^{1}\ue89e{v}_{t}^{\mathrm{new}}\ue8a0\left({\phi}_{t}^{\mathrm{new}}\ue8a0\left(x\right)\right)\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74cx+x,& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e24\right)\end{array}$  where

${D}_{{\phi}_{1}\ue8a0\left({x}_{n}\right)}\ue89e{\phi}_{t,1}=\left(\frac{\partial {\phi}_{1}\xb7{\phi}_{t}^{1}}{\partial {y}_{j}}\ue89e{\ue85c}_{{\phi}_{i}\ue8a0\left({x}_{n}\right)}\right).$ 
$\begin{array}{cc}\underset{{v}_{t}:{\phi}_{t}={v}_{t}\ue8a0\left({\phi}_{t}\right),t\in \left[0,1\right],{z}_{n}}{\mathrm{min}}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{v=1}^{V}\ue89e{\uf605{O}^{\left(v\right)}\ue89e\phi \ue8a0\left({x}_{n}\right)+{b}^{\left(v\right)}{z}_{n}^{\left(v\right)}\ue89e{P}_{n}^{\left(v\right)}\uf606}^{2}=\underset{{v}_{1}\ue89et\in \left[0,1\right]}{\mathrm{min}}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e\sum _{v=1}^{V}\ue89e{\left({O}^{\left(v\right)}\ue89e\phi \ue8a0\left({x}_{n}\right)+{b}^{\left(v\right)}\right)}^{t}\ue89e{Q}_{n}^{\left(v\right)}\ue8a0\left({O}^{\left(v\right)}\ue89e\phi \ue8a0\left({x}_{n}\right)+{b}^{\left(v\right)}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e25\right)\end{array}$  Using the initialization v^{new}=0,φ^{new}(x)=x, x ∈ CAD, a mapping is generated from the new vector field by running an iterative algorithm until convergence:

$\begin{array}{cc}{v}_{1}^{\mathrm{new}}\ue8a0(\xb7)=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89eK\ue8a0\left({\phi}_{t}^{\mathrm{new}}\ue8a0\left({x}_{n}\right),\xb7\right)\ue89e{D}_{{\phi}_{t}^{\mathrm{new}}\ue8a0\left({x}_{n}\right)}\ue89e{\phi}_{t,1}^{\mathrm{tnew}}\ue8a0\left(\sum _{v=1}^{V}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{O}^{\left(v\right)\ue89et}\ue89e{Q}_{n}^{\left(v\right)}\ue8a0\left({O}^{\left(v\right)}\ue89e{\phi}^{\mathrm{new}}\ue8a0\left({x}_{n}\right)+{b}^{\left(v\right)}\right)\right);& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e26\right)\\ {\phi}^{\mathrm{new}}\ue8a0\left(x\right)={\int}_{0}^{1}\ue89e{v}_{t}^{\mathrm{new}}\ue8a0\left({\phi}_{t}^{\mathrm{new}}\ue8a0\left(x\right)\right)\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74cx+x,\text{}\ue89e\mathrm{Where}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{D}_{{\phi}_{t}\ue8a0\left({x}_{n}\right)}\ue89e{\phi}_{t,1}=\left(\frac{\partial {\phi}_{1}\xb7{\phi}_{t}^{1}}{\partial {y}_{j}}\ue89e{\ue85c}_{{\phi}_{t}\ue8a0\left({x}_{n}\right)}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e27\right)\end{array}$  In a further embodiment, the deformation may be performed in real time for the case when the rigid motions (i.e., the rotation/translation) which bring the avatar into correspondence with the one or more source 2D projection are not known. A similar approach to the one above is used, with the addition of an estimation of the rigid motions using the techniques described herein. The initialization u^{new}=0 is used. Rigid motions are calculated using the rotation/translation techniques above to register the CAD model xx+u^{new}(x) to each photograph, generating rigid motions O^{(v)new},b^{(v)new},v=1, 2, . . . O^{(v)new},b^{(v)new }are fixed from the previous step, and the deformation of the CAD model xx+u^{new}(x) or large deformation xφ(x) are computed using the above techniques to solve the realtime small deformation or large deformation problem:

$\begin{array}{cc}\left(\mathrm{small}\right)\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\underset{u}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{v=1}^{V}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left({O}^{\left(v\right)}\ue8a0\left({x}_{n}+u\ue8a0\left({x}_{n}\right)\right)+{b}^{\left(v\right)}\right)}^{t}\ue89e{Q}_{n}^{\left(v\right)}\ue8a0\left({O}^{\left(v\right)}\ue8a0\left({x}_{n}+u\ue8a0\left({x}_{n}\right)\right)+{b}^{\left(v\right)}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e28\right)\\ \left(\mathrm{large}\right)\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\underset{{v}_{t},t\in \left[0,1\right]}{\mathrm{min}}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{v=1}^{V}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e({{O}^{\left(v\right)}\ue8a0\left(\phi \ue8a0\left({x}_{n}\right)+{b}^{\left(v\right)}\right)}^{t}\ue89e{Q}_{n}^{\left(v\right)}\ue8a0\left({O}^{\left(v\right)}\ue8a0\left(\phi \ue8a0\left({x}_{n}\right)\right)+{b}^{\left(v\right)}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e29\right)\end{array}$  In another embodiment, the avatar is deformed in realtime using diffeomorphic deformations. The solution to the realtime deformation algorithm generates a deformation which may be used as an initial condition for the solution of the diffeomorphic deformation. Realtime diffeomorphic deformation is accomplished by incorporating the realtime deformations solution as an initial condition and then performing a small number (in the region of 1 to 10) iterations of the diffeomorphic deformation calculation.
 The deformation may include affine motions. For the affine motion A: x→Ax where A is the 3×3 generalized linear matrix so that

$\begin{array}{cc}\underset{A,{z}_{n}}{\mathrm{min}}\ue89e\sum _{n=1}^{N}\ue89e\sum _{v=1}^{V}\ue89e{\uf605{O}^{\left(v\right)}\ue89e{\mathrm{Ax}}_{n}+{b}^{\left(v\right)}{z}_{n}^{\left(v\right)}\ue89e{P}_{n}^{\left(v\right)}\uf606}^{2}=\underset{A}{\mathrm{min}}\ue89e\sum _{n=1}^{N}\ue89e\sum _{v=1}^{V}\ue89e{\left({O}^{\left(v\right)}\ue89e{\mathrm{Ax}}_{n}+{b}^{\left(v\right)}\right)}^{1}\ue89e{Q}_{n}^{\left(v\right)}\ue8a0\left({O}^{\left(v\right)}\ue89e{\mathrm{Ax}}_{n}+{b}^{\left(v\right)}\right),& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e30\right)\end{array}$  the least squares estimator Â: x→Âx is computed:

$\begin{array}{cc}\hat{A}={\left(\sum _{n=1}^{N}\ue89e{X}_{n}^{t}\ue89e{\stackrel{\_}{Q}}_{n}\ue89e{X}_{n}\right)}^{1}\ue89e\left(\sum _{n=1}^{N}\ue89e{X}_{n}^{t}\ue89e\sum _{v=1}^{V}\ue89e{O}^{\left(v\right)\ue89et}\ue89e{Q}_{n}^{\left(v\right)}\ue89e{b}^{\left(v\right)}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e31\right)\end{array}$  In many cases, both feature items in the projective imagery as well as the imagery itself can be used to drive the deformation of the avatar. Augmentation of source data to incorporate source imagery may improve the quality of the fit between the deformed avatar and the target face. To implement this, one more term is added to the deformation techniques. Let I be the measured imagery, which in general includes multiple measured images I^{(v)},v=1, 2, . . . corresponding to an indexed sequence of pixels indexed by p ∈ [0,1]^{2}, with the projection mapping points

$x=\left(x,y,z\right)\in {\mathrm{IR}}^{3}\mapsto p\ue8a0\left(x\right)=\left({p}_{1}\ue8a0\left(x\right)=\frac{{a}_{1}\ue89ex}{z},{p}_{2}\ue8a0\left(x\right)\ue89e\frac{{a}_{2}\ue89ey}{z}\right).$  For the discrete setting of pixels in the source image plane with color (R,G,B) template, the observed projective Π(p) is an (R,G,B) vector and the projective matrix becomes

${P}_{x}=\left(\begin{array}{ccc}\frac{{a}_{1}}{z+n}& 0& 0\\ 0& \frac{{a}_{2}}{z+n}& 0\end{array}\right),$  operating on points (x,y,z) ∈ IR^{3 }according to the projective matrix

${P}_{x}\ue89e\text{:}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\left(x,y,z\right)\mapsto \left({p}_{1}\ue8a0\left(x,y,z\right),{p}_{2}\ue8a0\left(x,y,z\right)\right)=\left(\begin{array}{ccc}\frac{{a}_{1}}{z+n}& 0& 0\\ 0& \frac{{a}_{2}}{z+n}& 0\end{array}\right)\ue89e\left(\begin{array}{c}x\\ y\\ z\end{array}\right),$  the point x(p) being the revealed point which is not occluded (closest point to the projection on the ray) on the 3D CAD model which projects to the point p in the image plane. Next, the projected template matrices resulting from finite differences on the (R,G,B) components at the projective coordinate p of the template value are required. The norm is interpreted componentwise:

$\begin{array}{cc}{\nabla}^{t}\ue89e\prod \left(p\right)=\left(\begin{array}{c}\frac{\partial}{\partial {p}_{1}}\ue89e\prod \left({p}^{r}\right)\ue89e\frac{\partial}{\partial {p}_{2}}\ue89e\prod \left({p}^{r}\right)\\ \frac{\partial}{\partial {p}_{1}}\ue89e\prod \left({p}^{g}\right)\ue89e\frac{\partial}{\partial {p}_{2}}\ue89e\prod \left({p}^{g}\right)\\ \frac{\partial}{\partial {p}_{1}}\ue89e\prod \left({p}^{b}\right)\ue89e\frac{\partial}{\partial {p}_{2}}\ue89e\prod \left({p}^{b}\right)\end{array},\right)\ue89e\stackrel{.}{I}\ue8a0\left(p\right)\ue89e\stackrel{.}{=}\ue89eI\ue8a0\left(p\right)\prod \left(p\right),& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e32\right)\\ \begin{array}{c}{\stackrel{~}{\nabla}}^{t}\ue89e\prod \left(p\right)={\nabla}^{t}\ue89e\prod \left(p\right)\ue89e{P}_{x\ue8a0\left(p\right)}\\ =\left(\begin{array}{c}\frac{\partial \prod {\left(p\right)}^{r}}{\partial {p}_{1}}\ue89e\frac{{\alpha}_{1}}{z\ue8a0\left(p\right)+n}\ue89e\frac{\partial \prod {\left(p\right)}^{r}}{\partial {p}_{2}}\ue89e\frac{{\alpha}_{2}}{z\ue8a0\left(p\right)+n}\ue89e0\\ \frac{\partial \prod {\left(p\right)}^{g}}{\partial {p}_{1}}\ue89e\frac{{\alpha}_{1}}{z\ue8a0\left(p\right)+n}\ue89e\frac{\partial \prod {\left(p\right)}^{g}}{\partial {p}_{2}}\ue89e\frac{{\alpha}_{2}}{z\ue8a0\left(p\right)+n}\ue89e0\\ \frac{\partial \prod {\left(p\right)}^{b}}{\partial {p}_{1}}\ue89e\frac{{\alpha}_{1}}{z\ue8a0\left(p\right)+n}\ue89e\frac{\partial \prod {\left(p\right)}^{g}}{\partial {p}_{2}}\ue89e\frac{{\alpha}_{2}}{z\ue8a0\left(p\right)+n}\ue89e0\end{array}\right),\end{array}& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e33\right)\end{array}$  with matrix norm ∥A−B∥^{2}=A^{r}−B^{r}^{2}+A^{g}−B^{g}^{2}+A^{b}−B^{b}^{2}.
 Associated with each image is a translation/rotation assumed already known from the previous rigid motion calculation techniques. The following assumes there is one 2D image, with O,b identity, and let any of the movements be represented as x→x+u(x). Then u(x)=Ox−x is a rotational motion, u(x)=b is a constant velocity, u(x)=Σ_{i}e_{i}E_{i}(x) is a constrained motion to a basis function such as “chin rotation,” “eyebrow lift,” etc., and the general motion u is given by:

$\begin{array}{cc}\underset{u}{\mathrm{min}}\ue89e\sum _{p\in {\left[0,1\right]}^{2}}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\uf605I\ue8a0\left(p\right)\prod \left(p\right){\nabla}^{t}\ue89e\prod \left(p\right)\ue89e{P}_{x\ue8a0\left(p\right)}\ue89eu\ue8a0\left(x\ue8a0\left(p\right)\right)\uf606}^{2}=\underset{u}{\mathrm{min}}\ue89e\sum _{p\in {\left[0,1\right)}^{2}}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\uf605\stackrel{.}{I}\ue8a0\left(p\right){\stackrel{~}{\nabla}}^{t}\ue89e\prod \left(p\right)\ue89eu\ue8a0\left(x\ue8a0\left(p\right)\right)\uf606}^{2}& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e34\right)\\ =\underset{u}{\mathrm{min}}\ue89e2\ue89e\sum _{p\in {\left[0,1\right]}^{2}}\ue89e{u\ue8a0\left(x\ue8a0\left(p\right)\right)}^{t}\ue89e\stackrel{.}{I}\ue8a0\left(p\right)\ue89e\stackrel{~}{\nabla}\ue89e\prod \left(p\right)+\sum _{p\in {\left[0,1\right]}^{2}}\ue89e{u\ue8a0\left(x\ue8a0\left(p\right)\right)}^{t}\ue89e\stackrel{~}{\nabla}\ue89e\prod \left(p\right)\ue89e{\stackrel{~}{\nabla}}^{t}\ue89e\prod \left(p\right)\ue89eu\ue8a0\left(x\ue8a0\left(p\right)\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e35\right)\end{array}$  This is linear in u so closedform expressions exist for each of the forms of u, for example, for the unconstrained general spline motion,

û(x(p))=({tilde over (∇)}Π(p){tilde over (∇)}^{t}Π(p))^{−1} İ(p){tilde over (∇)}Π(p). (Equation 36)  This approach can be incorporated into the other embodiments of the present invention for the various possible deformations described herein.
 In the situation where large numbers of feature points, curves, and subvolumes are to be automatically generated from the source projection(s) and 3D data (if any), image matching is performed directly on the source imagery or on the fundamental 3D volumes into which the source object can be divided. For the case where the avatar is generated from 2D projective photographic images, the measured target projective image has labeled points, curves, and subregions generated by diffeomorphic image matching. Defining a template projective exemplar face with all of the labeled submanifolds from the avatar, the projective exemplar can be transformed bijectively via the diffeomorphisms onto the target candidate photograph, thereby automatically labeling the target photographs into its constituent submanifolds. Given these submanifolds, the avatars can then be matched or transformed into the labeled photographs. Accordingly, in the image plane a deformation φ: xφ(x) satisfying φ=∫_{0} ^{1}v_{t}(φ_{t}(x))dt+x, x ∈ R^{2 }is generated. The template and target images I_{0}, I_{1 }are transformed to satisfy

$\begin{array}{cc}\underset{{v}_{t},t\in \left[0,1\right],{z}_{n}}{\mathrm{min}}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e{\uf605{I}_{0}\xb7{\phi}^{1}{I}_{1}\uf606}^{2}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e37\right)\end{array}$  The given diffeomorphism is applied to labeled points, curves, and areas in the template I_{0 }thereby labeling those points, curves, and areas in the target photograph.
 When the target source data are in 3D, the diffeomorphisms are used to define bijective correspondence in the 3D background space, and the matching is performed in the volume rather than in the image plane.
 The following techniques may be used to select avatar models automatically. Given a collection of avatar models {CAD^{a},a=1, 2, . . . }, and a set of measured photographs of the face of one individual, the task is to select the avatar model which is most representative of the individual face being analyzed. Let the avatar models be specified via points, curves, surfaces, or subvolumes. Assume for example N feature initial and target points, x_{n} ^{a},x_{n }∈ IR^{d},n=1, . . . , N, with x_{n}=x_{n} ^{a}+u(x_{n} ^{a}), one for each avatar a=1, 2, . . . .
 In one embodiment, the avatar is deformed with small deformations only and no rigid motions. For this embodiment, it is assumed that the measured feature items are all points from a single camera viewing which generated the projected source image in which the feature points were measured. The matching x_{n} ^{a} x_{n} ^{a}+u(x_{n} ^{a}), n=1, . . . , N is constructed and the CAD^{a }model which is of smallest metric distance is selected. The optimum selected avatar model is the one closest to the candidate in the metric. Any of a variety of distance functions may be used in the selection of the avatar, including the large deformation metric from the diffeomorphism mapping technique described above, the realtime metric described above. the Euclidean metric, and the similitude metric of Kendall. The technique is described herein using the realtime metric. When there is no rigid motion, then the CAD model is selected to minimize the metric based on one or several sets of features from photographs, here described for one photo:

$\begin{array}{cc}\begin{array}{c}{\mathrm{CAD}}^{\hat{a}}=\ue89e\mathrm{arg}\ue89e\underset{{\mathrm{CAD}}^{a},a=1,2,\dots}{\mathrm{min}}\ue89e\underset{u,{z}_{n}}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\\ \ue89e\sum _{n=1}^{N}\ue89e{\uf605\left({x}_{n}^{a}+u\ue8a0\left({x}_{n}^{a}\right)\right){z}_{n}\ue89e{P}_{n}\uf606}^{2}\\ =\ue89e\mathrm{arg}\ue89e\underset{{\mathrm{CAD}}^{a},a=1,2,\dots}{\mathrm{min}}\ue89e\underset{u}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\\ \ue89e\sum _{n=1}^{N}\ue89e{\left(\begin{array}{c}{x}_{n}^{a}+\\ u\ue8a0\left({x}_{n}^{a}\right)\end{array}\right)}^{t}\ue89e{Q}_{n}\ue8a0\left(\begin{array}{c}{x}_{n}^{a}+\\ u\ue8a0\left({x}_{n}^{a}\right)\end{array}\right).\end{array}& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e38\right)\end{array}$  In other embodiments both metrics including unknown or known rigid motions, large deformation metrics, or affine motions can be used such as described in equations 28 and 29, respectively.
 For selecting avatars given 3D information such as features of points, curves, surfaces and or subvolumes in the 3D volume, then the metric is selected which minimizes the distance between the measurements and the family of candidate CAD models. First, the K matrix defining the quadratic form metric measuring the distance is computed:

$\begin{array}{cc}K=\left(\begin{array}{cccc}K\ue8a0\left({x}_{1}^{a},{x}_{1}^{a}\right)& \mathrm{Kx}\ue89e{(}_{1}^{a}\ue89e,{x}_{2}^{a})& \phantom{\rule{0.3em}{0.3ex}}& {\mathrm{Kx}}_{1}^{a},{x}_{N}^{a}))\\ \vdots & \vdots & \vdots & \vdots \\ K\ue8a0\left({x}_{N}^{a},{x}_{1}^{a}\right)& K\ue8a0\left({x}_{N}^{a},{x}_{2}^{a}\right)& \phantom{\rule{0.3em}{0.3ex}}& {\mathrm{Kx}}_{N}^{a},{x}_{N}^{a})\end{array}\right);& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e39\right)\end{array}$  K maybe, for example, K (x_{i},x_{j}) =diag (e^{−a∥x} ^{ i } ^{−x} ^{ j } ^{∥}). Next, the metric between the CAD models and the candidate photographic feature points is computed according to (x_{n} ^{a} x_{n} ^{1}=Ax_{n} ^{a}+b+u(x_{n} ^{a}), n=1, . . . , N and CAD^{a }of small distance is or inexact matching (σ=0 or inexact a σ≠0) can be used:

$\begin{array}{cc}{\mathrm{exactCAD}}^{\hat{a}}=\mathrm{arg}\ue89e\underset{{\mathrm{CAD}}^{a},a=1,2,\dots}{\mathrm{min}}\ue89e\underset{A,b}{\mathrm{min}}\ue89e\sum _{\mathrm{ij}=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left(\begin{array}{c}{\mathrm{Ax}}_{i}+\\ b\\ {x}_{i}^{1}\end{array}\right)}^{t}\ue89e{\left({K}^{1}\right)}_{\mathrm{ij}}\ue89e\left(\begin{array}{c}{\mathrm{Ax}}_{j}+\\ b\\ {x}_{j}^{1}\end{array}\right);& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e40\right)\\ {\mathrm{inexactCAD}}^{\hat{a}}=\mathrm{arg}\ue89e\underset{{\mathrm{CAD}}^{a},a=1,2,\dots}{\mathrm{min}}\ue89e\underset{A,b}{\mathrm{min}}\ue89e\sum _{\mathrm{ij}=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left(\begin{array}{c}{\mathrm{Ax}}_{i}+\\ b\\ {x}_{i}^{1}\end{array}\right)}^{t}\ue89e{\left({\left(\begin{array}{c}K+\\ {\sigma}^{2}\ue89eI\end{array}\right)}^{1}\right)}_{\mathrm{ij}}\ue89e\left(\begin{array}{c}{\mathrm{Ax}}_{j}+\\ b\\ {x}_{j}^{1}\end{array}\right).& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e41\right)\end{array}$  The minimum none is determined by the error between the CAD model feature points and the photographic feature points.
 The present invention may also be used to match to articulated target objects. The diffeomorphism and realtime mapping techniques carry the template 3D representations bijectively onto the target models. carrying all of the information in the template. The template models are labeled with different regions corresponding to different components of the articulated object. For example, in the case of a face, the articulated regions may include teeth, eyebrows, eyelids, eyes, and jaw. Each of these subcomponents can be articulated during motion of the model according to an articulation model specifying allowed modes of articulation. The mapping techniques carry these triangulated subcomponents onto the targets, thereby labeling them with their subcomponents automatically. The resulting selected CAD model therefore has its constituent parts automatically labeled, thereby allowing each avatar to be articulated during motion sequences.
 In the case when direct 3D measurement of the source object is available x_{n}y_{n }∈ ^{3}, n=1, . . . , N from points, curves, surfaces, or subvolumes, the techniques for determining the rotation/translation correspondences are unchanged. However, since the matching terms involve direct measurements in the volume, there is no need for the intermediate step to determine the dependence on the unknown zdepth via the MMSE technique.
 Accordingly, the best matching rigid motion corresponds to:

$\begin{array}{cc}\underset{O,b}{\mathrm{min}}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\uf605{\mathrm{Ox}}_{n}+b{y}_{n}\uf606}^{2}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e42\right)\end{array}$  The realtime deformation corresponds to:

$\begin{array}{cc}\underset{u,{z}_{n}}{\mathrm{min}}\ue89e{\uf605\mathrm{Lu}\uf606}^{2}+\sum _{n=1}^{N}\ue89e{\uf605\left({x}_{n}+u\ue8a0\left({x}_{n}\right)\right){y}_{n}\uf606}^{2}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e43\right)\end{array}$  The diffeomorphism deformation corresponds to with

$\begin{array}{cc}\phi ={\int}_{0}^{1}\ue89e{v}_{t}\ue8a0\left({\phi}_{t}\ue8a0\left(x\right)\right)\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+x,x\in {\mathbb{R}}^{3}:\text{}\ue89e\underset{{v}_{1}\ue89et\in \left[0,1\right],{z}_{n}}{\mathrm{min}}\ue89e{\int}_{0}^{1}\ue89e{\uf605{\mathrm{Lv}}_{t}\uf606}^{2}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\uf605\phi \ue8a0\left({x}_{n}\right){y}_{n}\uf606}^{2}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e44\right)\end{array}$  The techniques described herein also allow for the automated calibration camera parameters, such as the aspect ratio and field of view. The set of x_{j}=(x_{j},y_{j},z_{j}), j=1, . . . , N features are defined on the CAD model. The positive depth projective geometry mapping with

${a}_{1}\ue89e\frac{1}{{\gamma}_{1}},{a}_{2}=\frac{1}{{\gamma}_{2}}$  is defined, according to

$\left(x,y,z\right)\mapsto p\ue8a0\left(x,y,z\right)=\left(\frac{x}{{\gamma}_{1}\ue89ez},\frac{y}{{\gamma}_{2}\ue89ez}\right),z\in \left[0\ue89e\infty \right),n>0.$  Given are observations of some features through the projective geometry

${p}_{j}=\left(\frac{{x}_{j}}{{\gamma}_{2}\ue89e{z}_{j}},\frac{{y}_{j}}{{z}_{j}}\right).$  The calibration of the camera is determined under the assumption that there is no transformation (affine or other) of the avatar. The z value is parameterized by incorporating the n frustrum distance so that all depth coordinates are the above coordinates plus frustrum depth. Videos can show different aspect ratios

$\mathrm{AR}=\frac{{a}_{1}}{{a}_{2}}$  and fieldsofview

$\mathrm{FOV}=2\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\mathrm{tan}}^{1}\ue89e\frac{1}{{a}_{1}}.$  The technique estimates the aspect ratios γ_{1},γ_{2 }from measured points P_{i}(γ_{1}P_{i1},γ_{2}P_{i2}, 1)^{t}, i=1, . . . N:

$\begin{array}{cc}\underset{O,{\mathrm{by}}_{1},{y}_{2},{z}_{i},i=1,\dots}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\uf605\begin{array}{c}{\mathrm{Ox}}_{i}+\\ b{z}_{i}\ue89e{P}_{i}\end{array}\uf606}^{2}=\underset{O,{\mathrm{by}}_{1},{y}_{2}}{\mathrm{min}}\ue89e\sum _{i=1}^{N}\ue89e\uf605{\mathrm{Ox}}_{i}+{b}^{2}\uf606\sum _{i=1}^{N}\ue89e\frac{{\uf603{\left({\mathrm{Ox}}_{i}+b\right)}^{t}\ue89e{P}_{i}\uf604}^{2}}{{\uf605{P}_{i}\uf606}^{2}}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e45\right)\end{array}$  Using the initialization γ_{1} ^{new}=γ_{2} ^{new}=1, the calculation is run to convergence.
 In the first step, the data terms

${P}_{i}=\left(\begin{array}{c}{\gamma}_{1}^{\mathrm{old}}\ue89e{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\\ {\gamma}_{2}^{\mathrm{old}}\ue89e{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\\ 1\end{array}\right)$  are solved, and the following rotations/translations are computed:

$\begin{array}{cc}{Q}_{i}=\left(I\frac{{{P}_{i}\ue8a0\left({P}_{i}\right)}^{t}}{{\uf605{P}_{i}\uf606}^{2}}\right),\text{}\ue89e\stackrel{\_}{Q}=\sum _{i=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{Q}_{i},{X}_{Q}=\sum _{i=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{Q}_{i}\ue89e{X}_{i},{M}_{i}={X}_{i}{\stackrel{\_}{Q}}^{1}\ue89e{X}_{Q};& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e46\right)\\ \hat{O}=\mathrm{arg}\ue89e\underset{O}{\mathrm{min}}\ue89e{O}^{t}\ue8a0\left(\sum _{i=1}^{N}\ue89e{M}_{i}^{1}\ue89e{Q}_{i}\ue89e{M}_{i}\right)\ue89eO,\text{}\ue89e\begin{array}{c}\hat{b}={\left(\sum _{i=1}^{N}\ue89e{Q}_{i}\right)}^{1}\ue89e\sum _{i=1}^{N}\ue89e{Q}_{i}\ue89e\hat{O}\ue89e{x}_{i},{\psi}_{i}\\ =\left(\begin{array}{c}{\left(\hat{O}\ue89e{x}_{i}+\hat{b}\right)}_{1}\ue89e{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\\ {\left(\hat{O}\ue89e{x}_{i}+\hat{b}\right)}_{2}\ue89e{P}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\\ {\left(\hat{O}\ue89e{x}_{i}+\hat{b}\right)}_{3}\end{array}\right)\end{array}& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e47\right)\end{array}$  Next, the expression

${\mathrm{max}}_{\gamma \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1,\mathrm{\gamma 2}}\ue89e\sum _{i=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\frac{{\uf603{\psi}_{1\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89ei}\ue89e{\gamma}_{1}+{\psi}_{2\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89ei}\ue89e{\gamma}_{2}+{\psi}_{3\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89ei}\uf604}^{2}}{{P}_{1\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89ei}^{2}\ue89e{\gamma}_{1}^{2}+{P}_{2\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89ei}^{2}\ue89e{\gamma}_{2}^{2}+1}$  is maximized using an optimization method, such as Newton Raphson, gradient or conjugate gradient. Using the gradient algorithm, for example, the calculation is run to convergence, and the first step is repeated. The gradient method is shown here, with stepsize selected for stability:

$\begin{array}{cc}\left(\begin{array}{c}{\gamma}_{1}^{\mathrm{new}}\\ {\gamma}_{2}^{\mathrm{new}}\end{array}\right)=\left(\begin{array}{c}{\gamma}_{1}^{\mathrm{old}}\\ {\gamma}_{2}^{\mathrm{old}}\end{array}\right)+{\partial}_{\gamma}\ue89e\left({\gamma}^{\mathrm{old}}\right)\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{step}\mathrm{size}& \left(\mathrm{Equation}\right)\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e48)\\ {\partial}_{{\gamma}_{1}}\ue89e=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2\ue89e{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue89e\frac{\begin{array}{c}{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue89e{\gamma}_{1}+{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue89e{\gamma}_{2}+\\ {\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e3}\end{array}}{{\left(\begin{array}{c}{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}^{2}\ue89e{\gamma}_{1}^{2}+{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}^{2}\ue89e{\gamma}_{2}^{2}+\\ 1\end{array}\right)}^{2}}\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2\ue89e{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue89e\frac{{\left(\begin{array}{c}{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue89e{\gamma}_{1}+{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue89e{\gamma}_{2}+\\ {\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e3}\end{array}\right)}^{2}}{{\left(\begin{array}{c}{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}^{2}\ue89e{\gamma}_{1}^{2}+{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}^{2}\ue89e{\gamma}_{2}^{2}+\\ 1\end{array}\right)}^{2}}& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e49\right)\\ {\partial}_{{\gamma}_{2}}\ue89e=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2\ue89e{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue89e\frac{\begin{array}{c}{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue89e{\gamma}_{1}+{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue89e{\gamma}_{2}+\\ {\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e3}\end{array}}{{\left(\begin{array}{c}{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}^{2}\ue89e{\gamma}_{1}^{2}+{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}^{2}\ue89e{\gamma}_{2}^{2}+\\ 1\end{array}\right)}^{2}}\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2\ue89e{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue89e\frac{{\left(\begin{array}{c}{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue89e{\gamma}_{1}+{\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue89e{\gamma}_{2}+\\ {\psi}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e3}\end{array}\right)}^{2}}{{\left(\begin{array}{c}{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}^{2}\ue89e{\gamma}_{1}^{2}+{p}_{i\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}^{2}\ue89e{\gamma}_{2}^{2}+\\ 1\end{array}\right)}^{2}}.& \left(\mathrm{Equation}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e50\right)\end{array}$  The techniques described herein may be used to compare a source 3D object to a single reference object. 2D representations of the source object and the reference object are created, and the correspondence between them is characterized using mathematical optimization and projective geometry. Typically, the correspondence is characterized by specifying the viewpoint from which the 2D source projection was captured.
 Refer now to
FIG. 4 , which illustrates a hardware system 400 incorporating the invention. As indicated therein, the system includes a video source 402 (e.g., a video camera or a scanning device) which supplies a still input image to be analyzed. The output of the video source 402 is digitized as a frame into an array of pixels by a digitizer 404. The digitized images are transmitted along the system bus 406 over which all system components communicate, and may be stored in a mass storage device (such as a hard disc or optical storage unit) 408 as well as in main system memory 410 (specifically, within a partition defining a series of identically sized input image buffers) 412.  The operation of the illustrated system is directed by a centralprocessing unit (“CPU”) 414. To facilitate rapid execution of the imageprocessing operations hereinafter described, the system preferably contains a graphics or imageprocessing board 416; this is a standard component wellknown to those skilled in the art.
 The user interacts with the system using a keyboard 418 and a positionsensing device (e.g., a mouse) 420. The output of either device can be used to designate information or select particular points or areas of a screen display 420 to direct functions performed by the system.
 The main memory 410 contains a group of modules that control the operation of the CPU 414 and its interaction with the other hardware components. An operating system 424 directs the execution of lowlevel, basic system functions such as memory allocation, file management and operation of mass storage devices 408. At a higher level, the analyzer 426, implemented as a series of stored instructions, directs execution of the primary functions performed by the invention, as discussed below: and instructions defining a user interface 428 allow straightforward interaction over screen display 422. The user interface 428 generates words or graphical images on the display 422 to prompt action by the user, and accepts commands from the keyboard 418 and/or positionsensing device 420. Finally, the memory 410 includes a partition 430 for storing for storing a database of 3D reference avatars, as described above.
 The contents of each image buffer 412 define a “raster,” i.e., a regular 2D pattern of discrete pixel positions that collectively represent an image and may be used to drive (e.g., by means of imageprocessing board 416 or an image server) screen display 422 to display that image. The content of each memory location in a frame buffer directly governs the appearance of a corresponding pixel on the display 422.
 It must be understood that although the modules of main memory 410 have been described separately, this is for clarity of presentation only; so long as the system performs all the necessary functions, it is immaterial how they are distributed within the system and the programming architecture thereof Likewise, though conceptually organized as grids, pixelmaps need not actually be stored digitally in this fashion. Rather, for convenience of memory utilization and transmission, the raster pattern is usually encoded as an ordered array of pixels.
 As noted above, execution of the key tasks associated with the present invention is directed by the analyzer 426, which governs the operation of the CPU 414 and controls its interaction with main memory 410 in performing the steps necessary to match and deform reference 3D representations to match a target multifeatured object.
FIG. 5 illustrates the components of a preferred implementation of the analyzer 426. The projection module 502 takes a 3D model and makes a 2D projection of it onto any chosen plane. In general, an efficient projection module 502 will be required in order to create numerous projections over the space of rotations and translations for each of the candidate reference avatars. The deformation module 504 performs one or more types of deformation on an avatar in order to make it more closely resemble the source object. The deformation is performed in 3D space, with every point defining the avatar mesh being free to move in order to optimize the fit to the conditional mean estimates of the reverse projected feature items from the source imagery. In general. deformation is only applied to the bestfitting reference object, if more than one reference object is supplied. The rendering module 506 allows for the rapid projection of a 3D avatar into 2D with the option of including the specification of the avatar lighting. The 2D projection corresponds to the chosen lighting of the 3D avatar. The feature detection module 508 searches for specific feature items in the 2D source projection. The features may include eyes, nostrils, lips, and may incorporate probes that operate at several different pixel scales. 
FIG. 6 illustrates the functions of the invention performed in main memory. In step 602, the system examines the source imagery and automatically detects features of a face. such as eyeballs, nostrils, and lips that can be used for matching purposes, as described above. In step 604, the detected feature items are reverse projected into the coordinate frame of the candidate avatar, as described above and using equation 7. In step 606, the optimum rotation/translation of the candidate avatar is estimated using the techniques described above and using equations 8, 9 and 10. In step 608, any prior information that may be available about the position of the source object with respect to the available 2D projections is added into the computation, as described herein using equations 1 113. When 3D measurements of the source are available, this data is used to constrain the rigid motion search as shown in step 610 and as described above with reference to equations 4143. When the rotation/translation search 606 is completed over all the reference 3D avatars, the bestfitting avatar is selected in step 612, as described above, with reference to equations 3840. Subsequently, the bestfitting avatar located in step 612 is deformed in step 614. 3D measurements of the source object 610, if any, are used to constrain the deformation 614. In addition, portions of the source imagery 616 itself may be used to influence the deformation 614.  The invention provides for several different kinds of deformation which may be optionally applied to the bestfitting reference avatar in order to improve its correspondence with the target object. The deformations may include realtime deformation without rigid motions in which a closed form expression is found for the deformation, as described above using equations 18, 19. A diffeomorphic deformation of the avatar with no rigid motions may be applied (equations 2224). Alternatively, a real time deformation with unknown rigid motion of the avatar may be deployed (equations 28, 29). A realtime diffeormorphic deformation may be applied to the avatar by iterating the realtime deformation. The avatar may be deformed using affine motions (equations 30, 31). The deformation of the avatar may be guided by matching a projection to large numbers of feature items in the source data, including the identification of submanifolds within the avatar, as described above with reference to equation 37. When the target object is described by an articulated model, the deformations described above may be applied to each articulated component separately.
 The invention enables camera parameters, such as aspect ratio and field of view to be estimated as shown in step 618 and described above. with reference to equations 4449.
 As noted previously, while certain aspects of the hardware implementation have been described for the case where the target object is a face and the reference object is an avatar, the invention is not limited to the matching of faces, but may be used for matching any multifeatured object using a database of reference 3D representations that correspond to the generic type of the target object to be matched.
 It will therefore be seen that the foregoing represents a highly extensible and advantageous approach to the generation of 3D models of a target multifeatured object when only partial information describing the object is available. The terms and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. For example, the various modules of the invention can be implemented on a generalpurpose computer using appropriate software instructions, or as hardware circuits, or as mixed hardwaresoftware combinations (wherein, for example, pixel manipulation and rendering is performed by dedicated hardware components).
Claims (8)
176. (canceled)
77. A method involving at least one source 2D image of a source 3D object, the method comprising:
providing a reference 3D representation; and
simultaneously searching over rigid motions and deformations of the reference 3D representation to identify a bestmatch reference 3D representation most resembling the at least one source 2D projection,
wherein simultaneously searching comprises searching over rigid motions of the reference 3D representation and for each orientation of the reference 3D representation in the search over rigid motions of the reference 3D representation, using a closed form expression to compute a deformed reference 3D representation that generates a bestfit with the at least one source 2D image, and
wherein the bestmatch 3D representation is the deformed reference 3D representation for the orientation in the search that yields a best bestfit.
78. The method of claim 77 , wherein the closed form expression corresponds to a backprojection into 3D space of the at least one source 2D image.
79. The method of claim 1, wherein the closed form expression corresponds to deprojected positions in 3D space of feature items from the at least one source 2D image.
80. The method of claim 1, wherein simultaneously searching over rigid motions and deformations of the reference 3D representation to identify the bestmatch reference 3D representation most resembling the at least one source 2D projection is performed without generating any 2D projections of the reference 3D representation or deformed versions thereof.
81. A method involving at least one source 2D image of a source 3D object, the method comprising:
providing a reference 3D representation; and
searching over rigid motions and deformations of the reference 3D representation to identify a best match reference 3D representation most resembling the at least one source 2D projection,
wherein said searching involves applying both rigid motion and deformation operators to that reference 3D representation to generate multiple versions of the reference 3D representation, and for each version of the reference 3D representation, computing a measure of fit between that version of the reference 3D representation and the at least one source 2D image,
wherein the bestfit 3D representation is the version of the reference 3D representation that yields a best measure of fit, and
wherein the deformation operators are numerical representations of transformations that are of infinite dimension.
82. The method of claim 1, wherein searching over rigid motions and deformations of the reference 3D representation to identify a best match reference 3D representation most resembling the at least one source 2D projection is performed without actually generating any projections.
83. A method involving at least one source 2D image of a source 3D object, the method comprising:
providing a reference 3D representation; and
searching over rigid motions and deformations of the reference 3D representation to identify a best match reference 3D representation most resembling the at least one source 2D projection,
wherein said searching involves applying both rigid motion and deformation operators to that reference 3D representation to generate multiple versions of the reference 3D representation, and for each version of the reference 3D representation, computing a measure of fit between that version of the reference 3D representation and the at least one source 2D image,
wherein the bestfit 3D representation is the version of the reference 3D representation that yields a best measure of fit, and
wherein searching over rigid motions and deformations of the reference 3D representation to identify a best match reference 3D representation most resembling the at least one source 2D projection is performed without actually generating any projections.
Priority Applications (5)
Application Number  Priority Date  Filing Date  Title 

US45243103P true  20030306  20030306  
US45242903P true  20030306  20030306  
US45243003P true  20030306  20030306  
US10/794,353 US7643685B2 (en)  20030306  20040305  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
US12/635,999 US20100295854A1 (en)  20030306  20091211  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

US12/635,999 US20100295854A1 (en)  20030306  20091211  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
Related Parent Applications (1)
Application Number  Title  Priority Date  Filing Date  

US10/794,353 Continuation US7643685B2 (en)  20030306  20040305  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
Publications (1)
Publication Number  Publication Date 

US20100295854A1 true US20100295854A1 (en)  20101125 
Family
ID=32995971
Family Applications (4)
Application Number  Title  Priority Date  Filing Date 

US10/794,352 Active  Reinstated 20260517 US7643683B2 (en)  20030306  20040305  Generation of image database for multifeatured objects 
US10/794,353 Active  Reinstated 20261007 US7643685B2 (en)  20030306  20040305  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
US10/794,943 Active 20280310 US7853085B2 (en)  20030306  20040305  Viewpointinvariant detection and identification of a threedimensional object from twodimensional imagery 
US12/635,999 Abandoned US20100295854A1 (en)  20030306  20091211  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
Family Applications Before (3)
Application Number  Title  Priority Date  Filing Date 

US10/794,352 Active  Reinstated 20260517 US7643683B2 (en)  20030306  20040305  Generation of image database for multifeatured objects 
US10/794,353 Active  Reinstated 20261007 US7643685B2 (en)  20030306  20040305  Viewpointinvariant image matching and generation of threedimensional models from twodimensional imagery 
US10/794,943 Active 20280310 US7853085B2 (en)  20030306  20040305  Viewpointinvariant detection and identification of a threedimensional object from twodimensional imagery 
Country Status (4)
Country  Link 

US (4)  US7643683B2 (en) 
EP (3)  EP1599830A1 (en) 
JP (3)  JP2006520055A (en) 
WO (3)  WO2004081855A1 (en) 
Cited By (10)
Publication number  Priority date  Publication date  Assignee  Title 

US20100289664A1 (en) *  20081015  20101118  Panasonic Corporation  Light projection device 
US20110316853A1 (en) *  20100623  20111229  Microsoft Corporation  Telepresence systems with viewer perspective adjustment 
US20120086783A1 (en) *  20100608  20120412  Raj Sareen  System and method for body scanning and avatar creation 
US20130201187A1 (en) *  20110809  20130808  Xiaofeng Tong  Imagebased multiview 3d face generation 
US8570343B2 (en) *  20100420  20131029  Dassault Systemes  Automatic generation of 3D models from packaged goods product images 
WO2014043755A1 (en) *  20120919  20140327  Commonwealth Scientific And Industrial Research Organisation  System and method of generating a nonrigid model 
US20160078663A1 (en) *  20100608  20160317  Styku, Inc.  Cloud server body scan data system 
US20160088284A1 (en) *  20100608  20160324  Styku, Inc.  Method and system for determining biometrics from body surface imaging technology 
US20160275688A1 (en) *  20131101  20160922  The Research Foundation For The State University Of New York  Method for Measuring the Interior ThreeDimensional Movement, Stress and Strain of an Object 
US10269175B2 (en)  20150714  20190423  Samsung Electronics Co., Ltd.  Three dimensional content generating apparatus and three dimensional content generating method thereof 
Families Citing this family (127)
Publication number  Priority date  Publication date  Assignee  Title 

EP1451775A1 (en) *  20011124  20040901  TDV Technologies Corp.  Generation of a stereo image sequence from a 2d image sequence 
US7421098B2 (en) *  20021007  20080902  Technion Research & Development Foundation Ltd.  Facial recognition and the open mouth problem 
US6947579B2 (en) *  20021007  20050920  Technion Research & Development Foundation Ltd.  Threedimensional face recognition 
AU2004282790A1 (en) *  20031021  20050428  Nec Corporation  Image collation system and image collation method 
US20050152504A1 (en) *  20040113  20050714  Ang Shih  Method and apparatus for automated tomography inspection 
EP1766552A2 (en) *  20040623  20070328  Strider Labs, Inc.  System and method for 3d object recognition using range and intensity 
US7542034B2 (en)  20040923  20090602  Conversion Works, Inc.  System and method for processing video images 
US7480414B2 (en) *  20041014  20090120  International Business Machines Corporation  Method and apparatus for object normalization using object classification 
ES2373366T3 (en) *  20041022  20120202  Shiseido Co., Ltd.  Lip makeup procedure. 
US20060127852A1 (en) *  20041214  20060615  Huafeng Wen  Image based orthodontic treatment viewing system 
KR100601989B1 (en) *  20050207  20060710  삼성전자주식회사  Apparatus and method for estimating 3d face shape from 2d image and computer readable media for storing computer program 
WO2006093250A1 (en) *  20050304  20060908  National University Corporation NARA Institute of Science and Technology  Movement measuring device, movement measuring system, onvehicle equipment, movement measuring method, movement measuring program, and computerreadable recording medium 
US7760917B2 (en)  20050509  20100720  Like.Com  Computerimplemented method for performing similarity searches 
US7657126B2 (en)  20050509  20100202  Like.Com  System and method for search portions of objects in images and features thereof 
US8732025B2 (en)  20050509  20140520  Google Inc.  System and method for enabling image recognition and searching of remote content on display 
US7945099B2 (en)  20050509  20110517  Like.Com  System and method for use of images with recognition analysis 
US7519200B2 (en)  20050509  20090414  Like.Com  System and method for enabling the use of captured images through recognition 
US7660468B2 (en)  20050509  20100209  Like.Com  System and method for enabling image searching using manual enrichment, classification, and/or segmentation 
US7783135B2 (en)  20050509  20100824  Like.Com  System and method for providing objectified image renderings using recognition information from images 
US20080177640A1 (en)  20050509  20080724  Salih Burak Gokturk  System and method for using image analysis and search in ecommerce 
EP1886257A1 (en)  20050511  20080213  Optosecurity Inc.  Method and system for screening luggage items, cargo containers or persons 
WO2006119629A1 (en) *  20050511  20061116  Optosecurity Inc.  Database of target objects suitable for use in screening receptacles or people and method and apparatus for generating same 
US7899232B2 (en)  20060511  20110301  Optosecurity Inc.  Method and apparatus for providing threat image projection (TIP) in a luggage screening system, and luggage screening system implementing same 
US7991242B2 (en)  20050511  20110802  Optosecurity Inc.  Apparatus, method and system for screening receptacles and persons, having image distortion correction functionality 
JP4691158B2 (en) *  20050616  20110601  ストライダー ラブス，インコーポレイテッド  Recognition system and method in the twodimensional image using 3D class model 
JP2007064836A (en) *  20050831  20070315  Kyushu Institute Of Technology  Algorithm for automating camera calibration 
US20070080967A1 (en) *  20051011  20070412  Animetrics Inc.  Generation of normalized 2D imagery and ID systems via 2D to 3D lifting of multifeatured objects 
US7961937B2 (en) *  20051026  20110614  HewlettPackard Development Company, L.P.  Prenormalization data classification 
US8094928B2 (en)  20051114  20120110  Microsoft Corporation  Stereo video for gaming 
US20070124330A1 (en) *  20051117  20070531  Lydia Glass  Methods of rendering information services and related devices 
TWI318756B (en) *  20051229  20091221  Ind Tech Res Inst  
WO2007087389A2 (en)  20060123  20070802  HyKo Products Company  Key duplication machine 
US9101990B2 (en)  20060123  20150811  HyKo Products  Key duplication machine 
WO2007092629A2 (en) *  20060209  20070816  Nms Communications Corporation  Smooth morphing between personal video calling avatars 
US8571272B2 (en)  20060312  20131029  Google Inc.  Techniques for enabling or establishing the use of face recognition algorithms 
US9690979B2 (en)  20060312  20170627  Google Inc.  Techniques for enabling or establishing the use of face recognition algorithms 
JP4362489B2 (en) *  20060329  20091111  中部日本電気ソフトウェア株式会社  3dimensional face data registering system, 3dimensional face data registering, and registration processing program 
JP4785598B2 (en) *  20060407  20111005  株式会社日立製作所  Similar shape search unit 
US8233702B2 (en) *  20060818  20120731  Google Inc.  Computer implemented technique for analyzing images 
WO2008036354A1 (en)  20060919  20080327  Braintech Canada, Inc.  System and method of determining object pose 
TWI332639B (en) *  20060927  20101101  Compal Electronics Inc  Method for displaying expressional image 
WO2008076942A1 (en) *  20061215  20080626  Braintech Canada, Inc.  System and method of identifying objects 
US8170297B2 (en) *  20070119  20120501  Konica Minolta Holdings, Inc.  Face authentication system and face authentication method 
US8655052B2 (en)  20070126  20140218  Intellectual Discovery Co., Ltd.  Methodology for 3D scene reconstruction from 2D image sequences 
JP2008187591A (en) *  20070131  20080814  Fujifilm Corp  Imaging apparatus and imaging method 
US20080225045A1 (en) *  20070312  20080918  Conversion Works, Inc.  Systems and methods for 2d to 3d image conversion using mask to model, or model to mask, conversion 
US8274530B2 (en)  20070312  20120925  Conversion Works, Inc.  Systems and methods for filling occluded information for 2D to 3D conversion 
US20080228449A1 (en) *  20070312  20080918  Conversion Works, Inc.  Systems and methods for 2d to 3d conversion using depth access segments to define an object 
US8494210B2 (en)  20070330  20130723  Optosecurity Inc.  User interface for use in security screening providing image enhancement capabilities and apparatus for implementing same 
JP4337064B2 (en) *  20070404  20090930  ソニー株式会社  The information processing apparatus, information processing method, and program 
JP5096776B2 (en) *  20070404  20121212  キヤノン株式会社  An image processing apparatus and an image retrieval method 
NO327279B1 (en) *  20070522  20090602  Metaio Gmbh  Camera Pose Estimation Device and progress feed for enhanced reality image 
US7966567B2 (en) *  20070712  20110621  Center'd Corp.  Character expression in a geospatial environment 
US8416981B2 (en)  20070729  20130409  Google Inc.  System and method for displaying contextual supplemental content based on image content 
EP2048599B1 (en)  20071011  20091216  MVTec Software GmbH  System and method for 3D object recognition 
US8059888B2 (en) *  20071030  20111115  Microsoft Corporation  Semiautomatic plane extrusion for 3D modeling 
US8862582B2 (en) *  20071115  20141014  At&T Intellectual Property I, L.P.  System and method of organizing images 
US8190604B2 (en) *  20080403  20120529  Microsoft Corporation  User intention modeling for interactive image retrieval 
US20100017033A1 (en) *  20080718  20100121  Remus Boca  Robotic systems with user operable robot control terminals 
JP5253066B2 (en) *  20080924  20130731  キヤノン株式会社  Position and orientation measuring apparatus and method 
US8368689B2 (en) *  20080925  20130205  Siemens Product Lifecycle Management Software Inc.  System, method, and computer program product for radial functions and distributions of three dimensional object models 
US8559699B2 (en)  20081010  20131015  Roboticvisiontech Llc  Methods and apparatus to facilitate operations in image based systems 
US8159327B2 (en) *  20081113  20120417  Visa International Service Association  Device including authentication glyph 
JP2010186288A (en) *  20090212  20100826  Seiko Epson Corp  Image processing for changing predetermined texture characteristic amount of face image 
US9740921B2 (en)  20090226  20170822  Tko Enterprises, Inc.  Image processing sensor systems 
US9277878B2 (en) *  20090226  20160308  Tko Enterprises, Inc.  Image processing sensor systems 
US9299231B2 (en) *  20090226  20160329  Tko Enterprises, Inc.  Image processing sensor systems 
CN102448636B (en)  20090501  20140910  海高制品有限公司  Key blank identification system with bitting analysis 
MX343763B (en)  20090501  20161118  HyKo Products  Key blank identification system with groove scanning. 
WO2010131371A1 (en) *  20090512  20101118  Toyota Jidosha Kabushiki Kaisha  Object recognition method, object recognition apparatus, and autonomous mobile robot 
US20100313141A1 (en) *  20090603  20101209  Tianli Yu  System and Method for Learning User Genres and Styles and for Matching Products to User Preferences 
US8553972B2 (en) *  20090706  20131008  Samsung Electronics Co., Ltd.  Apparatus, method and computerreadable medium generating depth map 
US20110025689A1 (en) *  20090729  20110203  Microsoft Corporation  AutoGenerating A Visual Representation 
US8604796B2 (en) *  20091008  20131210  Precision Energy Services, Inc.  Steerable magnetic dipole antenna for measurementwhiledrilling applications 
US9366780B2 (en)  20091008  20160614  Precision Energy Services, Inc.  Steerable magnetic dipole antenna for measurement while drilling applications 
JP2011090466A (en) *  20091021  20110506  Sony Corp  Information processing apparatus, method, and program 
KR20110070056A (en) *  20091218  20110624  한국전자통신연구원  Method and apparatus for easy and intuitive generation of usercustomized 3d avatar with highquality 
US20110268365A1 (en) *  20100430  20111103  Acer Incorporated  3d hand posture recognition system and vision based hand posture recognition method thereof 
EP2385483B1 (en)  20100507  20121121  MVTec Software GmbH  Recognition and pose determination of 3D objects in 3D scenes using geometric point pair descriptors and the generalized Hough Transform 
US8452721B2 (en) *  20100615  20130528  Nvidia Corporation  Region of interest tracking for fluid simulation 
US8416990B2 (en)  20100817  20130409  Microsoft Corporation  Hierarchical video subvolume search 
US8432392B2 (en) *  20100902  20130430  Samsung Electronics Co., Ltd.  Display system with image conversion mechanism and method of operation thereof 
US9317533B2 (en)  20101102  20160419  Microsoft Technology Licensing, Inc.  Adaptive image retrieval database 
US8463045B2 (en)  20101110  20130611  Microsoft Corporation  Hierarchical sparse representation for image retrieval 
US8711210B2 (en) *  20101214  20140429  Raytheon Company  Facial recognition using a sphericity metric 
WO2012082077A2 (en) *  20101217  20120621  Agency For Science, Technology And Research  Poseindependent 3d face reconstruction from a sample 2d face image 
EP2656245A1 (en) *  20101221  20131030  Ecole Polytechnique Fédérale de Lausanne (EPFL)  Computerized method and device for annotating at least one feature of an image of a view 
DE102011008074A1 (en) *  20110107  20120712  Martin Tank  Method and dental restoration detection system for the detection of dental restorations 
US9952046B1 (en)  20110215  20180424  Guardvant, Inc.  Cellular phone and personal protective equipment usage monitoring system 
US9198575B1 (en)  20110215  20151201  Guardvant, Inc.  System and method for determining a level of operator fatigue 
US8988512B2 (en) *  20110414  20150324  Mediatek Inc.  Method for adjusting playback of multimedia content according to detection result of user status and related apparatus thereof 
CA2845730A1 (en) *  20110818  20130221  Utherverse Digital, Inc.  Systems and methods of virtual world interaction 
JP5143262B1 (en) *  20110830  20130213  株式会社東芝  Stereoscopic video processing apparatus, a stereoscopic image processing method 
US9111331B2 (en)  20110907  20150818  Rapiscan Systems, Inc.  Xray inspection system that integrates manifest data with imaging/detection processing 
KR20130063310A (en) *  20111206  20130614  엘지전자 주식회사  Mobile terminal and control method for mobile terminal 
WO2013086137A1 (en)  20111206  20130613  1800 Contacts, Inc.  Systems and methods for obtaining a pupillary distance measurement using a mobile computing device 
US10013787B2 (en) *  20111212  20180703  Faceshift Ag  Method for facial animation 
WO2013145496A1 (en) *  20120327  20131003  日本電気株式会社  Information processing device, information processing method, and program 
US20130314401A1 (en)  20120523  20131128  1800 Contacts, Inc.  Systems and methods for generating a 3d model of a user for a virtual tryon product 
US9483853B2 (en)  20120523  20161101  Glasses.Com Inc.  Systems and methods to display rendered images 
US9286715B2 (en)  20120523  20160315  Glasses.Com Inc.  Systems and methods for adjusting a virtual tryon 
US9141848B2 (en) *  20120904  20150922  Intel Corporation  Automatic media distribution 
US9020982B2 (en)  20121015  20150428  Qualcomm Incorporated  Detection of planar targets under steep angles 
US9743002B2 (en) *  20121119  20170822  Magna Electronics Inc.  Vehicle vision system with enhanced display functions 
WO2014094199A1 (en) *  20121217  20140626  Intel Corporation  Facial movement based avatar animation 
US9990373B2 (en) *  20130206  20180605  John A. Fortkort  Creation and geospatial placement of avatars based on realworld interactions 
US9381426B1 (en) *  20130315  20160705  University Of Central Florida Research Foundation, Inc.  Semiautomated digital puppetry control 
US9990004B2 (en) *  20130402  20180605  Samsung Dispaly Co., Ltd.  Optical detection of bending motions of a flexible display 
US9449392B2 (en) *  20130605  20160920  Samsung Electronics Co., Ltd.  Estimator training method and pose estimating method using depth image 
US9839761B1 (en)  20130704  20171212  Hal Rucker  Airflow control for pressurized air delivery 
US9355123B2 (en)  20130719  20160531  Nant Holdings Ip, Llc  Fast recognition algorithm processing, systems and methods 
KR101509934B1 (en) *  20131010  20150416  재단법인대구경북과학기술원  Device of a front head pose guidance, and method thereof 
US9613449B2 (en)  20131018  20170404  Nvidia Corporation  Method and apparatus for simulating stiff stacks 
US9589383B2 (en)  20131018  20170307  Nvidia Corporation  Unified position based solver for visual effects 
WO2015089115A1 (en) *  20131209  20150618  Nant Holdings Ip, Llc  Feature density object classification, systems and methods 
US9501498B2 (en) *  20140214  20161122  Nant Holdings Ip, Llc  Object ingestion through canonical shapes, systems and methods 
US9710699B2 (en) *  20141031  20170718  Irvine Sensors Corp.  Three dimensional recognition from unscripted sources technology (TRUST) 
RU2582852C1 (en) *  20150121  20160427  Общество с ограниченной ответственностью "Вокорд СофтЛаб" (ООО "Вокорд СофтЛаб")  Automatic construction of 3d model of face based on series of 2d images or movie 
WO2017024043A1 (en)  20150803  20170209  HyKo Products Company  High security key scanning system 
US20170103563A1 (en) *  20151007  20170413  Victor Erukhimov  Method of creating an animated realistic 3d model of a person 
KR101755248B1 (en) *  20160127  20170707  (주)에이아이퍼스트  Method and system of generating 3D model and mobile device for the same 
US9868212B1 (en) *  20160218  20180116  X Development Llc  Methods and apparatus for determining the pose of an object based on point cloud data 
US10302807B2 (en)  20160222  20190528  Rapiscan Systems, Inc.  Systems and methods for detecting threats and contraband in cargo 
US20180060703A1 (en) *  20160901  20180301  Uptake Technologies, Inc.  Detection of Anomalies in Multivariate Data 
FR3060170A1 (en) *  20161214  20180615  Smart Me Up  object recognition system based on a generic model adaptive 3d 
EP3420903A1 (en) *  20170629  20190102  Siemens Healthcare GmbH  Visualisation of at least one indicator 
WO2019117959A1 (en) *  20171215  20190620  HewlettPackard Development Company, L.P.  Patterns for locations on threedimensional objects 
Citations (7)
Publication number  Priority date  Publication date  Assignee  Title 

US6002782A (en) *  19971112  19991214  Unisys Corporation  System and method for recognizing a 3D object by generating a 2D image of the object from a transformed 3D model 
US20010020946A1 (en) *  20000310  20010913  Minolta Co., Ltd.  Method and apparatus for data processing recognizing an object represented as twodimensional image 
US6381346B1 (en) *  19971201  20020430  Wheeling Jesuit University  Threedimensional face identification system 
US6434278B1 (en) *  19970923  20020813  Enroute, Inc.  Generating threedimensional models of objects defined by twodimensional image data 
US6556196B1 (en) *  19990319  20030429  MaxPlanckGesellschaft Zur Forderung Der Wissenschaften E.V.  Method and apparatus for the processing of images 
US6956569B1 (en) *  20000330  20051018  Nec Corporation  Method for matching a two dimensional image to one of a plurality of three dimensional candidate models contained in a database 
US20070031028A1 (en) *  20050620  20070208  Thomas Vetter  Estimating 3d shape and texture of a 3d object based on a 2d image of the 3d object 
Family Cites Families (28)
Publication number  Priority date  Publication date  Assignee  Title 

US5159361A (en) *  19890309  19921027  Par Technology Corporation  Method and apparatus for obtaining the topography of an object 
US5825936A (en) *  19940922  19981020  University Of South Florida  Image analyzing device using adaptive criteria 
US5761638A (en) *  19950317  19980602  Us West Inc  Telephone network apparatus and method using echo delay and attenuation 
US5742291A (en)  19950509  19980421  Synthonics Incorporated  Method and apparatus for creation of threedimensional wire frames 
US5844573A (en)  19950607  19981201  Massachusetts Institute Of Technology  Image compression by pointwise prototype correspondence using shape and texture information 
US5898438A (en) *  19961112  19990427  Ford Global Technologies, Inc.  Texture mapping of photographic images to CAD surfaces 
US6094199A (en) *  19970523  20000725  University Of Washington  3D objects morphing employing skeletons indicating symmetric differences to define intermediate objects used in morphing 
US5990901A (en)  19970627  19991123  Microsoft Corporation  Model based image editing and correction 
EP0907145A3 (en) *  19971003  20030326  Nippon Telegraph and Telephone Corporation  Method and equipment for extracting image features from image sequence 
US6249600B1 (en)  19971107  20010619  The Trustees Of Columbia University In The City Of New York  System and method for generation of a threedimensional solid model 
US6226418B1 (en)  19971107  20010501  Washington University  Rapid convolution based large deformation image matching via landmark and volume imagery 
US6362833B2 (en)  19980408  20020326  Intel Corporation  Method and apparatus for progressively constructing a series of morphs between twodimensional or threedimensional models 
JP3467725B2 (en) *  19980602  20031117  富士通株式会社  Shadow removal process of the image, the image processing apparatus and a recording medium 
US6366282B1 (en) *  19980908  20020402  Intel Corporation  Method and apparatus for morphing objects by subdividing and mapping portions of the objects 
IT1315446B1 (en)  19981002  20030211  Cselt Centro Studi Lab Telecom  Procedure for the creation of threedimensional facial models apartire by images of faces. 
JP4025442B2 (en)  19981201  20071219  富士通株式会社  The threedimensional model conversion apparatus and method 
US6296317B1 (en) *  19991029  20011002  Carnegie Mellon University  Visionbased motion sensor for mining machine control 
GB0004165D0 (en)  20000222  20000412  Digimask Limited  System for virtual threedimensional object creation and use 
US6807290B2 (en)  20000309  20041019  Microsoft Corporation  Rapid computer modeling of faces for animation 
JP2001283216A (en)  20000403  20011012  Nec Corp  Image collating device, image collating method and recording medium in which its program is recorded 
JP4387552B2 (en) *  20000427  20091216  富士通株式会社  Image matching processing system 
US6853745B1 (en)  20001103  20050208  Nec Laboratories America, Inc.  Lambertian reflectance and linear subspaces 
US6975750B2 (en) *  20001201  20051213  Microsoft Corp.  System and method for face recognition using synthesized training images 
GB2383915B (en)  20011123  20050928  Canon Kk  Method and apparatus for generating models of individuals 
US7221809B2 (en)  20011217  20070522  Genex Technologies, Inc.  Face recognition system and method 
US20030169906A1 (en)  20020226  20030911  Gokturk Salih Burak  Method and apparatus for recognizing objects 
FR2849241B1 (en) *  20021220  20050624  Biospace Instr  Method and radiographic imaging device 
CN100520809C (en)  20030328  20090729  富士通株式会社  Photographic apparatus 

2004
 20040305 JP JP2006509131A patent/JP2006520055A/en active Pending
 20040305 EP EP20040718062 patent/EP1599830A1/en not_active Withdrawn
 20040305 WO PCT/US2004/006827 patent/WO2004081855A1/en active Application Filing
 20040305 US US10/794,352 patent/US7643683B2/en active Active  Reinstated
 20040305 US US10/794,353 patent/US7643685B2/en active Active  Reinstated
 20040305 WO PCT/US2004/006604 patent/WO2004081853A1/en active Application Filing
 20040305 US US10/794,943 patent/US7853085B2/en active Active
 20040305 JP JP2006509187A patent/JP2006522411A/en active Pending
 20040305 WO PCT/US2004/006614 patent/WO2004081854A1/en active Application Filing
 20040305 EP EP20040717974 patent/EP1599828A1/en not_active Withdrawn
 20040305 EP EP20040717979 patent/EP1599829A1/en not_active Withdrawn
 20040305 JP JP2006509130A patent/JP2006520054A/en active Pending

2009
 20091211 US US12/635,999 patent/US20100295854A1/en not_active Abandoned
Patent Citations (7)
Publication number  Priority date  Publication date  Assignee  Title 

US6434278B1 (en) *  19970923  20020813  Enroute, Inc.  Generating threedimensional models of objects defined by twodimensional image data 
US6002782A (en) *  19971112  19991214  Unisys Corporation  System and method for recognizing a 3D object by generating a 2D image of the object from a transformed 3D model 
US6381346B1 (en) *  19971201  20020430  Wheeling Jesuit University  Threedimensional face identification system 
US6556196B1 (en) *  19990319  20030429  MaxPlanckGesellschaft Zur Forderung Der Wissenschaften E.V.  Method and apparatus for the processing of images 
US20010020946A1 (en) *  20000310  20010913  Minolta Co., Ltd.  Method and apparatus for data processing recognizing an object represented as twodimensional image 
US6956569B1 (en) *  20000330  20051018  Nec Corporation  Method for matching a two dimensional image to one of a plurality of three dimensional candidate models contained in a database 
US20070031028A1 (en) *  20050620  20070208  Thomas Vetter  Estimating 3d shape and texture of a 3d object based on a 2d image of the 3d object 
Cited By (15)
Publication number  Priority date  Publication date  Assignee  Title 

US20100289664A1 (en) *  20081015  20101118  Panasonic Corporation  Light projection device 
US8446288B2 (en) *  20081015  20130521  Panasonic Corporation  Light projection device 
US8570343B2 (en) *  20100420  20131029  Dassault Systemes  Automatic generation of 3D models from packaged goods product images 
US20120086783A1 (en) *  20100608  20120412  Raj Sareen  System and method for body scanning and avatar creation 
US20160088284A1 (en) *  20100608  20160324  Styku, Inc.  Method and system for determining biometrics from body surface imaging technology 
US20160078663A1 (en) *  20100608  20160317  Styku, Inc.  Cloud server body scan data system 
US20110316853A1 (en) *  20100623  20111229  Microsoft Corporation  Telepresence systems with viewer perspective adjustment 
US8928659B2 (en) *  20100623  20150106  Microsoft Corporation  Telepresence systems with viewer perspective adjustment 
US20130201187A1 (en) *  20110809  20130808  Xiaofeng Tong  Imagebased multiview 3d face generation 
WO2014043755A1 (en) *  20120919  20140327  Commonwealth Scientific And Industrial Research Organisation  System and method of generating a nonrigid model 
US9928635B2 (en)  20120919  20180327  Commonwealth Scientific And Industrial Research Organisation  System and method of generating a nonrigid model 
AU2013317700B2 (en) *  20120919  20190117  Commonwealth Scientific And Industrial Research Organisation  System and method of generating a nonrigid model 
US20160275688A1 (en) *  20131101  20160922  The Research Foundation For The State University Of New York  Method for Measuring the Interior ThreeDimensional Movement, Stress and Strain of an Object 
US10013767B2 (en) *  20131101  20180703  The Research Foundation For The State University Of New York  Method for measuring the interior threedimensional movement, stress and strain of an object 
US10269175B2 (en)  20150714  20190423  Samsung Electronics Co., Ltd.  Three dimensional content generating apparatus and three dimensional content generating method thereof 
Also Published As
Publication number  Publication date 

EP1599830A1 (en)  20051130 
WO2004081854A1 (en)  20040923 
JP2006520055A (en)  20060831 
US20040175041A1 (en)  20040909 
WO2004081853A1 (en)  20040923 
US7643683B2 (en)  20100105 
US7643685B2 (en)  20100105 
US20040190775A1 (en)  20040930 
JP2006520054A (en)  20060831 
US7853085B2 (en)  20101214 
EP1599828A1 (en)  20051130 
EP1599829A1 (en)  20051130 
WO2004081855A1 (en)  20040923 
JP2006522411A (en)  20060928 
US20040175039A1 (en)  20040909 
Similar Documents
Publication  Publication Date  Title 

Decarlo et al.  Optical flow constraints on deformable models with applications to face tracking  
Jourabloo et al.  Largepose face alignment via CNNbased dense 3D model fitting  
Hu et al.  Automatic 3D reconstruction for face recognition  
Grochow et al.  Stylebased inverse kinematics  
US7492927B2 (en)  Facial image processing methods and systems  
Beymer et al.  Image representations for visual learning  
AU734055B2 (en)  Principle component analysis of image/controlpoint location coupling for the automatic location of control points  
EP1550082B1 (en)  Three dimensional face recognition  
US7450126B2 (en)  Methods and systems for animating facial features, and methods and systems for expression transformation  
DeCarlo et al.  An anthropometric face model using variational techniques  
US7483548B2 (en)  Method for detecting object formed of regions from image  
Pighin et al.  Resynthesizing facial animation through 3d modelbased tracking  
US7583271B2 (en)  Method and apparatus for data processing recognizing an object represented as twodimensional image  
US7894636B2 (en)  Apparatus and method for performing facial recognition from arbitrary viewing angles by texturing a 3D model  
CN1871622B (en)  Image collation system and image collation method  
US6434278B1 (en)  Generating threedimensional models of objects defined by twodimensional image data  
Li et al.  Realtime facial animation with onthefly correctives.  
US7254256B2 (en)  Method and computer program product for locating facial features  
US20020041285A1 (en)  Nonlinear morphing of faces and their dynamics  
Jones et al.  Multidimensional morphable models: A framework for representing and matching object classes  
US7646909B2 (en)  Method and system for constructing a 3D representation of a face from a 2D representation  
US6047078A (en)  Method for extracting a threedimensional model using appearancebased constrained structure from motion  
US8259101B2 (en)  Sketchbased design system, apparatus, and method for the construction and modification of threedimensional geometry  
US7321370B2 (en)  Method and apparatus for collating object  
US20160203361A1 (en)  Method and apparatus for estimating body shape 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: ANIMETRICS, INC., NEW HAMPSHIRE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MILLER, MICHAEL I.;REEL/FRAME:025462/0950 Effective date: 20040303 

STCB  Information on status: application discontinuation 
Free format text: ABANDONED  FAILURE TO RESPOND TO AN OFFICE ACTION 