JP2005317000A - Method for determining set of optimal viewpoint to construct 3d shape of face from 2d image acquired from set of optimal viewpoint - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for determining a set of optimal viewpoints to acquire a 3D shape of a face. <P>SOLUTION: A view-sphere is tessellated with a plurality of viewpoint cells. The face is at an approximate center of the view-sphere. Selected viewpoint cells are discarded. The remaining viewpoint cells are clustered to a predetermined number of viewpoint cells according to a silhouette difference metric. The predetermined number of viewpoint cells are searched for a set of optimal viewpoint cells to construct a 3D model of the face. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates generally to image processing, and more particularly to modeling and recognizing faces according to 3D models and 2D images.

  In computer graphics, it is still a fundamental challenge to synthetically construct a real human head, particularly the face. In the following, when referring to “head” or “face”, the present invention targets the largest portion of the head extending from the chin to the eyebrows and from the ears to the ears. Most prior art methods require considerable manual work by skilled artists, expensive active 3D scanners (Lee et al. "Realistic Modeling for Facial Animations" Proceedings of SIGGRAPH 95, pp. 55-62, August 1995) or accurate High quality texture images can be used as an alternative to simple face geometry (Guenter et al. “Making Faces” Proceedings of SIGGRAPH 98, pp 55-66, July 1998, Lee et al. “Fast Head Modeling” for Animation ”Image and Vision Computing, Vol. 18, No. 4, pp. 355-364, March 2000, Tarini et al.“ Texturing Faces ”Proceedings Graphics Interface 2002, pp. 89-98, May 2002) is required.

  To obtain a 3D model of a human face by active sensing, it is necessary to install an expensive scan. For this reason, several techniques have been developed to restore the 3D shape of a face from a 2D image, or “projection”. Some of these methods are based on a direct method that uses the dense 2D correspondence of the image to obtain the 3D position of the reference point of the face ("Regularized bundle-adjustment to model heads from image" by P. Fua. sequences without calibration data '' International Journal of Computer Vision, 38 (2) pp. 153-171, 2000, F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. Salesin, `` Synthesizing realistic facial expressions from photographs "Proceedings of SIGGRAPH 98, 1998, and" Model-based bundle adjustment with application to face modeling "by Y. Shan, Z. Liu and Z. Zhang, Proceedings of ICCV 01, pp. 644-651, July 2001).

  There is also a method of parametrizing a 3D face model and searching for an optimum parameter that best describes a 2D input image (“Face recognition based on fitting a 3D morphable model” by P. 25 (9), 2003. J. Lee, B. Moghaddam, H. Pfister, and R. Machiraju, "Silhouette-based 3D face shape recovery," Proc. Of Graphics Interface, pp. 21-30, 2003, and B. Moghaddam, J. Lee, H. Pfister and R. Machiraju “Model-based 3D face capture using shape-from-silhouettes” Proc. Of Advanced Modeling of Faces & Gestures, 2003).

  In any case, the viewpoint and the number of 2D input images are important parameters for recognizing a high quality 3D model. Intuitively, the more input images taken from different viewpoints, the higher the quality of the 3D model and subsequent reconstruction. However, this increases processing time and equipment costs.

  However, if the optimal viewpoint set can be determined, fewer cameras can be used, and the resulting 2D images can provide better 3D modeling accuracy.

  So far, a systematic method for obtaining the optimal number of viewpoints and hence input images for the purpose of constructing a 3D model of a face cannot be used. It would also be advantageous to automatically select a specific image, an image selected corresponding to the optimal viewpoint for improving face recognition, from a series of images in the video.

  It is known that different objects have different prototypes or aspect viewpoints (C. M. Cyr and B. B. Kimia, “Object recognition using shape similarity-based aspect graph” Proc. Of ICCV, pp. 254-261, 2001).

  It would be desirable to determine a canonical set of optimal viewpoints for a particular class of objects with particularly high intraclass similarity, particularly human faces.

  When dealing only with lighting, it is possible to empirically determine the optimal configuration of nine point sources that span the general subspace of the face under various lighting (K. Lee, J. Ho, and D. Kriegman, “Nine points of light: Acquiring subspaces for face recognition under variable lighting” Proc. Of CVPR, pp. 519-526, 2001).

  It would be desirable to solve problems related to the subject's pose, or equivalently, the camera viewpoint. That is, it is desirable to obtain the optimal K viewpoint set corresponding to the spatial configuration of K cameras that best describes the 3D human face by projection from the viewpoint, that is, forms a silhouette in the 2D image.

  An important challenge in multi-view 3D modeling is finding the optimal set of viewpoints or “poses” needed to accurately 3D shape-estimate “generic” faces. To date, there is no analytical solution to this problem. On the contrary, partial solutions require an exhaustive combinatorial search.

  Based on a 3D modeling method, the present invention uses a contour-based silhouette matching method and extends the method by actively removing the observation sphere using viewpoint clustering and various other imaging constraints. Multi-view optimization searches are performed using both model-based (eigenhead) and data derivation (view volume) methods, resulting in equally optimal sets of viewpoints.

  The optimal set of viewpoints can be used to obtain the 3D shape of the face and can provide useful empirical guidelines for 3D face recognition systems.

  Since analytical formulation is not possible, the present invention uses an empirical approach. An observation sphere centered on the object is sampled (tessellated) to generate a finite set of viewpoint configurations. Each viewpoint is evaluated according to the resulting ensemble error for a representative data set of individual faces. Collective error relates to mean reconstruction error.

  Due to the large number of potential viewpoints, the observation sphere is actively removed by discarding a predetermined set of irrelevant or impractical viewpoints that may depend on the application. The present invention can use the aspect viewpoint for general 3D object recognition. The aspect viewpoint is a projection of the silhouette of an object from the viewpoint, and represents the range of similar viewpoints in the vicinity of the space of the observation sphere sampled uniformly. However, the present invention is not limited to the use of such aspect fields, since the method will work with any given set of fields that will be prominent.

  The size of the viewpoint space is reduced to an object class, for example a face. After sampling the observation sphere equally and applying high-level model-specific constraints such as facial symmetry and imaging geometry, the method uses a silhouette difference measure, or in contrast, a similarity measure, and Are combined to generate viewpoint clusters, and the original “centroid” of each cluster is selected as the aspect viewpoint. By exploring a reduction in the number of combined aspect viewpoint combinations for a given number of distinct fields of view (cameras), a set of viewpoints optimal for modeling the shape of the object is constructed.

Multi-view 3D Face Modeling The present invention provides a method for determining the optimal set of viewpoints required when building an accurate 3D model of a human face obtained from a 2D image taken from a set of viewpoints. A general method for constructing the 3D model of the present invention is described in US patent application Ser. No. 10 / 636,355 “Reconstructing Heads from 3D Models and 2D Silhouetts” by Lee et al., Filed Aug. 7, 2003, This is incorporated herein by reference.

  As shown in FIG. 6, the method of the present invention uses a number of cameras 600, for example 11 cameras, arranged on a “observation sphere” 200 around a human head 210. The placement of the camera determines the size of the portion of the head that is modeled. In practical applications, the observation sphere is constructed as a geodesic dome with a camera attached to the structural member of the dome. A person sits on a chair in the dome, during which time an image of the person's face is obtained. The face 210 is substantially at the center of the observation sphere 200.

  As described herein, the 3D modeling method of the present invention deals with the restoration of texture, ie appearance independent shape, ie geometric shape. Thus, the method of the present invention is robust with respect to illumination and texture variations. The method of the present invention can be distinguished from the “morphable” model (“Face recognition based on fitting a 3D morphable model” PAMI, 25 (9), 2003) by V. Blanz and T. Vetter as follows: It is.

  The shape is restored directly rather than with texture estimation. The shape is obtained from the occlusion outline or silhouette rather than from the texture. No object texture estimation is required. However, the texture can be easily obtained from the object using standard techniques after the shape is restored. The model approximation of the present invention uses a binary silhouette rather than an image intensity error. The method of the present invention essentially does not require an actual image. The method can use any other means of segmenting depth layer information into foreground / background segments. For example, the silhouette can be obtained using a distance sensor.

  Furthermore, the silhouette matching optimization of the present invention is simpler, has fewer free parameters, and is considerably faster, approximately 10 times faster.

  In the conventional modeling method according to the present invention, the optimal placement of the camera 600 was first found by trial and error and using “intuition” as to which viewpoint is more useful for shape acquisition.

  The present invention continues here.

  Thus, the goal is to eliminate the guesswork from the viewpoint selection process and to find the optimal geometric shape or field of view configuration for a given number K of cameras. For model building, the method of the invention used facial scans of adult women and adult men of various races and ages. Scans can be used to generate meshes. The number of points in each face mesh varies from approximately 50,000 to 100,000.

  All scanned faces in the database are resampled to obtain correspondence between points. Second, the resampled face is aligned with the reference face, and any variation in pose variation or any misalignment during scanning is removed. Third, Principal Component Analysis (PCA) is performed on the registered 3D face database to obtain the eigenvectors of the shape model of the present invention and their associated eigenvalues, ie the variance of each potential Gaussian distribution. It is done. This decomposition can be used to reconstruct a new or existing face through a linear combination of “inherent head” based features (in general, JJ Atick, PA Griffin, and N. Redlich, “Statistical approach to shape from shading face surfaces from single 2D images "Neural Computation, 8 (6) pp. 1321-1340, 1996).

Examination of the PCA eigenvalue spectrum and the resulting shape reconstruction showed that the first 60 eigenheads were sufficient to capture most of the prominent facial features in the database. Therefore, the corresponding shape parameter a _i is the optimization parameter of the present invention.

An arbitrary face model M (a) generates a polygon mesh that gives a parameter vector a = {a ₁ , a ₂ ,..., A _n }. The input silhouette image is S ^k _input , and k = 1 _,. A similarity transformation T aligns the reference model face with a realistic 3D face. The silhouette image S ^k _{model (a)} is rendered by projecting T (M (a)) onto the image plane using the pose information of the kth silhouette image. Parameter vector a is the total penalty

  Is estimated by minimizing. However, the cost function f measures the difference between two binary silhouettes. The cost function f in equation (1) is a simple difference measure between two binary silhouettes is the number of “on” pixels when a pixelwise exclusive OR (XOR) operation is applied. .

  Higher penalty for any mismatches near the boundary pixels of the input silhouette to prioritize the correct pixel matching on the occlusion contour and to promote uniqueness such that the cost function f has a global minimum Imposing. In particular,

  It is. However, D (S) is Euclidean distance conversion of the binary image S, and the image (˜) S is an original image of the image S. Note that the variable d represents a distance map from the silhouette contour. After the preprocessing step, the variance can be determined. This cost function is called boundary weighting XOR. This cost function provides a simple and effective alternative to exact contour matching. In addition, (~) S represents that there is ~ on S.

  Therefore, time consuming processing corresponding to edge linking, curve approximation, and distance calculation between contours is not necessary. Further, the boundary weighting XOR operation can be performed by hardware. If the cost function is inherently complex and non-linear and there is no analytical gradient, then stochastic downhill simplex method is used to minimize equation (1).

Finding the Optimal Viewpoint for 3D Face Modeling From now on, we will explain how to find the optimal set of viewpoints for 3D face modeling using any number K, for example 5 cameras or less, or “viewpoints”. Continue invention. A method for “removing” all possible viewpoint spaces obtained by equal tessellation of the observation sphere based on clustering adjacent viewpoints using silhouette difference measures or silhouette similarity measures obtained from shape projections explain. The selected set of aspect viewpoints is then examined using both the model-based method and the data-derived view volume method of the present invention.

Silhouette Generation The face silhouette resampled in the database of the present invention is very different from the silhouette obtained from the actual subject image. This is due to the lack of part of the head and upper torso. In order to simulate a silhouette image of an actual subject using the database of the present invention, a fully scanned 3D head is used as the prototype head / torso of the present invention.

  FIG. 1A is an image of the original face in the database, FIG. 1B is an image of the resampled face, FIG. 1C is an image of the complete “prototype” head that was laser scanned, and FIG. A rendered face image obtained by combining the resampled face with the scanned full head.

  Coupling is a “virtual” test with a full head and shoulders, aligning the face area of the prototype head with the resampled face by smooth deformation, stitching the head and face together This is done by compositing the subject. Thus, according to the present invention, it is possible to generate a complete silhouette image having exactly the same face shape as the face in the database while maintaining an appropriate geometric shape of the subject.

  This pre-processing step is only used instead of a full head scan and can be omitted if a full subject scan is available. The process of “stitching” the 3D face model into a common head shape highlights only those areas of the face that are critical as the “target region” that is important in subsequent analysis and optimization. By doing so, less important areas, such as the back of the head, effectively indicate that they are not important or inconspicuous with respect to the accurate reconstruction of the face area. However, the search method may remain the same regardless of which area is highlighted, i.e., marked as prominent.

Observation Sphere Tessellation As shown in FIGS. 2A and 5, the observation sphere 200 is equally tessellated with triangles using the dodecahedron section around the subject 210 (510). By this procedure, 120 triangles 201 are generated, which are called viewpoint cells. The vertex 202 of each triangle 201 is on the surface of the observation sphere 200.

  As shown in FIG. 2B and FIG. 5, the selected viewpoint cell is discarded (520). The viewpoint cells to be discarded include cells in the second half of the observation sphere with respect to the camera. This is because the face is shielded from the viewpoint of the second half sphere. Further, the viewpoint cells above and below a predetermined height are discarded. This is because the corresponding viewpoint is unlikely or impractical as the physical position of the camera. In the method of the present invention, the height of the viewpoint is limited to ± 45 degrees from the central horizontal plane.

  Furthermore, in many cases, obtaining an accurate facial contour from an oblique viewpoint is very difficult due to occlusion and consequently indistinguishable from hair and shoulders. Finally, because the face is roughly symmetrical, discard all the remaining viewpoints. This leaves the 44 viewpoints shown in FIG. 2B.

The remaining viewpoints still have too many combinations or subsets of viewpoints. For example, in order to find eleven optimal viewpoints by exhaustive search, it is necessary to evaluate a combination of approximately 7 × 10 ⁹ viewpoints. This is quite difficult to handle. Therefore, it is necessary to further reduce the search space.

Viewpoint Clustering In the present invention, we notice that 2D silhouette images of two adjacent viewpoints are often substantially similar. Accordingly, the silhouette difference between two adjacent viewpoints is measured, and if the silhouette difference is less than a predetermined threshold, two corresponding viewpoint cells are clustered (530).

  Next, the position of the viewpoint cell group (cluster) can be represented by the center of gravity of the cluster of viewpoint cells. More importantly, only the silhouette differences near critical areas of the face, such as the nose, eyes, ears, chin, and mouth are considered here. This is because face shape restoration is not affected by silhouette differences in other unrelated areas such as shoulders.

  To cluster, a look-up table (D) is first built that stores partial or face-limited XOR silhouette distances between every pair of viewpoints in the search space. Initially, every viewpoint in the cluster is considered, and the aspect viewpoint of the cluster is the viewpoint itself.

  The silhouette difference between two clusters is defined by the silhouette distance between each aspect viewpoint. That information is pre-calculated and stored in lookup table D. Find the two neighboring clusters with the smallest silhouette difference among all the other neighboring clusters and join these clusters. After combining the two clusters, a new aspect view of the newly combined cluster is determined. The new aspect viewpoint is the viewpoint that has the smallest value for the maximum silhouette difference compared to all other viewpoints in the same cluster. This process is repeated until a predetermined number of clusters remain.

  FIG. 3 shows 10 clusters 1-10 obtained using the clustering step 530 and roughly corresponding aspect viewpoints 300. Note that the resulting aspect viewpoint is not necessarily the geometric centroid of the cluster, but the viewpoint with the smallest silhouette difference relative to other parts of the cluster.

  In order to avoid any subject dependence and generalize this clustering, all entries in lookup table D average the silhouette difference distances for each pair of 50 different composite heads in the database. Generated by.

  Table A provides the coordinates of the aspect viewpoints 1-10 where the azimuth {90 °, 0 °, + 90 °} corresponds to {left, front, right} in the head center reference frame.

  FIG. 4 shows corresponding silhouettes 401-410 obtained from 10 aspect viewpoints, along with model silhouettes and critical facial areas used for error estimation. All reconstruction errors are limited to critical facial areas. Irrelevant input from hair and shoulders is ignored. Discard field of view 1 in FIG. Due to the downward angle, the corresponding face silhouette 401 is partially hidden and mixed with the torso. The field of view 2 is also preferred because the front viewpoint is preferred for facial texture acquisition, but is discarded because it provides little occlusion contours as shape restoration constraints.

Finding the optimal viewpoint For the remaining 8 aspect viewpoints 3 to 10, an optimal subset of K ≦ 8 viewpoints is searched and searched thoroughly (540). Using the K silhouettes, the 3D shape reconstruction closest to the original face is generated. The default reconstruction method is a model-based (proper head) 3D face shape restoration method according to the present invention described in the related US patent application Ser. No. 10 / 636,355.

  For comparison, a purely data-derived method was also tested using the view volume construction method. Note that the view volume itself is not suitable for accurate reconstruction even when using hundreds of viewpoints. The goal of this specification is to show that a similar optimal set of viewpoints is selected in a greedy search based on a data-derived method.

  In order to make the optimal set of viewpoints valid for general purpose face modeling and recognition, the viewpoints should be adapted to the general faces of all species, eg gender, race, age. Optimality should therefore be independent of the subject. For this reason, a representative subset of 25 people from the database was used and the optimal viewpoint selection of the present invention was based on a relative arrangement that minimized the total or average error of all subjects.

  When restoring a 3D shape from a silhouette image, a measure is needed to measure the error between ground truth and the reconstructed 3D geometry. In the present invention, since the face area of the restored shape is focused, a measure for measuring the difference in the extremely important face area between the restored shape and the original face is required. The basic method of error measurement is as follows.

  The first step is to find a dense set of points on the face area of the restored face geometry. In the case of a proper head shape model, the face points of the model of the present invention are found through mesh parameterization.

  However, finding the same facial point on the view volume is not obvious. Use ray casting to find facial points on the view volume. Since we have an original 3D head image and use it to generate an input silhouette image from the face points on the original head, we project the rays towards the view volume and on the view volume surface Get the corresponding sample.

  After obtaining the face points, the same ray casting scheme is used to obtain the corresponding sample on the ground truth mesh surface. The L2 distance of the face point on the restored face and the corresponding point on the ground truth is measured, and the L2 distance is used as the 3D error measure for the face area.

Model-based reconstruction An exhaustive search is performed on the remaining 8 aspect viewpoints in FIG. I found Therefore, the total number of possible reconstructions is 5450. To remove the data dependency inherent in a single individual reconstruction error, we use the average reconstruction error of 25 subjects randomly selected from the database.

  The results are presented in Table B, which shows the optimal set of viewpoints and the corresponding minimum average reconstruction error when K = {1, 2, 3, 4, 5}. See Table A for the exact coordinates of the aspect viewpoint.

  The standard deviation of the individual errors of all 25 subjects under the best configuration is also presented. The mean error mean and mean error standard deviation are based on the mean reconstruction error across all viewpoints. Since more viewpoints provide more constraints, both have a tendency to decrease as K increases, as expected.

View Volume Reconstruction Using the same search strategy described above for the 3D model based method, we will now evaluate the view volume reconstruction obtained from a given subset of silhouette images and compare the results to ground truth.

  Table C shows the optimal viewpoint for K = {2, 3, 4, 5} and the corresponding error value. The viewing volume from a single silhouette (K = 1) is omitted because it has no finite volume.

  Note that view volume reconstruction, especially from a small number of images, is not a very accurate representation. In contrast to the model-based results, in this case the reconstruction quality is much more field dependent than subject dependent. However, the visual field dependence decreases significantly as the number of viewpoints (K) increases. See error standard deviation. In both cases, viewpoints 3 and 10 appear to be most beneficial.

Effects of the Invention The method according to the invention seeks a set of viewpoints that are optimal for 3D face modeling, in particular a method for restoring a shape from a silhouette. The present invention provides useful guidelines for designing 3D face recognition systems and is consistent with existing implementations and intuitions. For example, the most prominent viewpoint 3 corresponds very closely to the established biometric standard “3/4 field of view (photographs 3/4 from the right of the face)” used in many ID photographs. 10 corresponds to the profile view used for the “Mugshot (for police criminal record)” photo. The results herein show that reconstruction does not substantially improve beyond 4 to 5 viewpoints. See the best errors listed in Table B and Table C.

  Additional physical and computational constraints can be incorporated into the method of the present invention. For example, a front-facing viewpoint is less conspicuous with respect to shape, but is a preferred field of view for capturing texture, and therefore this field of view is used in almost all 2D face recognition systems. This viewpoint can be selected in advance before searching.

  When a face is supplemented on a video basis, a plurality of virtual viewpoints are provided by the movement of the subject and changes in poses even when the camera is fixed. Thus, the method of the present invention can be applied to a series of images of surveillance video to automatically select the best pose, i.e., the best video frame for face recognition.

  Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Accordingly, it is the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the invention.

An image of the original face in the database. It is the image which resampled the face of FIG. 1A. It is a 3D model image of the head obtained by scanning. 1B is an image obtained by combining the resampled image of FIG. 1B with the model of FIG. 1C. A tessellated observation sphere. A tessellated observation sphere with the viewpoint discarded. An observation sphere with the viewpoints clustered. It is a silhouette obtained from 10 aspect viewpoints. Fig. 2 is a block diagram of a method according to the invention. It is a figure of the observation sphere by the present invention.

Claims (21)

A method for obtaining an optimal set of viewpoints for 3D shape construction of a face from a 2D image obtained by an optimal set of viewpoints,
Tessellating an observation sphere having a plurality of viewpoint cells, wherein the face is located approximately in the center of the observation sphere;
Discarding the selected viewpoint cell;
Clustering the remaining viewpoint cells into a predetermined number of viewpoint cells according to a silhouette difference measure determined from the 2D image;
And searching for the predetermined number of viewpoint cells of the optimal viewpoint set. A method for obtaining an optimal viewpoint set for 3D shape construction of a face from a 2D image. The method of claim 1, wherein triangles are used for tessellation. The method of claim 1, wherein the tessellation is a dodecahedron equal division. The method according to claim 1, wherein the selected viewpoint cell includes a viewpoint cell in a second half sphere of the observation sphere. The method according to claim 1, wherein the selected viewpoint cells include viewpoint cells that are above and below a predetermined height. The method according to claim 5, wherein the predetermined height is ± 45 degrees from a central horizontal plane of the observation sphere. The method according to claim 1, wherein the selected viewpoint cells include viewpoint cells in one half of the face since both halves of the face are approximately symmetrical. The method of claim 1, wherein the silhouette difference measure is measured for each viewpoint cell pair. The method of claim 1, wherein the predetermined number of viewpoint cells is ten. The method according to claim 1, wherein the position of the viewpoint cell is obtained by a center of gravity of a cluster of the viewpoint cell. The method according to claim 1, wherein only the silhouette difference near a very important face area is considered. The method of claim 1, wherein the critical facial areas include the nose, eyes, ears, chin, and mouth. The method of claim 1, wherein the silhouette difference is stored in a pre-calculated look-up table. The method of claim 1, wherein the search is done thoroughly. The method of claim 1, wherein the optimal set of viewpoint cells is applied to a series of images of a video to automatically select an optimal pose for facial modeling. The method of claim 1, wherein the optimal set of viewpoint cells is applied to a series of images of a video to automatically select an optimal pose for face recognition. The method of claim 1, wherein the optimal set of viewpoint cells is used to construct a 3D model of the face from 2D images obtained with the optimal set of viewpoints. The method of claim 1, wherein the search is a combinatorial search of all possible subsets of the optimal set of viewpoint cells. The method of claim 1, wherein optimality is determined by a minimum reconstruction error between the input face and the 3D model of the face. The method of claim 19, wherein the minimum reconstruction error is based on a combination of shape and texture. The method of claim 19, wherein the minimum reconstruction error is based on an average of a plurality of faces.

Images