JP2986441B2 - Generating 3D face models with arbitrary facial expressions - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention
Regarding the method of generating a three-dimensional face model, in particular, it can be used for video conferencing systems, contents such as virtual reality and VRML, etc., and the restored three-dimensional Euclidean structure is not completely accurate but is not necessarily required for an application that does not necessarily require an accurate expression. The present invention relates to a method for generating a three-dimensional face model having an effective arbitrary expression.

[0002]

2. Description of the Related Art Yasuhiro Mukaikawa, Yuichi Nakamura, Yuichi Ota, etc., for generating a face image of any direction and any facial expression from a small number of real images, "Face images of any direction and any facial expression by combining multiple face images" , "IEICE Transactions (D-II), Vol. J80-D-II, No. 6,
pp. 1555-1562, 1997 (Reference 1) or D. Beymer, A. Shashua, and T. Poggio, “Example
Based Image Analysis and Synthesis, ”AI Memo N
o. 1431, MIT AI Lab., 1993 (Reference 2).

[0003] In addition, D. Terzopoulos a
nd K. Waters, “Analysis and synthesis of facial i
magesequences using physical and anatomical model
s, ”IEEE Transactions on Pattern Analysis and Mach
ine Intelligence, VOL.15, No.6, pp. 569-579, June
1993 (Reference 3) or Y. Lee, D. Terzopoulos, and K.
Waters, “Realisticmodeling for facial animatio
n, ”In Computer Graphics, Annual ConferenceSeries,
pp. 55-62. ACM SIGGRAHP, 1995 (Reference 4).

Further, an approach for restoring a three-dimensional model from two real images having the same person and the same facial expression but different face directions is described in Z. Zhang, K. Isono, and S. Akamatsu, “Eucl.
idean structure from uncalibrated images using fuz
zy domain knowledge: Application to facial images
systhesis, ”In Proccedings of the InternationalCo
nference on Computer Vision (ICCV '98), pp. 784-7
89 India, January 1998 (Reference 5).

[0005]

The above-mentioned references 1 and 2
The end purpose of the method is image generation, not reconstruction of the three-dimensional Euclidean structure. However, virtual reality and the like often require not only an image but also a three-dimensional model. In Reference 1, a reference image is used for designating a face direction. For this reason, when images in various directions are desired, it is necessary to prepare a plurality of reference images corresponding to the images.

On the other hand, in the methods of References 3 and 4, arbitrary facial expressions are generated using a three-dimensional physical model. However, in this method, a basic three-dimensional physical model is required, and in order to obtain them, a special measuring device using a laser beam or the like is required. Is required. Furthermore, there is a problem that the adjustment of the facial expression parameter is delicate and difficult, it is difficult to make the face image natural, and the amount of calculation becomes enormous.

[0007] Therefore, a main object of the present invention is to provide a generation method for generating a three-dimensional model having a more natural arbitrary expression from a small number of real images without constructing a physical model or the like.

[0008]

SUMMARY OF THE INVENTION The present invention relates to a generation method for generating a three-dimensional Euclidean expression having an arbitrary expression from two input images having the same person and the same expression but different face directions by using knowledge about the expression. There, the first expression and the second
First, a feature point is designated on each image from a facial expression sample composed of a pair of facial expressions, and a facial expression difference vector is obtained by subtracting the characteristic point coordinates of the first facial expression from the characteristic point coordinates of the second facial expression. Step, a second step of generating a pseudo-expression image and a pseudo-expression Euclidean structure by adding the expression difference vector to the input image and the expression three-dimensional Euclidean structure restored from the input image, and features on the expression three-dimensional Euclidean structure The constraint that the coordinates of the point projected on the image are equal to the coordinates of the feature point on the pseudo-expression image obtained using the expression difference vector, and the expression 3
A third step of formulating a constraint condition that the feature point coordinates of the two- dimensional Euclidean structure are equal to the pseudo-expression Euclidean structure by the square of the Mahalanobis distance, and a three-dimensional expression using the nonlinear least squares method using these constraint conditions And a fourth step of generating a shape.

[0009]

FIG. 1 is a block diagram showing an entire configuration of an embodiment of the present invention, FIG. 2 is a block diagram of an expression difference data extracting unit in FIG. 1, and FIG. 3 is an input actual image. FIG. 4 is an image showing feature points.

Next, an embodiment of the present invention will be described with reference to FIGS. In FIG. 1, an input image 1 includes two uncorrected real images having the same person (person P) and the same facial expression (facial expression A) as shown in FIG. Is done. At this time, an image of another expression (expression B) of the person P is not given.

The feature point extracting unit 10 extracts feature points of an input image using an interactive processing function of a general-purpose graphics workstation. That is, a feature point is visually detected from the displayed input image, and the position of the feature point is specified using a cursor that moves on the surface of the input image by a mouse operation. This time, as shown in FIG. 4, 83 feature points were used.

The three-dimensional Euclidean structure restoring unit 20 for the expression A restores the Euclidean structure of the expression A by approximating the camera model with an affine camera, using the method described in the aforementioned reference 5. Considering a change in facial expression from an expression A (usually a neutral expression) to an expression B (smile, etc.), assuming that a facial feature point is given, the facial expression change is based on the characteristic point of the facial expression A and its characteristic point. The three-dimensional coordinates of the feature points of the corresponding facial expression B and the three-dimensional difference vector and the two-dimensional difference vector can be given as a difference vector in the image coordinates to be obtained. Since the image of the expression B of the person P is not given, the difference vector when the expression of the person P changes can not be directly obtained from the input image. Therefore, an expression face sample is prepared, and a difference vector is obtained using the expression face sample.

Next, the operation of acquiring difference data will be described with reference to FIG. The face sample 301 is composed of a pair of facial expressions A and B for a large number of people, and the faces are all facing forward. The feature point extracting unit 302 extracts feature points of the face sample using the interactive processing function of the general-purpose graphics workstation. That is, a feature point is visually detected from the displayed input image, and the position of the feature point is specified using a cursor that moves on the surface of the input image by a mouse operation. At this time, a point corresponding to the feature point on the input image is designated as the feature point.

The difference vector calculation unit 303 calculates a difference vector between the expression A and the expression B in each pair of face samples, and uses two types of difference vectors, a difference vector in space and a difference vector on the image.

As the three-dimensional difference vector, a pair of three-dimensional data of the expression A and the expression B of a plurality of persons is prepared, and the expression A and the expression B are prepared.
The difference vector at each feature point is calculated. Similarly, as the image difference vector, a difference vector between the expression A image and the expression B image is obtained from a plurality of image samples composed of a pair of the front images of the expression A and the expression B.

The expression difference data calculation unit 304 determines that the above-mentioned difference vector follows a Gaussian distribution and obtains a three-dimensional average difference vector δM _i and a covariance matrix Λ _Mi. Similarly, the image average difference vectors .delta.m _i and covariance matrix lambda _mi
Ask for.

Although the three-dimensional range data is used this time as the three-dimensional data, there is no problem if the data is obtained from different data. It is assumed that the three-dimensional data and the image data used in the sample have been normalized with respect to the size and the orientation.

The pseudo-expression B data generation processing unit 40 shown in FIG.
Generates pseudo-expression data using the input image and the expression difference vector. Pseudo image image of the average difference vector .delta.m _i the person P above the person P expression B (expression A)
Can be obtained by adding Δm described above
_{Since i} is obtained using the front image, the image of the person P also needs to be the front image.

When one front image of two input images is given as one of the input images, the given front image may be used, but if the front image is given as the input image, Otherwise, a front image can be obtained from the two input images using the Euclidean structure restored using the method of Reference 5. Further, since .delta.m _i is the average of a number of image samples, data are averaged, there is a possibility that expression change will come out relatively small.
Furthermore, it is necessary to consider the difference between the size of the image sample and the size of the front image of the person A. Therefore, it is assumed that addition of the weight omega _e regarding expression strength .delta.m _i, the value obtained by multiplying the weight omega _s of the size of the image in the front image. Here, ω _e is arbitrarily determined, and ω _s is a distance d between a straight line connecting the outer corners of the eyes of the face sample image and the midpoint of both ends of the mouth.
＾, _assuming that the distance between the straight line connecting the corners of both eyes of the front image of the person P and the midpoint of both ends of the mouth is d, ω _s = d / d ＾. Assuming that m ^V _i is the feature point coordinates on the pseudo image and m ^A _i is the feature point coordinates of the front image (expression A) of the person P, m ^V
_i can be represented by the following equation.

[0020] ^{_{m V i = m A i +}} ω e ω s δm i, ... (1) using the coordinates of the feature points on the obtained pseudo-image by the above for generating a pseudo-image in the following manner. First, the input image is divided into triangular patches having feature points as vertices. Next, each feature point is moved to the position determined by the equation (1), and the texture of the input image is mapped to the deformed triangular patch. Thereby, a pseudo-expression B image of the person P can be obtained.

[0021] In the same manner, using the Euclidean structure M ^A _i and the three-dimensional facial expression difference vector .DELTA.M _i of the person P obtained using the technique of Document 5, pseudo Nihyo information B Euclidean structure M
^V _i can be obtained.

[0022] In ^{_{M V i = M A i +}} ω E e ω E s δM i ... (2) where, ω _{^E e,} ω ^E _s is a weighting factor related to the size of the expression intensity and the face in the three-dimensional structure, respectively is there.

It is also possible to directly use a facial expression B image and facial expression B three-dimensional structure of another person without generating pseudo facial expression data. However, when data of another person is used, the generation result becomes unnatural. This is probably because the shape of the face varies from person to person, and the influence of the shape appears in the generated image. Therefore, in this case, the pseudo-expression data is generated using the average difference data to reduce the influence of the personal shape. Thus, a three-dimensional Euclidean structure having a more natural expression can be restored.

Expression B In the three-dimensional Euclidean structure generation processing unit 50, the projection on the three-dimensional structure image must be equal to the image from the geometric relationship between the image and the three-dimensional structure. However, the pseudo image obtained by the above method is
Using difference data obtained from the face image sample,
Since the pseudo three-dimensional structure uses difference data obtained from three-dimensional range data, these data are independent and the geometric relationship is incorrect. Therefore, in order to satisfy the geometric relationship between the image and the three-dimensional structure, the following constraint condition is given.

(1) Expression B The coordinates of the point at which the characteristic points of the three-dimensional Euclidean structure are projected on the image are equal to the coordinates of the characteristic points on the pseudo-expression image.

[0026] (2) feature point coordinates expressions B3 dimensional Euclidean structure is equal to the pseudo-expression B Euclidean structure.

The feature point of the pseudo image is m^V _i, Of those points
Let the coordinates in Euclidean space be M ^B _iThen, if
Assume that the pseudo image represents a correct expression image of the person P.
Then, M^B _iProjection coordinates m ′ on the image plane_i= GM^B _i+ T is m^V _i(Covariance matrix Α_mi)
You. This constraint can be expressed as the square of the Mahalanobis distance.
Can be.

[0028] ^{_{d 2 m = (m V i}} -m 'i) T Λ -1 mi (m V i -m' i) ... (3) pseudo-expression B Euclidean structure feature points M ^V _i of obtaining expression B The coordinates of the feature points of the three-dimensional Euclidean structure are expressed as M ^B _i
And When the pseudo expression B Euclidean structure is to represent the correct expression model of the person P, M ^B _i is M
^It should be equal to ^v _i (covariance matrix Α _Mi ). This constraint condition can be expressed by the square of the Mahalanobis distance.

[0029] ^{_{d 2 M = (M V i}} -M B i) T Λ -1 Mi (M V i -M B i) ... (4) using the methods described above, the pseudo-image as a pseudo expression B data
Generate an image and a pseudo-expression Euclidean structure to obtain d ² _m
And so that the sum of d ² _M is minimized by nonlinear least squares method, obtaining the three-dimensional Euclidean coordinates M ^B _i of the feature point in the person P · expression B.

J = Σd ² _m + Σd ² _M → min. ... (5) generates a triangular patch from the obtained M ^B _i, by mapping the texture obtained from the basic image, it is possible to obtain the expression B 3-dimensional Euclidean structure of the person P · expression B.

FIG. 5 is a view showing an example of a facial expression image obtained by one embodiment of the present invention. In FIG. 5, the weighting factor for the expression intensity is ω _e = ω ^E _s = 1.5. Since the texture inside the mouth at the time of opening cannot be obtained from the basic image, the texture obtained from any happiness (opening) image in the image sample was used.

[0032]

As described above, according to the present invention, two input images having the same person and the same facial expression but different face directions are provided.
In a method of generating a three-dimensional structure of a face having an arbitrary expression from expression data, the expression data is obtained by statistical analysis from a face sample image and three-dimensional face sample data, and a pseudo-expression image and a three-dimensional image are obtained from the obtained expression data. Since a pseudo-facial expression structure was generated, and a constraint obtained from the geometrical relationship between the image and the three-dimensional structure was given with a minimum Mahalanobis distance, the three-dimensional expression structure was generated by a nonlinear least squares method.
Given two images, a three-dimensional shape having an arbitrary expression can be restored from the two images. Moreover, once the three-dimensional shape can be restored, an image in any direction can be easily created. In addition, multiple reference images and 3
There is no need to use a three-dimensional physical model, and the present invention can be applied to applications that require a three-dimensional model such as virtual reality. Further, by generating various facial expressions, a facial expression animation or the like can be easily created.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of an embodiment of the present invention.

FIG. 2 is a specific block diagram of an expression difference data extracting unit shown in FIG.

FIG. 3 is a diagram illustrating an example of an input image.

FIG. 4 is a diagram showing feature points in an input image.

FIG. 5 is a diagram showing an example of an image generated according to an embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 1 input image 2 facial expression B three-dimensional Euclidean structure 10 feature point extraction unit 20 facial expression A three-dimensional Euclidean structure restoration processing unit 30 facial expression difference data extraction unit 40 pseudo facial expression B data generation processing unit 50 facial expression B three-dimensional Euclidean structure generation processing unit 301 face sample 302 Feature point extraction unit 303 Difference vector calculation unit 304 Expression difference data

────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Shigeru Akamatsu 5th Sanraya, Daiya, Seika-cho, Soraku-gun, Kyoto Pref. 62865 (JP, A) JP-A-8-63503 (JP, A) Kouji Komatsu, "Estimation of three-dimensional shape of face from face image and change of facial expression", Transactions of IEICE (D- ▲ II ▼), May 1990, Vol. 73, No. 5, p. 707 -716 (58) Field surveyed (Int.Cl. ⁶ , DB name) G06T 1/00 G06T 7/00-7/60 G06T 15/00 G06T 17/00-17/50

Claims (1)

(57) [Claims] 1. A generation method for generating a three-dimensional Euclidean expression structure having an arbitrary expression from two input images of the same person and the same expression but having different face directions by using knowledge about the expression, the first expression comprising: A feature point is designated on each image from a facial expression sample composed of a pair of a facial expression and a second facial expression, and a facial expression difference vector is obtained by subtracting the characteristic point coordinate of the first facial expression from the characteristic point coordinate of the second facial expression. A first step, a second step of generating a pseudo-expression image and a pseudo-expression Euclidean structure by adding the expression difference vector to the input image and the expression three-dimensional Euclidean structure restored from the input image, the second step; The constraint that the coordinates of the point where the above feature point is projected on the image is equal to the coordinates of the feature point on the pseudo-expression image obtained using the expression difference vector A third step of formulating the condition and the constraint that the feature point coordinates of the expression three-dimensional Euclidean structure are equal to the pseudo-expression Euclidean structure obtained using the expression difference vector by the square of the Mahalanobis distance; and A method for generating a three-dimensional face model having an arbitrary expression, comprising a fourth step of generating an expression three-dimensional Euclidean structure using a nonlinear least squares method.