JP2003141563A - Method of generating face 3d computer graphics, its program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method of generating face 3D CG reflecting a subject's appearance information and character information available from the subject's behavior. SOLUTION: The face characteristic points required to identify an individual are extracted from photographic information including images of the subject's head taken from two directions, front and side. Based on the characteristic points, the 3D structures of the parts of the face, including the head skeleton, nose, mouth, eyebrows, and eyes, and the face parts are integrated together to restore a face 3D configuration. The face 3D configuration is compared with average face data to emphasize the characteristic parts of the subject's face. The displacements of the characteristic points of the face are calculated on the basis of image information resulting from continued imaging of the subject's head. Based on the displacements of the characteristic points, the character of the subject is classified. The 3D configuration of each part of the face is varied depending on the classification.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention makes it possible to express the face of a character (character) in a virtual space or the like by three-dimensional computer graphics based on the appearance information and personality classification of the person. The present invention relates to a three-dimensional computer graphics generation method, a program therefor, and a recording medium recording the program.

[0002]

2. Description of the Related Art In recent years, users of a plurality of computers connected to a network have been constructed on the network.
Research and development of a multi-user system that participates in and communicates with a three-dimensional virtual space has been actively performed, and is used for remote education, remote conference, three-dimensional chat community, and the like. In this virtual space, the user is represented as his or her own altercation called a character, and moves in the virtual space while changing the position and direction of the alternation according to the viewpoint movement of the user with respect to the virtual space, and communicates with other users. I do. In such a virtual space system, it is important to represent a character, which is a user's alter ego, and in particular, an expression method for a face portion, which is a portion where individual characteristics are most reflected, is drawing attention.

Conventionally, "3D Me Now" (http: ///www.bio
virtual.com/2001.08.01) and "Face surface model for animation production" (Komatsu, "Face surface model for animation production", Information Processing Society of Japan, Vol.
30, No. 5, pp. 633-641, 1989), there is a method of generating a three-dimensional character having a shape similar to an actual head by using a captured face image (method 1). ). "Face 3D model for facial expression description" (Cui, Harajima, Takebe, "Facial expression description and synthesis based on 3D facial model", IEICE Transactions, V
OL. J73-A, NO.7, pp.1270-1280,
(1990), there is a face three-dimensional model representation method that synthesizes various facial expressions using a wireframe model (method 2). In addition, "Multi-person communication system on cyberspace" (Omuro, Ito, Shimada, Morishima, "Real-time synthesis of lip image synchronized with speech using lip feature extraction and speech analysis", electronic information Proceedings of the IEICE General Conference, A-15-11, pp.310, 2001)
As described above, there is a lip shape expression method that uses both images and sounds (method 3).

[0004]

However, in the above method 1, since the details are to be expressed, the three-dimensional shape model becomes complicated, and it is necessary to extract a large number of facial feature points from the image. Therefore, there is a problem that the calculation process required for the automatic generation increases, and the user's own labor for the operation increases.

Further, in the above method 2, since the processed image is pasted on the photographed image, it is suitable for restoration faithful to the actual head shape, but it is an abstract expression in which a characteristic face portion is emphasized. Not suitable for

Further, in the above method 3, since motion tracking in real time is the main subject, it is limited to reproduction of appearance information, and personal information, which is an important element in expressing the personality of a person, is not available. Not reflected.

The present invention has been made in view of the problems of the above-mentioned conventional methods, and a face three-dimensional computer graphics generation method that reflects the appearance information of the subject and the character obtained from the subject's gesture, and a program therefor. , And a recording medium having the program recorded therein.

[0008]

According to the present invention, a process of restoring a three-dimensional shape of a face based on appearance information of a subject, a personality of the subject is classified, and the personality of the classified subject is described above. The main feature is that the subject is roughly classified into a process of reflecting the three-dimensional shape of the face restored by appearance information.

In one embodiment of the present invention, in the process of restoring the three-dimensional shape of the face based on the appearance information of the subject, the individual's head is photographed from two directions, the front and the side, The two-dimensional arrangement information of face feature points necessary for identification is extracted, and the three-dimensional arrangement information of the face feature points is calculated based on the two-dimensional arrangement information of the face feature points to obtain the three-dimensional arrangement information of the face feature points. Based on this, the 3D structure of each face part such as the head skeleton, nose, mouth, eyebrows, and eyes is restored with a 3D geometrical figure expressed by a surface consisting of curves, straight lines, and each line, and these face parts are integrated. By doing so, the three-dimensional shape of the face is restored. Furthermore, in order to further emphasize the characteristic face parts of the target person, the placement location, scale, shape, etc. of each face part are compared with the average face data, and the three-dimensional shape of each face part is changed according to the difference. Let

Further, in the processing of reflecting the character of the target person in the three-dimensional shape of the face, the displacement amount of the two-dimensional arrangement information of the facial feature points is calculated from the image information obtained by continuously photographing the head of the target person from the front direction. Then, the personality of the target person is classified by collating a personality classification table or the like based on the displacement amount of the two-dimensional arrangement information of the facial feature points, and the three-dimensional shape of each face part is changed according to the classification. .

As described above, according to the present invention, the personality information, which is the most important factor in expressing the personality of a person, is not restored based on only the appearance information, but is reflected in the three-dimensional shape representation of the face. This is the difference from the conventional method.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 shows a configuration example of a face three-dimensional computer graphics (CG) generation system applied to one embodiment of the present invention. In the present configuration example, the user terminal device 10 and one or a plurality of photographing devices 20 are connected directly or via a network 30.

As shown in FIG. 2, the user terminal device 10 includes a CPU 11, a storage device 12 such as a hard disk, a keyboard 13, a mouse 14, and a photographing equipment interface 1.
5, a personal computer equipped with a display 16, etc.,
If necessary, it is connected to the photographing device 20 or the network 30 via the photographing device interface 15. CPU11
Controls each unit and executes the face three-dimensional CG generation processing of the present invention. The storage device 12 holds a program executed by the CPU 11, average face feature point arrangement information described later, a personality classification table, and the like. In addition, the storage device 12 is
It is also used to store the captured image information by the imaging device 20, the generated face three-dimensional shape information, and the like. The user terminal device 10 may be a camera-installed multimedia personal computer in which the photographing device 20 is built.

First, referring to FIGS. 1 and 2, an outline of operation in the face three-dimensional CG generation processing according to the present invention will be described. In the face three-dimensional CG generation process of the present invention, as shown in FIG. 3, the face three-dimensional shape is restored by the appearance information of the subject (process 310).
And the face parts having the three-dimensional shape of the restored face are roughly classified into emphasis deformation (process 320) depending on the character of the subject.

<Process 310> The user operates the keyboard 13
When a shooting request is input using the mouse 14 or the mouse 14, the user terminal device 10 (CPU 11) transmits the shooting request of the user to the shooting device 20 via the shooting device interface 15. When the imaging device 20 receives the imaging request from the user's terminal device 10, the imaging device 20 images the subject's head from the front and side surfaces and transmits the captured image information to the user terminal device 10. The user terminal device 10 receives the photographed image information from the photographing device 20 via the photographing device interface 15, and stores it in the storage device 12.

Next, the user device 10 (CPU 11)
From the photographed image information (appearance information of the target person) stored in the storage device 12, layout information of face feature points necessary for individual identification is extracted, and first, based on the layout information of the face feature points, a three-dimensional face is extracted. Restore the shape. Next, the user terminal device 10 preliminarily stores the extracted facial feature point arrangement information in the storage device 1.
The average facial feature point arrangement information stored in 2 is compared, and the three-dimensional shape of the face is transformed according to the difference so that the characteristic facial parts of the subject are more emphasized.

Up to this point, the face 3 according to the appearance information of the target person
The restoration of the dimensional shape is completed. The user terminal device 10 stores the restored face three-dimensional shape information in the storage device 12 and also displays it on the display 16 as necessary.

<Process 320> When the user wants to reflect the character of the target person in the three-dimensional shape of the face, the user inputs a continuous photographing request using the keyboard 13 or the mouse 14. The user terminal device 10 transmits a continuous shooting request of the user to the photographing device 20 via the photographing device interface 15. In addition,
The user terminal 10 may automatically transmit a continuous shooting request to the shooting device 20 after the operation of the process 310 is completed.

When the photographing device 20 receives the continuous photographing request from the user terminal device 10, the photographing device 20 repeats photographing the head of the subject and transmitting the photographed image information to the user terminal device 10 for a certain period of time. The user terminal device 10 is the photographing device 2
The photographed image information from 0 is received via the photographing device interface 15, sequentially stored in the storage device 12, and similarly, the extraction of the facial feature point arrangement information is repeated for a certain period of time.

Next, the user terminal device 10 determines the facial feature points based on the facial feature arrangement information extracted from the photographed image information (video information) obtained by continuously photographing the subject's head for the fixed time. Measure the amount of displacement. Then, the user terminal device 1
0 indicates the displacement amount of this facial feature point in advance in the storage device 1
The personality (behavioral personality) of the target person is classified by collating with the personality classification table held in No. 2, and the facial parts of the face three-dimensional shape obtained in the previous processing 310 are transformed according to the personality. To do.

The user terminal device 10 draws the three-dimensional shape of the face thus restored in three-dimensional CG and outputs it to the display 16. The user terminal device 10 is a three-dimensional C
When G is drawn and output to the display 16, if there is an edit request from the user, the keyboard 1 operated by the user
3 and the three-dimensional face shape changed according to the input from the mouse 14 are redrawn.

FIG. 4 shows a detailed flowchart of an embodiment of the face three-dimensional CG generation processing according to the present invention. In FIG. 4, steps 401 to 406 are a process of restoring the three-dimensional shape of the face based on the appearance information of the target person (process 310 in FIG. 3), and steps 407 to 412 are the personality classification (behavioral personality classification) of the target person.
3 is a process of deforming the three-dimensional shape of the face (process 320 in FIG. 3). An embodiment of the face three-dimensional CG generation processing according to the present invention will be specifically described below with reference to FIG.

<Step 401> User terminal device 10
Sends a shooting request to the shooting device 20, and the shooting device 20 receives the shooting request and sets the head of the subject to the front and side.
Shoot from the direction. FIG. 5 shows an example in which a side face is photographed from the side direction of the target person 40 by the photographing device 20,
FIG. 6 shows an example in which the front face of the target person 1002 is photographed by the photographing device 20 from the front direction. Here, when the number of the photographing devices 20 is one, the photographing is performed separately while moving the target or the photographing device, but when the number of the photographing devices 20 is two, as shown in FIG. You may install in each direction and shoot at the same time.

<Step 402> User terminal device 10
Receives the photographed image information of the side face and front face of the user 40 photographed by the photographing device 20, temporarily stores it in the storage device 12, and automatically extracts the facial feature points from the photographed image information.
The facial feature points are extracted as the two-dimensional arrangement information of the facial feature points existing on the outer contours of the side face and the front face and the edges of each face component area. This is done as follows, for example.

By the area division processing by the color information and the edge extraction processing using the differential filter, the face outlines of the side face and the front face and the edge information of the face parts are automatically extracted, respectively, and as shown in FIG. Fixed point groups (eyebrows,
Eyes, top of nose, bottom of nose, top of upper lip, center of lip, top of lower lip, center of chin, tip of chin, etc.) as well as fixed point groups (eyebrows, eyes, top of nose, bottom of nose, Fixed point groups (upper and lower left and right end points of the eyebrow and eye areas, left and right end points of the nose and mouth area, etc.) that exist on the top of the upper lip, the center of the lip, the top of the lower lip, the center of the chin, the tip of the chin, and the edges of each face part area. ) Is extracted as two-dimensional arrangement information of facial feature points. In FIG. 8, the face feature points 801 extracted for the side face and the front face are indicated by square marks.

Based on the extracted two-dimensional arrangement information of the facial feature points 801, the three-dimensional information of the facial feature points is calculated.
That is, the extracted face feature points 801 are converted into a three-dimensional coordinate system by front-side matching (stereo matching), and face three-dimensional feature points such as eye positions, vertical and horizontal widths, and nose heights are calculated.

<Step 403> The three-dimensional structures of the head skeleton, eyebrows, eyes, nose, mouth and the like are restored based on the three-dimensional arrangement information of the facial feature points. This is done as follows, for example.

The three-dimensional structure of the head skeleton is defined as an aggregate of horizontally sliced parts of the entire head. The slice position is determined by the three-dimensional arrangement information of the facial feature points, and the contour shape at each slice position is drawn by a straight line 901 passing through the facial feature point and a curve 902, as shown in FIG. All the contour shapes of these sliced portions are integrated, and the figure 903 that complements the surface formed by the straight line and the curved line is made a three-dimensional structure of the head skeleton.

The three-dimensional structure of the eyebrows is defined by a contour shape consisting of straight lines and curved lines, as shown in FIG. 10, based on the three-dimensional arrangement information of the facial feature points that form the eyebrow parts, and the straight lines and curved lines form. The figure 1001 that complements the surface has a three-dimensional structure of the eyebrows.

The three-dimensional structure of the eye is defined by the contour shape consisting of straight lines and curved lines as shown in FIG. 11 based on the three-dimensional arrangement information of the facial feature points forming the eye parts, and the surface formed by the straight lines and curved lines. The figure 1101 that complements the above is defined as the three-dimensional structure of the eye.

The three-dimensional structure of the nose defines a contour shape consisting of straight lines and curved lines as shown in FIG. 12 based on the three-dimensional arrangement information of the facial feature points forming the nose parts, and the surface formed by the straight lines and curved lines. The figure 1201 that complements the above is defined as the three-dimensional structure of the nose.

As for the three-dimensional structure of the mouth, as shown in FIG. 13, a contour shape consisting of straight lines and curved lines is defined based on the three-dimensional arrangement information of the facial feature points forming the mouth parts, and the straight lines and curved lines form. A figure 1301 that complements the surface is a three-dimensional structure of the mouth.

<Step 404> Each face part (head skeleton,
3D shape of the face is restored by integrating (eyebrows, eyes, nose, mouth, etc.). FIG. 14 shows an example of restoration of the three-dimensional face 1401.

<Steps 405 and 406> Emphasized expression (exaggerated expression) is performed on the restored three-dimensional shape of the face in order to make the individuality of the subject more remarkable. This is done, for example, as follows.

An average face feature point 1501 obtained by averaging the two-dimensional arrangement information of the face feature points of a plurality of targets as shown in FIG. 15 prepared in the storage device 12 and the face of the target person as shown in FIG. Based on the comparison with the feature points 801, the three-dimensional shape of each face part is changed to more prominently express the three-dimensional facial features of the subject. Specifically, the average face feature point arrangement information a is compared with the target person's feature point arrangement information b to obtain the difference (ba), and the product of this difference and an arbitrarily set exaggeration rate is corrected. The face part is enlarged (when the correction amount is positive) or reduced (when the correction amount is negative) by this correction amount.

For example, the nose is emphasized by enlarging or reducing the scale of the nose. In FIG. 16, the nose 1
Nose 601, enlarged nose 1602 and reduced nose 16
An emphasis example of 03 is shown.

<Steps 407, 408, 409> The user terminal device 10 transmits a continuous photographing request to the photographing device 20, and the photographing device 20 photographs the subject in the front direction of the head and the user of the photographed image information. The transmission to the terminal device 10 is repeated for a fixed time. The user terminal device 10 sequentially receives the photographed image information from the photographing device 20, and similarly repeats the automatic extraction of the facial feature points for a certain period of time.
The fixed time is set to a time in which the character can be estimated depending on the behavior of the target person, for example, about 1 minute to several minutes.

<Step 410> The displacement amount of the facial feature point obtained from the continuous photographing information (video information) of continuously photographing the head of the subject is extracted. For example, the amount of displacement of the motion of the eyebrows, eyes, mouth, etc. is extracted.

<Steps 411, 412> The displacements of the extracted facial feature points, that is, the displacements of the respective motions of the eyebrows, eyes, mouth, etc. are prepared in the storage device 12 as shown in FIG. The personality (behavioral personality) of the target person is classified by comparing with the personality classification table 1700, and 3 of each face part is classified.
Change the dimensional shape.

For example, if the widths of the upper and lower end points of the eyes fluctuate greatly, it is determined that the number of blinks is large and the character is impatient. If the width does not fluctuate much, it is determined that the character is non-slippery and the vertical size of the eyes is increased. In addition, when the inner characteristic points of the left and right end points of the eye often fall and the outer characteristic points rise, it is determined that the character is irritable, and the shape of the eye is lifted outward and deformed. Figure 1
8 shows an eye 1801 before being emphasized by the personality classification, an eye 1802 that is raised to the outside and an eye 1801 that is raised to the inside.
An emphasis example of 803 will be shown.

As described above, the classification of the personality and the degree thereof are determined by the degree of displacement of the facial feature points obtained from the continuously photographed image information of the subject, and the degree of shape deformation of the face part corresponding thereto is determined. To do.

Although one embodiment of the present invention has been described above, some or all of the processing functions of the CPU in the user terminal device shown in FIG. 2 are configured by a computer program, and the program is used by the computer. Needless to say, the present invention can be implemented by executing the program, or the processing procedure shown in FIGS. 3 and 4 can be configured by a program of a computer (CPU) and the program can be executed by the computer. ,
A computer-readable recording medium, such as an FD (floppy (registered trademark)), stores a program for realizing the processing function of the computer or a program for causing the computer to execute the processing procedure.
Disk), MO, ROM, memory card, CD, D
The program can be recorded in a VD, a removable disk, etc., and saved or provided, and the program can be distributed through a network such as the Internet.

[0043]

According to the present invention, there is an effect that a face three-dimensional CG reflecting the appearance information of the subject and the personality information obtained from the motion can be easily and automatically generated. Also,
Face 3D CG that emphasizes the individuality of the target person by emphasizing the facial parts based on information obtained from facial modeling and gestures
The effect is that the creation of can be realized.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an entire system to which the present invention is applicable.

FIG. 2 is a diagram showing a configuration example of a user terminal device.

FIG. 3 is an overall process flow diagram of the present invention.

FIG. 4 is a detailed processing flowchart of an embodiment of the present invention.

FIG. 5 is a diagram showing an example of photographing a side face.

FIG. 6 is a diagram illustrating an example of capturing a front face.

FIG. 7 is a diagram showing an example of simultaneous photographing of a side face and a side face.

FIG. 8 is a diagram showing an example of extraction of facial feature points.

FIG. 9 is a diagram showing an example of restoring a three-dimensional structure of a head skeleton.

FIG. 10 is a diagram showing an example of restoring a three-dimensional structure of eyebrows.

FIG. 11 is a diagram showing an example of restoring the three-dimensional structure of an eye.

FIG. 12 is a diagram showing an example of restoring the three-dimensional structure of the nose.

FIG. 13 is a diagram showing an example of restoring a three-dimensional structure of a mouth.

FIG. 14 is a diagram showing an example of restoration of a three-dimensional shape of a face.

FIG. 15 is a diagram showing an example of arranging facial feature points of an average face.

[Fig. 16] Fig. 16 is a diagram illustrating an example of emphasized expression of a nose.

FIG. 17 is a diagram showing an example of the correlation between the amount of displacement of the facial feature points 7 and the personality classification and shape deformation based on the personality classification table.

[Fig. 18] Fig. 18 is a diagram illustrating an example of eye enhancement based on personality classification.

[Explanation of symbols]

10 user terminal 20 Photography equipment 30 network 400 people 801 facial feature points 903 head skeleton 1001 eyebrows 1101th eye 1201 nose 1301 mouth 1401 three-dimensional face shape 1501 Average facial feature points 1601 Nose model before emphasis 1602 Enlarged nose model 1603 Reduced nose model 1801 Eye model before emphasis by personality classification 1802 Eye model lifted to the outside 1803 Eye model raised inside

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B050 AA10 BA07 BA08 BA09 BA12                       CA07 DA01 EA06 EA19 EA24                       EA28 FA02 FA08 FA17                 5B057 AA20 BA02 CA12 CA16 CB13                       CB17 CE15 DA08 DA17 DB02                       DC05 DC36                 5L096 CA02 FA09 GA08 HA04

Claims (7)

[Claims] 1. A face 3D computer graphics generating method for expressing a face of a subject by 3D computer graphics, a process of restoring a 3D shape of a face based on appearance information of the subject, and a personality of the subject. And a process of reflecting the classified personality in the face three-dimensional shape, and a face three-dimensional computer graphic generation method. 2. The face 3D computer graphics generation method according to claim 1, wherein in the process of restoring the 3D face shape based on the appearance information of the subject, the head of the subject is in two directions, the front and the side. Facial feature points are extracted from the captured image information captured from, and the three-dimensional structure of each facial part such as the head skeleton, eyebrows, eyes, nose, mouth, etc. is restored based on the facial feature points, and each facial part is extracted. A three-dimensional face computer graphics generation method, characterized in that the three-dimensional structure of the above is integrated to restore the three-dimensional shape of the face. 3. The face 3D computer graphics generation method according to claim 2, wherein in the process of restoring the 3D face shape based on the appearance information of the subject, the face feature points of the subject and the face features of the average face are included. A face three-dimensional computer graphics generation method characterized by comparing points with each other and modifying the face part structure of the restored face three-dimensional shape according to the difference. 4. The facial three-dimensional computer graphics generation method according to claim 1, wherein in the processing of reflecting the personality of the subject on the face three-dimensional shape, the head of the subject is photographed. The amount of displacement of the facial feature points is extracted from the continued image information, the personality of the target person is classified based on the amount of displacement of the facial feature points, and the face part having a three-dimensional face is formed according to the classified personality. A method for generating three-dimensional computer graphics of a face, characterized by giving a change to a structure. 5. The face three-dimensional computer graphics generation method according to claim 4, wherein in the process of reflecting the character of the target person in the three-dimensional shape of the face, the eyebrows are extracted from the image information obtained by continuously photographing the head of the target person. , The amount of movement of each face part such as eyes, mouth, etc. is extracted, by pattern matching according to the amount of movement of the face part, the action character of the target person is classified, and based on the action character, the face A face three-dimensional computer graphics generation method characterized by changing each face part structure of a three-dimensional shape. 6. A program for causing a computer to execute the face three-dimensional computer graphics generation method according to claim 1. 7. A recording medium recording a program for causing a computer to execute the face three-dimensional computer graphics generating method according to claim 1.