JPH0888850A - Expression reproducing device - Google Patents

Abstract

PURPOSE: To reproduce highly real expressions by synthesizing texture data corresponding to the expression with a wire frame model. CONSTITUTION: This device is provided with an expression texture database 109 for storing the texture data of six basic expressions of a human being and a head part texture switching part 111 for inferring the expression based on an expression change amount extracted by an expression change amount extraction part 103 and selecting and synthesizing textures stored in a non- expression texture database 107 and the expression texture database 109. In this case, a movement parameter based an the change of the expression of a person is detected and the wire frame model is worked based on the detected parameter. Also, the texture to be synthesized is generated based on the texture data corresponding to the recognized basic expression among the stored texture data and the generated texture data are synthesized with the worked wire frame model and displayed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a facial expression reproducing apparatus for reproducing a captured face image on a three-dimensional display in real time by using computer graphics technology, and more particularly to a facial expression reproduction for reproducing the face image by a wire frame model and texture. Regarding the device.

[0002]

2. Description of the Related Art In recent years, development of an immersive communication conference system that enables people in remote places to talk with each other as if they were having a conference at the same place has been under development (for example, incorporated associations, electronic information communications). Published by the Society, Technical Report, HC92-61).

Since subtle movements of facial expressions of participants in an actual conference have an important meaning for communication, it is necessary to reproduce subtle changes in human facial expressions in a realistic communication conference system. The development of facial expression reproduction devices that can reproduce highly realistic facial expressions is expected.

In the conventional facial expression reproducing apparatus, the receiving side holds a three-dimensional wire frame model of the head that approximates the standard form of the head of a person to be transmitted by the vertices and the line connecting the vertices, and the transmitting side The amount of movement of the marker attached to the person's face is tracked by a camera or the like, and the vertex coordinates of the head wireframe model are moved accordingly to reproduce the change in facial expression.

FIG. 5 is a block diagram of a conventional facial expression reproducing apparatus, and FIG. 6 is a diagram showing a specific example of the face image input section of FIG.

Referring to FIG. 5, the conventional facial expression reproducing apparatus includes a facial image input section 101 for inputting a facial image of a person and an facial expression change amount extracting section 10 for extracting the facial expression change amount from the movement of the facial image.
3, a head wireframe model storage unit 105 that stores the above-described head 3D wireframe model, the stored head 3D wireframe model, and the extracted facial expression change amount are combined to create a wireframe model. Wireframe transformation unit 11 that reproduces facial expression changes by transforming
3, an expressionless texture database 107 that stores the expressionless texture data (color information of the face), and an expression face image combining unit 115 that combines the deformed wireframe model and the expressionless texture data. It

The face image input section 101 is specifically constructed as shown in FIG. That is, the face 209 of the person to be transmitted.
A marker 203 is attached to the facial muscle of the person. A camera 201 that captures a face 209 as image data 205 is provided on the helmet 211 fixed to the head.

FIG. 7 is a flow chart for explaining a face image synthesizing process by the facial expression reproducing apparatus of FIG.

In step S 701, the face image including the marker is captured by the face image input unit 101. In step S702, the wire frame model is processed based on the movement amount of the marker. Step S
At 703, the expressionless texture data is combined with the processed wireframe model.

The facial expression is reproduced by the above process.

[0011]

However, in the conventional facial expression reproducing apparatus, since the head wireframe model is deformed based on the detected data and the color texture in the non-expressive state is combined with this, the wrinkles and the skin are removed. It was difficult to reproduce subtle facial expressions such as sagging, and it was not possible to reproduce realistic facial expressions.

Of course, if a head wireframe model having infinite vertex coordinates is used, a more realistic facial expression can be reproduced, but this is not possible in real time.

As a method of expressing wrinkles and sagging, Choi et al. ("Description and synthesis of facial expression based on three-dimensional model of face", IEICE Transactions, Vol. J73-A, No.
7, pp. 1270 to 1280) is disclosed, but this is not applicable to all people because the position where wrinkles are generated must be registered in advance. Moreover, since a part of the texture is cut out, discontinuity may occur in the texture.

In addition to the expressionless head model, a morphing (T. Beier, S. Nee) which stores a head model having an expression in advance and mixes the head model having an expression according to a change in expression.
ly, “Feature Based Image Metamorphosis”, Proc. of
A method called SIGGRAPH92, 1992) is disclosed. However, in morphing, shape deformation is performed while keeping the relative position ratio of some deformation driving points to the deformation driving points of neighboring points constant, and finally the facial expression of the person is a registered facial expression. It will be transformed into a head model. Therefore, the facial expression after deformation is always the same, and it is impossible to reproduce a subtle facial expression.

Therefore, a main object of the present invention is to provide a facial expression reproducing device capable of reproducing a subtle facial expression including wrinkles and sagging.

[0016]

According to another aspect of the present invention, there is provided a facial expression reproducing device for storing a wireframe model storing a wireframe model approximating a human face, and a plurality of basic facial expressions for the human face. Data storing means for storing a plurality of texture data corresponding to, a parameter detecting means for detecting a motion parameter based on a change in facial expression of a person, and a processing means for processing a wireframe model based on the detected parameters. A recognition means for recognizing the facial expression of the person's face as one of the basic facial expressions on the basis of the detected parameters; and synthesizing based on the texture data corresponding to the recognized basic facial expression among the stored texture data. Generating means for generating the desired texture data, and the wire frame processed for the generated texture data. Synthesizing means for synthetic reproduced expression image to the model, in which a display means for displaying the reproduced expression image.

A facial expression reproducing apparatus according to a second aspect is the first aspect.
In the facial expression reproducing device described above, the plurality of texture data includes expressionless texture data and facial expression texture data, and the generation means generates a human facial expression when the basic facial expression recognized by the recognition means corresponds to the facial expression texture data. With the passage of time since the facial expression was recognized,
It is a facial expression reproducing device for synthesizing expressionless texture data and facial expression texture data with different distributions and generating texture data to be synthesized.

[0018]

The facial expression reproducing apparatus according to the first aspect stores a wireframe model approximating a human face and a plurality of texture data corresponding to each of a plurality of basic facial expressions of the human. A motion parameter based on a change in a person's facial expression is detected, and a wireframe model is processed based on the detected parameter. The facial expression of the person is recognized as one of the basic facial expressions based on the detected parameters, and the texture to be synthesized is generated based on the texture data corresponding to the recognized basic facial expression in the stored texture data, and the generated texture is generated. The processed texture data is combined with the processed wireframe model and displayed.

A facial expression reproducing apparatus according to a second aspect is the first aspect.
In addition to the operation of the facial expression reproducing device described, a facial expression of a person is recognized when the basic facial expression recognized by the recognizing means corresponds to the facial expression texture data by storing the expressionless texture data and the facial expression texture data. With the passage of time from the time, the expressionless texture and the expression texture are combined with different distributions, and the combined texture is combined with the processed wireframe model.

[0020]

1 is a block diagram of a facial expression reproducing apparatus according to an embodiment of the present invention.

With reference to FIG. 1, the difference between the actual facial expression reproducing apparatus and the conventional facial expression reproducing apparatus shown in FIG. 5 will be described.

In addition to the conventional facial expression reproducing apparatus, the facial expression reproducing apparatus according to the present embodiment has six basic facial expression textures of "joy,""anger,""sadness,""disgust,""fear," and "surprise." Facial expression texture database 10 for storing data
9 and a head texture switching unit that infers a facial expression based on the facial expression change amount extracted by the facial expression change amount extraction unit 103, and selects and synthesizes the textures stored in the expressionless texture database 107 and the facial expression texture database 109. And 111.

FIG. 2 is a flowchart for explaining the face image synthesizing process performed in the facial expression reproducing apparatus of FIG. 1, and FIG. 3 is a diagram for explaining facial expression inference based on the movement amount of the marker in the flowchart of FIG. , Fig. 4
FIG. 3 is a diagram for explaining synthesis of a non-expression texture and a facial expression texture in the flowchart of FIG. 2.

The facial expression reproducing apparatus according to this embodiment stores a plurality of facial expression textures corresponding to six basic facial expressions in addition to the expressionless texture, infers the facial expression from the movement of the face marker, and selects the texture corresponding to the facial expression. It is characterized in that it is used or a mixture of the expressionless texture and the expressional texture at an arbitrary ratio is used as a texture (this method is called a texture blending method).

Referring to FIG. 2, in step S201, the face image input unit 101 takes in the face image with the marker attached, and the movement of the marker is tracked. In step S202, the facial expression is inferred based on the movement amount of the marker. As shown in FIG. 3, this inference is performed based on the amount of movement of the 13 markers 0 to 12 attached to the face from the position when there is no expression.

For example, as shown in the upper part of the table in FIG. 3, the movement amount (pixel unit) of the fifth marker is 5 <x, −10>.
y and the movement amount of the 6th marker is -5> x, -10>
y and the movement amount of the 11th marker is 0 <x, −10>
y and the movement amount of the 12th marker is 0> x, −10>
When the relationship of y is established, the facial expression is judged to be "joy."

The facial expressions inferred by the movement of the marker in step S203 are stored in the facial expression texture database 109, and are stored in the facial expression texture database 109 for humans of "joy", "anger", "sadness", "disgust", "fear", and "surprise". It is determined whether to be classified into 6 basic facial expressions. If it can be classified, step S204
In step S210, it is determined whether the classified facial expression is identical to the immediately preceding facial expression.
The time t from the start of blending is reset.

In step S205, it is determined whether the time t from the start of blending exceeds the blending-interval (for example, a preset time of about 3 to 4 seconds). If NO, the following will be performed in step S206. A combination of the expressionless texture and the expression texture according to the formula is used as a texture (composite texture) to be combined with the wireframe.

Texture to be synthesized = {(blending inter
val −t) × expressionless texture + t × expression texture}
/ Blending interval By this mixing, the expressionless texture and the expressional texture are synthesized as shown in FIG.
The weighting is the time t from the start of blending.
When is 0, the expressionless texture is 100% and the expressional texture is 0%, while t = blending inter
When val is set, the expression texture is 100% and the expressionless texture is 0%.

In step S207, the vertex coordinates of the wire frame model are moved based on the amount of movement of the marker, and the shape change of the face is reproduced. Step S2
At 08, the composite texture set in the previous process is combined with the deformed wireframe model.

On the other hand, in the case of NO at step S203, the expressionless texture is set as a synthetic texture at step S209.

If YES in step S205, the facial expression texture classified in step S211 is used as a synthetic texture.

By using the facial expression reproducing apparatus according to this embodiment, it is possible to reproduce wrinkles and sagging of the epidermis corresponding to 6 basic facial expressions of a human in real time, and from a state of no expression to a state of expression. The change to can be smoothly performed by changing the weighting of blending with time.

In this embodiment, six basic facial expressions are prepared as the facial expression texture, but the number of facial expressions may be arbitrary.

[0035]

The facial expression reproducing apparatus according to the first aspect of the invention is
By synthesizing the texture data according to the facial expression with the wire frame model, the highly realistic facial expression can be reproduced.

In addition to the effect of the facial expression reproducing apparatus according to the first aspect, the facial expression reproducing apparatus according to the second aspect of the present invention can change the facial expression smoothly and reproduce an image with less unnaturalness. You can

[Brief description of drawings]

FIG. 1 is a block diagram of a facial expression reproduction device that is an embodiment of the present invention.

FIG. 2 is a flowchart for explaining a face image synthesizing process performed in the facial expression reproduction device in FIG.

FIG. 3 is a diagram for explaining facial expression inference based on a movement amount of a marker in the flowchart of FIG.

FIG. 4 is a diagram for explaining synthesis of a non-expression texture and an expression texture in the flowchart of FIG.

FIG. 5 is a block diagram of a conventional facial expression reproduction device.

FIG. 6 is a diagram showing a specific example of the face image input unit of FIG.

7 is a flow chart for explaining a face image synthesizing process by the facial expression reproducing device of FIG. 5. FIG.

[Explanation of symbols]

 101 Face Image Input Unit 103 Facial Expression Change Extraction Unit 105 Head Wireframe Model Storage Unit 107 Expressionless Texture Database 109 Facial Expression Texture Database 111 Head Texture Switching Unit 113 Wireframe Deformation Unit 115 Facial Expression Image Synthesis Unit

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] September 20, 1994

[Procedure Amendment 1]

[Document name to be corrected] Drawing

[Correction target item name] All drawings

[Correction method] Change

[Correction content]

FIG.

[Fig. 2]

[Figure 3]

[Figure 4]

[Figure 6]

[Figure 5]

[Figure 7]

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Jun Otani No. 5 Mihiratani, Seiya Town, Seika-cho, Kyoto Prefecture Seika-cho, Osamu Osamu, Osamu Osamu, 5 Hiratani, ATR Communication Systems Laboratory

Claims (2)

[Claims] 1. A wireframe model storage means for storing a wireframe model approximating a person's face, and a texture data storage means for storing a plurality of texture data corresponding to each of a plurality of basic facial expressions of the person's face. A parameter detecting unit that detects a motion parameter based on a change in facial expression of the person; a processing unit that processes the wireframe model based on the detected parameter; and the person based on the detected parameter. Recognition means for recognizing the facial expression of the person as one of the basic facial expressions, and generating texture data to be synthesized based on the stored texture data corresponding to the texture data corresponding to the recognized basic facial expression. Means for converting the generated texture data into the processed wireframe module. A facial expression reproducing device comprising: a synthesizing unit that synthesizes a reproduced facial expression image with Dell; and a display unit that displays the reproduced facial expression image. 2. The plurality of texture data includes expressionless texture data and expression texture data, and the generation means, when the basic expression recognized by the recognition means corresponds to the expression texture data, The facial expression according to claim 1, wherein the expressionless texture data and the facial expression texture data are combined by changing the distribution with the passage of time after the facial expression of the person is recognized to generate texture data to be combined. Reproduction device.