JPH0812706B2 - Gaze animation display device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a field of animation display of a person or the like using computer graphics (hereinafter referred to as "CG"). The present invention relates to a technology for giving a feeling of being in the three-dimensional space, for example, a realistic communication conference technology.

[0002]

2. Description of the Related Art For the purpose of image communication and human interface, research is being conducted to express the facial expression of a person by CG. For example, "Journal of the Institute of Electronics, Information and Communication Engineers J
73-D-II, 3, 1990, pp. 351-3.
On page 59, a paper entitled "A Study on Facial Expression Synthesis for Intelligent Interfaces" is published. In this paper, a method of expressing a person's facial expression by CG is introduced as one of image combining methods for the purpose of image communication and human interface.

[0003]

However, the technique introduced in the above-mentioned paper is a method of synthesizing a face image using text information, emotional information, etc., and is applied as it is to a realistic communication conference system. There is a difficulty in doing so.
For example, in the case of a conversation or a meeting, the eye movement line of sight,
Blinking and the like are important information indicating the intentions of the participants. If such information is animated and displayed in real time as a CG image, a more natural presence can be obtained in the presence-oriented communication conference system. However, no proposal has yet been made on how to realize such technology.

Therefore, it is an object of the present invention to provide a visual line animation display device capable of displaying the movement of the eyes closer to a natural movement and displaying the movement in real time.

[0005]

According to a first aspect of the present invention, there is provided a visual line animation display device for detecting a visual line of a person to obtain coordinates of a gazing point of the person within a predetermined coordinate system, and blinking of the human figure. Means for obtaining information in real time, and a face model of a person is combined with an eyeball model approximated by a predetermined model and displayed by computer graphics, and a predetermined position determined by the coordinates of the gazing point is displayed. And an animation display unit for rotating the eyeball model so as to gaze. The animation display means determines a rotation amount of the eyeball model based on the coordinates of the gazing point, a means for displaying the animation synthesized by rotating the eyeball and the face model of the person, and the means for determining on the face model of the person. And means for displaying the eyelids and / or eyebrows in a manner determined by a predetermined relationship with respect to the amount of rotation that has been performed. The means for displaying the animation and the means for displaying the eyelids and / or the eyebrows, the eyeball model and the eyelid The display of the eyebrows or both are changed to blink.

The line-of-sight animation display device according to a second aspect is the device according to the first aspect, wherein smoothing processing is sequentially performed on the coordinates of the gaze point obtained in time series, and the gaze point is displayed on the animation display means. And means for providing as the coordinates of the.

[0007]

[0008]

[0009]

In the line-of-sight animation display device according to claim 1, when the coordinates of the gazing point of the person are obtained, an animation is made so that the eyeball model gazes at a predetermined position defined by the coordinates of the gazing point accordingly. It is rotated in real time by the display means. Since the eyelids and / or the eyebrows are displayed on the face model of the person displayed together with the eyeball model in a manner that changes with the rotation of the eyeball model, the face model has a more natural facial expression. In addition, depending on the blink information of the person, the eyeball model and the eyelids and / or the eyebrows change in a blinking manner and are displayed in real time, so the displayed face model is more natural and The state of is reproduced more faithfully.

In the eye-gaze animation display device according to the second aspect, the smoothing process is sequentially performed on the coordinates of the gazing point obtained in time series, and the eyeball model of the eyeball model is created based on the coordinates of the gazing point obtained by the smoothing process. Since the animation of rotation is displayed, the movement of the eyeball model becomes more natural.

[0011]

[0012]

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A visual line animation display device according to an embodiment of the present invention will be described below with reference to the drawings. In the following embodiments, a line-of-sight detection device (Japanese Patent Publication No. 3-51407) made by the same applicant as the present applicant is shown as the line-of-sight detection device. This line-of-sight detection device uses pupil reflected light by infrared rays. However, the present invention is not limited to this, and other methods such as EOG method, scleral reflection method, and magnetic induction method can be used.

Referring to FIG. 1, the visual axis animation display device of the present invention is a pupil image / face feature point detection unit for detecting a pupil reflection image by infrared rays and a face feature point of a person 52 as a subject by image processing. 34, the pupil image, the pupil reflection image detected by the face feature point detection unit 34, and the face feature points, and information about the posture and position of the face of the person 52 and information necessary for eyeball display such as the line of sight and blinks. The CG information acquisition unit 50 for generation, the CG information acquisition unit 50, and the CG image generation apparatus 48 for generating a CG image according to the information obtained from the CG information acquisition unit 50, which are connected by the communication line 44.
And a display device 46.

The pupil image / face feature point detecting section 34 includes a pair of cameras 56 for obtaining image information for detecting the pupil reflection image and face feature points by infrared rays, and a predetermined image for this image information. An image processing unit (not shown) that performs processing to obtain a pupil reflection image and facial feature points is included.

The CG information acquisition unit 50 detects the position and orientation of the head from the pupil image and the head feature points detected by the face feature point detection unit 34, and the pupil image, Based on the number of pixels of the pupil reflected light detected by the facial feature point detection unit 34, a blink determination unit 38 for determining whether or not the eyes are closed, and the eyes are opened by the blink determination unit 38. When it is determined that the person 52 is detected based on the pupil image, the pupil position detected by the facial feature point detection unit 34, and the head position detected by the head position detection unit 36.
A gazing point detection unit 40 for detecting the coordinates of the gazing point of
A smoothing unit 42 for performing a smoothing process on the data detected by the gazing point detection unit 40 is included. The information obtained by the CG information acquisition unit 50 is given to the CG image generating device 48 via the communication line 44, and the CG image generated by the CG image generating device 48 is displayed by the display device 46.

The flow of information is arranged as shown in FIG. First, the pupil image / face feature point detection unit 34 detects the pupil reflection image and face feature points by image processing. The head position detection unit 36 detects the position and orientation of the head from the pupil image and the head feature points detected by the face feature point detection unit 34.
The blink determination unit 38 obtains the number of pixels of the pupil image and the pupil reflected light detected by the facial feature point detection unit 34, and determines that the eyes are closed if the number of pixels is equal to or less than a preset value. If it is equal to or greater than the set value, it is determined that the eyes are open. Blink determination unit 38
When it is determined that the eyes are opened by, from the pupil image, the pupil position detected by the facial feature point detection unit 34 and the head position detected by the head position detection unit 36,
The gazing point detector 40 detects the coordinates of the gazing point. Further, these coordinates are smoothed by the smoothing unit 42 and given to the CG image generating device 48 and displayed by the display device 50. When the blink determination unit 38 determines that the eyes are closed, the processing by the gazing point detection unit 40 and the smoothing unit 42 is not performed.

Referring again to FIG. 1, a method of detecting the coordinates of the gazing point by the pupil image / face feature point detecting section 34, the head position detecting section 36, and the gazing point detecting section 40 will be described. This method uses the method described in Japanese Patent Publication No. 3-51407. Note that in this method, the gazing point V on the plane 54 separated from the person 52 by the distance d
The (x, y) coordinate of is detected.

Referring to FIG. 3, first, a desired coordinate system (X, Y, Z) is defined in space. Coordinates of three feature points on the face of the person 52 in the coordinate system (X, Y, Z) shown in FIG. 3 are obtained by the pair of cameras 56 shown in FIG.
These characteristic points are designated as P1, P2, and P3 shown in FIG.

Then, as a head coordinate system, a triangle (P
1, P2, the center of gravity P _G of P3) as the origin, y-axis and the direction perpendicular to the direction of a line connecting the P1 and P2 x axis, the x-axis,
Consider a coordinate system (x, y, z) having the z-axis as the normal direction of the triangle.

In FIG. 3, a vector having the center of gravity P _G as the starting point and the eyeball center 62 as the end point (eyeball center position vector) e
Is a constant vector. The vector e can be obtained in advance by calibration. Center of gravity P _G is 3 points P
Since it can be obtained from 1, P2, and P3, the coordinates of the eyeball center 62 in the coordinate system (X, Y, Z) can also be obtained. On the other hand, the coordinates in the coordinate system (X, Y, Z) of the pupil center can be directly obtained from the camera image.
The half line connecting the center of the eyeball and the center of the pupil is the line of sight. Considering this line of sight as a vector, and obtaining the intersection of the plane 54 and this vector, the plane 5
The coordinates on 4 are obtained. This is the process performed by the pupil image / face feature point detection unit 34, the head position detection unit 36, and the gazing point detection unit 40.

Human eye movements include involuntary movements such as jumping movements. Therefore, the gazing point detection unit 40
If the data detected by the above method is directly used for displaying a CG image, it gives an unnatural feeling. In order to make the CG image more natural, the smoothing unit 42 performs a smoothing process on the detected data. As the smoothing method, a method of using the average value of a predetermined number of past sampling data as output data is used.

The detection data of the gazing point at time t is (x
_t , y _t ). When using the average value of three points as the output data, the output (x _ot , y _ot ) is calculated by the following formula.

[0024]

[Equation 1]

FIG. 4 shows the detection data of the gazing point when the line of sight is moved left and right, and the smoothed data. The smoothing data 84 in which the average value of three points is output data is smoother than the graph of the detection data 82, and when the line of sight of the eyeball of the CG image is changed based on this, A more natural feeling is given as compared with the case where the detection data 82 is used. It should be noted that although three points of data are used for smoothing here, the number of samples used for smoothing is not limited to three.

Referring to FIG. 5, the smoothing section 4 of FIG.
1 shows a CG person image whose line of sight is changed according to the output of FIG.
It is generated by the CG image generating device 48 shown in FIG. Here, as a CG model of a face, a model in which an eyeball approximated by a sphere model is combined with a face model created based on three-dimensional data of an actual human face is used. Of course, an eyeball approximated by an ellipsoid may be combined.

Referring to FIG. 5, the relationship between the eyeball model used here, the gazing point and the line of sight is as follows.
A straight line connecting the eyeball center 104, the pupil center 106, and the gazing point 110 is defined as a line of sight 108. With respect to the change of the gazing point 110, the eyeball 102 is rotated so that the direction of the line of sight 108 matches the gazing point 110. In this way, as the gazing point information for rotating the eyeball model, the smoothing unit 4
By generating the CG image using the output of 2, the change in the line of sight of the person can be displayed in real time and in a natural form as an animation.

By the way, in the expression of an actual person,
The eyelids and eyebrows change as the eye moves. In order to display animations naturally, it is necessary to express such movements of the eyelids and eyebrows. FIG. 6 is a schematic diagram for explaining movements of eyelids and eyebrows due to changes in the line of sight. FIG. 6 schematically shows the positions of the eyelids 122 and the eyebrows 124 when the eyeball 102 faces front.

With reference to FIG. 7, consider a case where the line of sight is directed upward and the eyeball 102 is rotated upward. With the rotation of the eyeball 102, the eyelid 122 is also lifted up. The eyebrows 124 also go up.

On the other hand, as shown in FIG. 8, when the line of sight becomes downward and the eyeball 102 rotates downward, the eyelid 122
Also goes down. In this case, it is more natural that the eyebrows 124 are not lowered.

The display of the eyelids and eyebrows shown in FIGS. 6 to 8 is performed as follows. Referring to FIG. 9, the center of the eyeball 102 is the eyeball center 104. An angle formed by a line of sight 108 connecting the eyeball center 104 and the pupil center 106 with a straight line connecting the eyeball center 104 and the front direction is the eyeball 1
The rotation angle θ is 02.

Referring to FIG. 10, it is assumed that the CG image of eyelid 122 is composed of vertices 132a, 132b, ..., 132n. These vertices 132a to 132n
Is the amount of change that changes with the rotation of the eyeball 102
And The relationship between the rotation angle θ shown in FIG. 9 and the movement amount ξ of each vertex shown in FIG. 10 is obtained in advance, and the vertices 132a to 132n are displayed at the positions of ξ determined according to the rotation angle θ.

FIG. 11 is a graph showing an example of the relationship between the movement amount ξ and θ. Θ when facing the front is 0. Let θ _{+ be} the maximum rotation angle of the eyeball upward and θ _{− be} the maximum rotation angle of the eyeball downward. Let ξ _{+ be} the maximum value of the amount of change in the eyelids due to the upward rotation of the eyeball, and ξ _{− be} the maximum value of the amount of change due to the downward rotation. As shown in FIG. 11, in the θ-ξ coordinate system, a point corresponding to the maximum value (θ ₊ , ξ ₊ ) and a point corresponding to the minimum value (θ ₋ ,
xi] _-) to be united to each origin, determined between the points corresponding to the minimum and the point corresponding to the maximum value by polygonal line obtained thereby, the variation xi] for the angle theta. It should be noted that the interpolation is not limited to the linear interpolation as described above, and non-linear interpolation may be performed. The point is to define a relationship between θ and ξ so that the change of the eyelid can be obtained in a natural form. You can leave it.

Further, when the blink determination section 38 determines that the person as the subject is blinking, the eyelids of the CG image are changed to blinks to hide the display of the eyeball model as if the CG image was hidden. By displaying the person as if he or she blinked, it is possible to perform a more realistic display.

As described above, in the visual line animation display device according to this embodiment, if the coordinates of the gazing point are obtained,
It is possible to animate the rotation of the eyeball and the accompanying changes in the eyelids and eyebrows. In an immersive teleconferencing system, an eyeball model that can change the line of sight in real time can be displayed on the participant's face model, and further changes in the eyelids and eyebrows that accompany rotation of the eyeball and blinking of a person are animated in real time. Because you can
The realism of the meeting is improved.

[0036]

As described above, according to the invention described in claim 1, the eyeball model is rotated by the animation display means so as to gaze at a predetermined position according to the coordinates of the gaze point of the obtained person. Therefore, it becomes possible to reproduce the facial expression of a person in a more realistic manner in real time. Since the eyelids, the eyebrows, or both are displayed in a manner that changes with the rotation of the eyeball, it is possible to display the line of sight of a more natural person in real time. Furthermore, depending on the information about the blink of the person, the eyelids and / or eyebrows of the eyeball model or both of them change in a blinking manner and are displayed in real time, so the displayed face model is more natural and The state of is reproduced more faithfully in real time.

In the line-of-sight animation display device according to the second aspect, the animation of the rotation of the eyeball model is displayed based on the coordinates of the gazing point obtained by the smoothing process. As a result, the movement of the eyeball becomes smoother and more natural. As a result, the facial expression of a person can be displayed as an animation in a real-time manner in real time.

[0038]

[0039]

That is, according to the first and second aspects of the present invention, it is possible to provide a line-of-sight animation display device capable of displaying the movement of the eyes of the CG image of the person in a more natural manner in real time.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a line-of-sight animation display device according to an embodiment of the present invention.

FIG. 2 is a diagram schematically showing the configuration and operation of a visual line animation display device according to an embodiment of the present invention.

FIG. 3 is a schematic diagram for explaining a method of obtaining coordinates of a gazing point.

FIG. 4 is a graph showing detected data and data after smoothing.

FIG. 5 is a schematic diagram showing a method of displaying an eyeball model corresponding to a gazing point 110.

FIG. 6 is a schematic diagram showing the vicinity of the eyes of a person who is facing the front.

FIG. 7 is a schematic diagram showing the vicinity of the eyes of an upward facing person.

FIG. 8 is a schematic diagram showing the vicinity of a downward facing person's eyes.

FIG. 9 is a side view of an eyeball model.

FIG. 10 is a front view of an eyelid.

FIG. 11 is a graph showing the relationship between the rotation angle of the eyeball and the amount of change in the eyelids.

[Explanation of symbols]

 34 Pupil image, face feature point detection unit 36 Head position detection unit 38 Blink determination unit 40 Gaze point detection unit 42 Smoothing unit 44 Communication line 46 Display device 48 CG image generation device 50 CG information acquisition unit

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor, Fumio Kishino, No. 5, Mihiratani, Seiji-cho, Seika-cho, Soraku-gun, Kyoto Pref., ITR Communication Systems Laboratories, Inc. (56) References Information Processing Society of Japan, Research Report, HI-42, pp. 11-16, 1992, Fumio Kishino et al., "Three-dimensional animation display of eye and eye-gaze matching" IEICE Technical Report, IE-90, pp. 37-44, 1991, Hidetomo Sakaino et al. , "A Study on Coordination between Head and Eye Movements on Large Screen"

Claims (2)

[Claims] 1. A means for obtaining the coordinates of the gazing point of the person within a predetermined coordinate system by detecting the line of sight of the person, a means for obtaining information on the blink of the person in real time, and a face model of the person. In, to display by computer graphics by synthesizing an eyeball model approximated by a predetermined model, and for rotating the eyeball model so that the eyeball model gazes at a predetermined position defined by the coordinates of the point of gaze. An animation display means, wherein the animation display means determines an amount of rotation of the eyeball model based on the coordinates of the gazing point, rotates the eyeball, and displays an animation combined with the face model of the person. And a mode determined on the face model of the person by a predetermined relationship with respect to the determined rotation amount, A means for displaying a lid and / or eyebrows, or both, wherein the means for displaying the animation and the means for displaying the eyelids and / or eyebrows obtain the information on the blink of the person in real time. An eye gaze animation display device that changes the display of the eyeball model and the eyelid or eyebrow or both in a blinking manner in accordance with the blinking information obtained in real time by the means for. 2. The method according to claim 1, further comprising means for sequentially performing smoothing processing on the coordinates of the gazing point obtained in time series and giving the animation display means as the coordinates of the gazing point. Gaze animation display device.