JP2534617B2 - Real-time recognition and synthesis method of human image - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for real-time recognition and synthesis of a human image, and in particular, it comprises a human model on the transmitting side which is color texture mapped to a three-dimensional wire frame model, and the human model is constructed by computer graphics. Placed in a virtual space created using technology,
Multiple people in different places can reproduce the movement of a person on the receiving side's 3D display and display it in 3D.
The present invention relates to a real-time recognition and synthesis method of a human image, which is applied to communication such that a person can hold a conference as if he / she were in a hall.

[0002]

2. Description of the Related Art Conventionally, the detection and recognition methods of human facial expressions and movements, and the synthesis (reproduction) of movements in human models have mostly been studied separately.

For the detection of facial expressions and movements of people, attempts have been made to recognize facial expressions and movements by capturing a scene including a person as an image with a television camera and processing the image, but either method has been used. Also had a problem in processing accuracy, and the processing time was far from the real-time processing.

[0004]

On the other hand, the synthesis of motion in a human model has been mainly dealt with in the field of computer graphics. That is, usually, a person's movement is photographed with a video camera, several key frames are selected from the continuous images, and based on this, the designer appropriately transforms the person model to synthesize the movement, For the frames between the frames, smooth motion was synthesized by performing interpolation as appropriate. However, when such a method is used, only limited motions prepared in advance can be combined. Furthermore, it is impossible to reproduce the motion in the human model by using the information from the human motion detection unit.

On the other hand, it has been attempted to attach an Exo-skeleton type displacement detection device to the face and reproduce the facial expression in the human figure generated by computer graphics, but it is necessary to wear a large device, There was a big problem in usability.

Therefore, the main object of the present invention is to combine a human motion detection unit and a motion synthesis unit in a human model, and further to realize real-time recognition of a human image so that the human motion can be reproduced in real time in the human model. It is to provide a synthetic method.

[0007]

According to the present invention, a model of a person on the transmitting side is constructed by color texture mapping to a three-dimensional wire frame model, and this person model is generated in a virtual space using computer graphics technology. In this communication method, in which a person's movement is reproduced and displayed stereoscopically on the receiving side stereoscopic display, a marker is pasted on the skin surface above the facial expression muscle of the transmitting side person, and the face image of this person is displayed. The image is captured and the markers in the image are tracked in real time, and the corneal reflection image when the cornea of the person is irradiated with light is captured to detect the line of sight and blink, and the body, head, finger, etc. of the person. Motion information is detected in real time.

In the three-dimensional human model, the wire frame model corresponding to the face is driven based on the tracking result of the marker, and the black eye of the eyeball of the three-dimensional human model is detected based on the detection result of the line of sight and blink. It is configured to open and close the position and the eyelid and drive the corresponding parts of the three-dimensional human model according to the detected motion information.

[0009]

According to the real-time method for recognizing a human image according to the present invention, a marker is attached to the facial expression muscle of a person, the marker and the corneal reflection image are tracked in real time, and movements of the body, head, fingers, etc. Is also detected in real time, and texture mapping is performed in real time in the three-dimensional human model, so that the motion can be reproduced in real time in the three-dimensional human model.

[0010]

1 is a block diagram showing the overall construction of an embodiment of the present invention. In this embodiment, a person image three-dimensional model generation unit 20 for generating a three-dimensional model of the person 2 in the real space 1 and a real-time detection unit 3 for detecting the movement of the person.
0, and a real-time generation unit 40 that generates a human image based on the detected motion information. The person image three-dimensional model generation unit 20
For each part 21 of the human body, the shape input device 22 acquires the three-dimensional wireframe model 23, and the image input device 24 acquires the color texture 25. The wire frame model 23 is obtained by approximating the surface of the human body with triangular patches, and the color texture is color information of the surface of the human body.

During the communication conference, the movement of the person is detected by the real-time detecting section 30. The person's movement real-time detection unit 30 includes a facial expression detection unit 31, a head rotational movement detection unit 32, a finger movement detection unit 33, and a body rotational movement detection unit 34, each of which is a human body. The motion information of the part is detected. The person image generation unit 40 includes the three-dimensional wire frame generated by the person image three-dimensional model generation unit 20 and the motion information of each part of the human body detected by the motion detection unit 30 by the three-dimensional wire frame model transformation unit 41. The three-dimensional wire frame model is deformed based on the model 23 and the shape change due to the movement of the human body is reproduced. Then, in the texture mapping unit 42, the transformation unit 41
The color texture 25 is attached to the triangular patch included in the wire frame model deformed in (3). In this way, the generated three-dimensional human model 3 is placed in the virtual space 44 generated by using the computer graphics technology based on the virtual space three-dimensional data 43. By displaying the virtual space 44 as the virtual space 4 including the person image 3 on the stereoscopic display 5 on the receiving side,
The conference participant 6 on the receiving side can hold the conference with the person 2 on the transmitting side as if he / she were in one place.

The structure of each part shown in FIG. 1 will be described in detail below. FIG. 2 is a human image three-dimensional model generation unit 2 of FIG.
It is a figure which shows the operation principle of 0. The human body of the person 2 includes parts 21a (head), 21b (upper body), 21c (upper arm), 21
It is divided into d (lower arm) and 21e (hand), and shape data and color data are acquired. Although FIG. 2 shows only the upper half of the body, the lower half of the body is similarly processed. In order to obtain the shape data, for example, the laser range scanner 2
6 is used. As such a laser range scanner, a Cyberware Color 3D Dig is available.
Itizer or the like is used. The laser range scanner 26 irradiates the object with laser stripe light while rotating around the object, and observes the deformation of the laser stripe on the surface of the object to acquire a three-dimensional point data set. At the same time, the color information (color texture) of the surface of the object can be acquired. For example, considering the part 21c, the three-dimensional wire frame model 23c and its color texture 25c are obtained based on the three-dimensional data set obtained by the laser range scanner 26.

However, the laser range scanner 26 is effective for parts close to a rotating body such as the upper arm 21c, but the TV camera 27 is used for parts that cannot be called a rotating body such as a hand including fingers. The color texture 25e is obtained by using the three-dimensional point data 23e by manual operation.

As described above, the three-dimensional wire frame model 23 and the color texture 25 of each human body part are obtained. These parts are connected in such a way that the movement of the joint between them can be reproduced.

On the other hand, in the motion detecting section 30, the facial expression detecting section 31 uses image processing as described later. For the head and body rotation movement detection units 32 and 34, for example, Polhemus magnetic sensors are used. This magnetic sensor is capable of real-time detection of a total of six posture parameters, namely movement in the coordinate axis direction in three-dimensional space and rotation around the coordinate axis. In the finger movement detection unit 33, for example, V
PL data gloves are used. This glove can detect the bending degree of a finger in real time by an optical fiber.

FIG. 3 is a diagram for explaining the principle of the facial expression detecting section 31. As shown in FIG. 3, a color marker 51 is attached to the skin surface above the facial muscles of the face 50, the television camera 52 captures this as an image 55, and the color marker 51 is tracked in the image 55. The image 55 is binarized so that only the portion corresponding to the color marker 51 can be detected. In order to stabilize the tracking of the color marker 51, the person having the face 50 is the helmet 5 as shown in FIG.
Suffer 3. A frame 54 is fixed to the helmet 53, and the television camera 52 is fixed to the frame 54. With this configuration, even if the face 50 changes its direction, the face image 55 is always obtained from a fixed position with respect to the face 50. In order to speed up the tracking of the color marker 51, the face image 5
5, the window 5 for each color marker 51
6 is set, and the center of gravity of the pixel considered to correspond to the color marker 51 is obtained in this window 56 and set as the position of the color marker 51.

The facial expression detection unit 31 not only deforms the surface of the skin but also detects the line of sight. That is, a light bulb 57 as a light source is arranged next to the television camera 52 provided in the frame 54 shown in FIG. 3, and the television camera 52 captures a corneal reflection image 58 of the light bulb produced when the light bulb 57 is turned on. Then, the obtained image 55 is tracked in the same manner as the color marker 51. Here, by providing a camera 59 different from the television camera 52 exclusively for detecting the corneal reflection image 58, it is possible to aim at higher detection accuracy. As a result, the direction of the line of sight is detected,
Blinking is also detected depending on whether or not the corneal reflection image 58 is observed.

In the person image generating section 40 shown in FIG. 1, the deforming section 41 deforms the three-dimensional wire frame model 23 of the person image based on the motion information detected by the detecting section 30. First, a method of transforming the wire frame model of the face will be described. Originally, the face 50 has a three-dimensional structure, whereas the movement of the color marker 51 is detected as a two-dimensional movement in the image 55 acquired by the television camera 52, so that the three-dimensional model is driven. Requires knowledge and constraints. Here, using the marker of the nose as an immovable reference point, the distance to each marker when there is no expression is obtained in advance, and based on the movement information of the marker in which the distance change has occurred in accordance with the expression change, for example, as follows,
The three-dimensional model 23 is driven.

FIG. 4 is a diagram showing the wire frame driving method when the movement of the marker under the lower lip is small, and FIG. 5 is a diagram showing the wire frame driving method when the movement of the marker under the lower lip is large. FIG. 6 is a diagram showing a wire frame driving method when the markers at both ends of the lip move outward, and FIG. 7 is a diagram showing a wire frame driving method when the markers at both ends of the lip move inward. .

First, the lower jaw is driven by the movement of the marker on the lower lip. When the movement of the marker is small, it is determined that the mouth is opened small, as shown in FIG. 4, and in this case, on the cylindrical surface with the straight line connecting the lower jaw and the upper jaw on both sides of the face as the central axis. According to the knowledge that the lower jaw moves, the part corresponding to the lower jaw region including the lips is driven. When moving beyond a certain threshold, it is determined that the mouth is wide open, and the lower jaw region is moved vertically downward, as shown in FIG.

The shape of the lips is determined by the movement of the markers at both ends of the lips (mouth corner points). Lips pulled (protruding mouth) by moving the corner of mouth outward (inward) with respect to lips
Judge. When the mouth corner points move outward, as shown in FIG. 6, it is assumed that the corresponding model part around the lips moves on the plane determined by the mouth corner points and the ears, and the three-dimensional coordinates of each vertex are shown. To decide. When the corner point of the mouth moves inward, as shown in FIG. 7, the coordinates of the vertices around the mouth are determined based on the data obtained in advance so that the lip is projected.

The nasal groove and upper lip markers are used for adjusting the above movement. In other words, in order to change the shape of the lips, we drive a wide range of vertices, including around the nose, but if we follow only the movement of the markers at the corners of the mouth, the movement of these vertices becomes unnatural. So the aim is to prevent this. If movement of the marker on the upper cheek point is detected,
The vertices around the marker are driven based on the actually measured data.

On the other hand, the facial expression detection unit 31 detects the line of sight and blinks, while the deformation unit 41 determines the position of the black eye portion and opens and closes the eyelid in the eyeball portion of the wire frame model 23 of the human figure. .

Since the three-dimensional position and the rotation of each part can be discriminated by the rotational movement detection part 32 of the head and the rotational movement detection part 34 of the body, the triangles included in the wire frame model corresponding to the part between the joints. The movement is reproduced on the three-dimensional wire frame model 23 by calculating the three-dimensional coordinates of the vertices of the patch. Similarly, the finger model is appropriately transformed by the information obtained by the finger movement detection unit 33.

In this way, the wire frame model 23 is deformed, and the texture mapping unit 42 pastes a color texture to the deformed triangular patch in real time. Since the color texture is obtained in the form corresponding to the triangular patch in the initial state, it corresponds to the deformation of the triangular patch accompanying the movement of the person,
Color texture thinning and interpolation are performed as appropriate.

The three-dimensional human model 23 created as described above is placed in the virtual space created by the computer graphics technique using the virtual space data 43.

FIG. 8 is a diagram showing an embodiment for realizing the operation of the present invention, which is an embodiment for two conference participants at two places. Workstation 61 on one side
Is, for example, IRIS WS Crimson, Rea
light Engine Silicon Graph
It is made of ics, enables high-speed texture mapping, and is used for real-time three-dimensional display of the person 81 on the other side. The three-dimensional image of the person 81 is displayed on the three-dimensional screen 64 for stereoscopic viewing. An example of a three-dimensional screen is one in which images for the left and right eyes are alternately displayed at a frequency of about 60 Hz, and the person 71 on one side is wearing stereoscopic eyeglasses 72 (for the left and right eyes in synchronization with the screen 64). The shutter is opened / closed), and the data glove 74 provided with the magnetic sensor 73 is worn to perform a cooperative work with the person 81 on the other side.

On the other hand, the facial expression of the person 81 is detected on the other side.
63 out of one workstation 62, 63
(Eg IRIS WS 4D340 / VGX). The image information obtained from the television camera 52 attached to the helmet 53 shown in FIG. 3 described above is branched, and one of them is a chroma key 6 for tracking a blue marker.
7 to perform binarization for separating the blue portion and other color portions, and the other for inputting to the level key 68 for detecting the corneal reflection image to perform binarization and tracking at the workstation 63. Perform processing.

The other workstation 6 on the other side
2 (eg IRIS WS 4D240 / VGS)
Are used for information processing from the data glove 85 worn by the person 81 and the magnetic sensor 84. Also, on the other side, a virtual space similar to that on the one side is configured by this workstation 62 and is displayed on the stereoscopic three-dimensional screen 65. However, due to the limitation of the texture mapping capability, the human figure display of the person 81 is simple. Sometimes done by computer graphics.

The motion information of the person 81 obtained by the two workstations 62 and 63 on the other side is Eth.
workstation 6 on one side via ernet64
1 and the three-dimensional model of the person 81 is driven.
Further, a microphone 94 and a speaker 93 are arranged on one side and a microphone 91, a speaker 95, and a delay device 92 are prepared on the other side so that the persons 71 and 81 can communicate by voice. Delay device 9
2 is for delaying the voice in consideration of the processing time for detecting and generating the facial expression.

The performance of the embodiment shown in FIG. 8 is as follows. As a result of experiments of marker tracking and corneal reflection image tracking in various situations, it was found that the operation can be performed at a speed of about 11 frames / sec. A three-dimensional model 23 of a person's upper body image has about 6700 vertices (the head is 1400).
When configured with a wireframe model consisting of
On the other hand, as for the motion generation, as a result of conducting an experiment separately from the motion detection system, it was possible to display at a speed of about 11 frames / sec. When the generation is performed in conjunction with the facial expression detection system, it is 5 to 6 frames / sec when there is something other than a person in the virtual space, and 6 to 7 frames / sec when there is only a person.

[0032]

As described above, according to the present invention, a marker is attached to a facial expression muscle, an image of the face is captured, the marker is tracked in real time, and movements of the body, head, fingers, etc. Is detected in real time and a device capable of texture mapping in real time is used for the three-dimensional human model, so that the motion can be reproduced in real time in the three-dimensional human model.

[Brief description of drawings]

FIG. 1 is a block diagram showing the overall configuration of an embodiment of the present invention.

FIG. 2 is a diagram for explaining the principle of generating a three-dimensional human figure.

FIG. 3 is a diagram for explaining the principle of a facial expression detection unit.

FIG. 4 is a diagram showing a wire frame driving method when the movement of a marker under the lower lip is small.

FIG. 5 is a diagram showing a wire frame driving method when the movement of a marker under the lower lip is large.

FIG. 6 is a diagram showing a wire frame driving method when the markers on both ends of the lip move outward.

FIG. 7 is a diagram showing a wire frame driving method in the case where the markers on both ends of the lips move inward.

FIG. 8 is a diagram showing an embodiment for realizing the operation principle of the present invention.

[Explanation of symbols]

 1 Real Space 2 Person 3 3D Person Model 4 Virtual Space 5 3D Display 6 Conference Participant on Receiving Side 20 Human Image 3D Model Generation Unit 21 Human Body Parts 21a Head 21b Upper Body 21c Upper Arm 21d Lower Bowl 21e Hand 22 Shape Input Device 23 three-dimensional wire frame model 24 video input device 25 color texture 26 laser range scanner 27 TV camera 30 human movement real-time detection unit 31 facial expression detection unit 32 head rotation detection unit 33 finger movement detection unit 34 body Rotational movement detection unit 40 Human image generation unit 41 3D wireframe model deformation unit 42 Color texture mapping unit 43 Virtual space 3D data 44 Virtual space 50 Face 51 Color marker 52 TV camera 53 Helmet 54 Frame 55 Face image 56 window 57 Light bulb 58 Corneal reflection image 59 Television camera 61,62,63 Workstation 64,65 Stereoscopic three-dimensional screen 66 Ethernet 67 Chroma key 68 Level key 74,85 Data glove 73,84 Magnetic sensor 91,94 Microphone 92 Delay device 93, 95 speakers

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Taiichi Kitamura Taiichi Kitamura, Seika-cho, Kyoto Prefecture Prefectural Satoshi-cho, Osamu Osamu, Mihiratani No.5, ATR Communication Systems Laboratories, Inc. (72) Inventor, Fumio Kishino Soraku, Kyoto Prefecture Gunma Seika-cho, Osamu Osamu, Osamu Osamu, 5 Hiratani, AT Communication System Research Institute, Inc. (56) References Yamanashi University Faculty of Engineering Research Report No. 42 (December 1991) P. 16-20

Claims (1)

(57) [Claims] 1. A human model on the transmitting side is configured by color texture mapping to a three-dimensional wire frame model, and this human model is arranged in a virtual space generated by using computer graphics technology, and a three-dimensional image on the receiving side is formed. In a communication method for reproducing a person's movement on a display in a stereoscopic manner, a marker is attached to the skin surface on the facial expression muscle of the person on the transmitting side, a face image of this person is captured, and the image in the image is displayed. While tracking the marker in real time, the cornea reflection image when the cornea of the person is irradiated with light is captured to detect the line of sight and blink, and the movement information of the person's body, head, fingers, etc. is obtained in real time. In the three-dimensional human model, the wire frame model of the portion corresponding to the face is driven based on the tracking result of the marker to detect the line of sight and blink. Based on the above, the position of the iris of the eyeball of the three-dimensional human model and the opening and closing of the eyelids are performed, and each corresponding portion of the three-dimensional human model is driven by the detected motion information. Real-time recognition synthesis method.