JP2001092990A - Three-dimensional virtual space participant display method, three-dimensional virtual space display device and recording medium stored with three-dimensional virtual space participant display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To report the expression of an operator in real time in a three- dimensional(3D) virtual space system. SOLUTION: A glance direction, the voice of the operator, a 3D face model prepared by an initial input video processing part 13 and the real face video of the operator determined by a display face direction discriminating part 14 or real face video of the operator photographed in all directions are transmitted from a data transmitting part 15 to an other person information receiving part 20 of the 3D virtual space display device by a present information receiving part 10. In the other person information receiving part 20, the real face video or 3D CG face model in the direction determined by the display face direction discriminating part 23 is displayed by a face video display part 26 or face model display part 25.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method in which a plurality of participants communicate with each other through a communication line using a plurality of computer terminals, and a three-dimensional virtual space constructed by using computer graphics technology. The present invention relates to a three-dimensional virtual space participant display method in a system shared by the members.

[0002]

2. Description of the Related Art As a prior art, there is a three-dimensional virtual space participant display method announced at the 57th National Convention of the Information Processing Society of Japan, announcement number 5M-8. According to this method, avatar textures obtained by previously capturing the whole body of a participant in a virtual space from multiple directions are prepared, and the avatar textures are switched and displayed so as to be suitable for the direction seen by other participants. Also, in the image processing technology, when a person's face is photographed, the color components in the image are decomposed into luminance, saturation, and hue, and a region that is considered to represent skin color in the hue is fixedly set. ,
2. Description of the Related Art A technique for extracting a face area of a person from a background in a video is widely used.

[0003]

As a conventional method of displaying the alter ego, there is a method of joining a real image of a face taken from one direction to a body CG image as it is, and a method of manually displaying a three-dimensional space based on the actual image. There is a method of creating a plate-like area, pasting an actual image there, and creating and displaying an image of the alter ego.

According to the method, when another participant looks from the side, his / her expression cannot be understood.

The method described above has a drawback that it takes several hours or more because it is performed manually, and cannot be displayed in real time.

An object of the present invention is to provide a method and apparatus for displaying a three-dimensional virtual space participant capable of transmitting an expression of an operator in a three-dimensional virtual space in real time, and a recording medium storing a three-dimensional virtual space participant display program. Is to provide.

[0007]

SUMMARY OF THE INVENTION According to the present invention, a three-dimensional face model and a three-dimensional torso created from a composite of a real shot image and a three-dimensional torso model and a real shot face image based on the virtual space and the situation of the alter ego of the operator. One of the models is selected and displayed.

When displaying an actual image as it is, face images are taken from a plurality of directions, and a face image to be displayed is selected based on the line of sight of the operator or another participant. As a result, it is possible to display the expression of the operator irrespective of the gaze direction of the operator or other participants.

When displaying a three-dimensional face model, a standard skeleton model is prepared in advance, and a captured face image is applied to the skeleton model to create and display a face model. This enables automation and real-time processing of creating a three-dimensional face model.

[0010]

Next, embodiments of the present invention will be described with reference to the drawings.

FIGS. 1A and 1B are processing configuration diagrams on the transmitting side and the receiving side of a three-dimensional virtual space display apparatus according to an embodiment of the present invention.

The three-dimensional virtual space table device transmits information of each user from a self information transmitting unit 10 for transmitting own information to a terminal used by another virtual space user and a terminal used by another virtual space user. It has a third party information receiving unit 20 for receiving. The self-information transmitting unit 10 includes a gaze direction input unit 11 and a face image capturing unit 1
2, face display direction determination unit 13 and initial input face image processing unit 14
And a data transmission unit 15 and a voice input unit 16.
The other person information receiving section 20 includes a data receiving section 21, a face display method determining section 22, a display face direction determining section 23, a sound pressure detecting section 24, a face model display section 25, and a face video display section 26.

The gaze direction input unit 11 is an input device such as a mouse for an operator to move in a virtual space. The face image capturing unit 12 captures a real face image of the operator. The initial input face image processing unit 13 takes an image of the operator with the face image capturing unit 12 and creates a three-dimensional face model that resembles the operator in the virtual space.
The display face direction determination unit 14 determines from which camera the state of the operator's face imaged is displayed in a virtual space displayed on a terminal used by a virtual space user other than the operator. The voice input unit 16 is used to input an utterance of the operator.
An input device such as a microphone. The data transmitting unit 15 includes a viewpoint position and a viewpoint direction in the virtual space of the operator determined based on the information input from the line-of-sight direction input unit 11.
Only the three-dimensional face model, the voice data emitted by the operator, and the real face image of the operator photographed from all directions or the real face image of the operator photographed from the direction determined by the display face direction determination unit 14 Is transmitted to the other party information receiving unit 20 of the three-dimensional virtual space display device used by the virtual space user other than the operator.

The data receiving unit 21 includes a viewpoint position and a viewpoint direction transmitted from the own information transmitting unit 10 of the three-dimensional virtual space display device used by another virtual space user, and voice data generated by another virtual space user. , A real face image captured from all directions of other users, or one of the real face images, a three-dimensional face model is received. The face display method determination unit 22 determines whether to display a face model or a face video according to the status of the virtual space set in advance. The face image display unit 26 removes the background of the operator, corrects the position of the face image for display so as to be appropriate for display with the torso model, and displays the face image. The display face direction determining unit 23 is configured to transmit a real face image of the operator photographed from all directions from the own information transmitting unit 10 of the three-dimensional virtual space display device used by another virtual space user. First, it is determined from the relationship between the relative position and the direction in the virtual space with the other party to determine which real face image is displayed by the user. The face model display unit 25 displays the face model created by the initial input face image processing unit 13 of the three-dimensional virtual space display device used by another virtual space user. The sound pressure detection unit 24 detects voice data received from another virtual space user, and transmits the state to the face model display unit 25.

Here, the reason why the display face direction determination unit is provided on the transmitting side and the receiving side will be described.

First, in the three-dimensional virtual space display apparatus of the present invention, a pair of the own information transmitting section and the other information transmitting section is one set.

Each participant transmits its own information to another participant, receives information from the other participant, and displays the avatar of the other participant on its own terminal. My avatar is not displayed on my terminal. When transmitting one's own face image to another person, transmitting all of the face images taken from all directions to the other party, and selecting the face image to be displayed to the other party based on the relationship between the relative position and the direction to oneself In some cases, the user himself / herself calculates the relationship between the relative position and direction with the other party, and transmits only a face video taken from a certain required angle. In the former case, the display face direction determining unit 23 in the other party information receiving unit 20 of the other party's three-dimensional virtual space display device is required, and in the latter case, the own information transmitting unit 10 of the own three-dimensional virtual space display device is used. A certain display face direction determination unit 14 is required.

This mechanism is necessary to keep the computer load and the communication load of each operator's terminal in an appropriate state according to the size of the number of persons participating in the virtual space. When the number of participants is small, a method is adopted in which all face images are sent and the other party selects a face direction in order to reduce the computational load of the own terminal and switch face images quickly according to the movement of the other party in the virtual space.
When the number of participants is large, in order to reduce the communication load and the processing load of the partner information, the own terminal determines the face direction and transmits only the face video shot from a required angle.

FIG. 2 is a diagram showing a state around an operator terminal of the three-dimensional virtual space display device. Five cameras 31 to 35 for photographing the face of the operator 36 participating in the virtual space sitting in front of the computer are provided. These five cameras 31 to 35 are viewed from above the operator 36.
As shown in FIG. 3, when the front is set at 12 o'clock,
It is installed in each of five directions of 9:00, 10:30, 1:30, and 3:00, and photographs the operator's face from each angle. Reference numerals 41 to 45 denote images taken by the cameras 31 to 35, 46 denotes the position of the operator, and 47 denotes the position of the observed participant.

FIG. 4 is a flowchart showing details of the processing of the initial input face image processing section 13. In this processing unit, a model resembling the operator is created (step 51). A face image taken from the direction of 12:00 is used. At the time of photographing, as shown in FIG. 5, guidelines 61 and 62 for aligning the center line of the face with the height of the eyes are displayed, and the operator adjusts the photographing position along the guidelines. Next, a face area is extracted from the captured video (step 52). The entire video is divided into regions based on hue, and a region including an intersection between the guidelines 61 and 62 is extracted as a face portion. The positions of both eyes and the mouth are specified from the area intersecting the guidelines 61 and 62 in the extracted face area. The distance between the eyes in the shot video,
Based on the distance from the eyes to the mouth, the size of the generalized three-dimensional head CG model is adjusted in advance by expanding and contracting to match the scale of the operator's face (step 53). Paste the extracted face area on the adjusted face model (Step 5)
4). The face model created as described above is displayed on the face model display unit 25 in a necessary situation.

FIG. 6 is a flowchart showing details of the processing of the face image display section 26. First, the initial input face image is
A frame, a face image taken immediately after the initial input face image is taken as a second frame (step 71). Also, the average hue of the face image extracted from the initial input face image is recorded. As shown in FIG. 7, a difference video obtained by taking a difference between the first frame and the second frame is prepared. Only the information of the subject moved between the first frame and the second frame remains in the difference video. Next, the second frame is divided into regions by hue using the same method as that for the first frame (step 7).
2). Here, an area having a large overlap with the face area of the first frame and an area in which the average hue of the area of the second frame is closest to the average hue of the recorded first frame are extracted from the second frame, and the face in the second frame is extracted. The area is set (step 73). Further, the average hue of the extracted area is newly recorded as the average hue of the first frame for the next processing. After that, an area adjacent to the face area in the second frame is extracted from the area having a large overlap with the difference image. The operator's area is extracted from the second frame by the above method. The extracted region is displayed after performing position correction and size correction using the affine transformation (Video Analysis Handbook, University of Tokyo Press, p. 427) (steps 74 to 76). The above is one cycle of the face display processing, and the second frame is newly added to the first frame, and the video shot immediately after the second frame is newly added to the second frame.
The processing cycle is repeated as a frame.

FIG. 8 shows the details of the processing of the display face direction judging units 14 and 23. Is the angle formed by the frontal directions of the participants. When θ <−112.5 ° and 112.5 ° <θ, the face model is displayed without displaying the face image by the determination of the face display method determination unit 22. When −90 ° ≦ θ ≦ 90 °, the display face direction determining unit 14 displays a face image suitable for the angle with the other party according to the determination. That is, −112.5 ° ≦ θ
When <−67.5 °, the face image at the angle indicated by the symbol 41 in FIG. 3 is used. When −67.5 ° ≦ θ <−22.5 °, the face image at the angle indicated by the symbol 42 in FIG. , -22.5 ° ≦ θ <2
When the angle is 2.5 °, the face image at the angle indicated by the symbol 43 in FIG. 3 is displayed, and when 22.5 ° ≦ θ <67.5 °, the symbol image in FIG.
Is 27.5 ° ≦ θ <67.5 °, the face image at the angle indicated by the symbol 44 in FIG. 3 is 67.5 °.
When .ltoreq..theta..ltoreq.112.5 DEG, a face image at the angle indicated by the symbol 45 in FIG. 3 is displayed.

The face display method judging section 22 displays a face image at a place previously set in the virtual space, and displays a face model otherwise. FIG. 9 shows an avatar display method in each case. When displaying a face image, a face display screen 93 is set up on the torso model, and the face image is attached to the screen 93. The screen 93 always faces the observer. In the present embodiment, the setting is made so that the faces seen by the participants on the table of the prepared conference room are switched from the face model to the face video.

FIG. 10 shows an avatar when a CG face model is displayed. The display of the face model is switched according to the utterance of another operator input through the microphone. The sound pressure detection unit 24 informs the face model display unit 25 of the utterance status of the operator. The face model display unit 25 switches to display the open face model 102 when the utterance is confirmed, and to display the closed face model 101 when the utterance is not confirmed.

FIGS. 11A and 11B are configuration diagrams of the virtual space display device according to the embodiment of the present invention. The own information transmission unit (FIG. 11A) of the three-dimensional virtual space display device is the input device 11.
1, a photographing device 112, a microphone 13, a storage device 114, a transmitting device 115, a recording medium 116, and a data processing device 117.
It is composed of The input device 111 is, for example, a mouse for inputting a line-of-sight direction. The microphone 13 corresponds to the voice input unit 16 in FIG. The photographing device 112 is a camera for photographing the face of the operator. The storage device 114 is a hard disk. The transmitting device 115 transmits the real face image, the three-dimensional face model, and the like to the other information receiving unit of the virtual space display device (FIG.
1 (2)). Recording medium 11
Reference numeral 6 denotes a recording medium, such as a floppy disk, CD-ROM, or magneto-optical disk, on which a three-dimensional virtual space display program including the components 11, 13 to 15 in FIG. 1 is recorded. The data processing device 117 is a CPU that reads the three-dimensional virtual space participant display program from the recording medium 116 and executes the program.

The other party information receiving section (FIG. 11 (2)) of the three-dimensional virtual space display device includes a receiving device 121, a storage device 122, a display device 123, a recording medium 124, and a data processing device 12
5. The receiving device 121 is a transmitting device 115
Receives data sent from. The storage device 122 is a hard disk. The display device 123 is a display that displays a real image of another operator or a three-dimensional CG face model. The recording medium 124 is a recording medium, such as a floppy disk, a CD-ROM, or a magneto-optical disk, on which a three-dimensional virtual space display program including the components 21 to 26 in FIG. 1 is recorded. The data processing device 125 is a CPU that incorporates the three-dimensional virtual space display program from the recording medium 124 and executes the program.

[0027]

As described above, according to the present invention,
The face area can be extracted regardless of the lighting condition and the background condition in the photographing of the operator's face, and the expression and gaze direction of the operator can be transmitted in real time in a three-dimensional virtual space.

[Brief description of the drawings]

FIG. 1 is a processing configuration diagram of a three-dimensional virtual space participant display device according to an embodiment of the present invention.

FIG. 2 is a diagram showing an arrangement of cameras 31 to 35 for photographing a face of an operator sitting in front of an operator terminal.

FIG. 3 is a view showing an operator's face captured by each of the cameras 31 to 35;

FIG. 4 is a flowchart showing processing of an initial input face image processing unit 13;

FIG. 5 is a diagram illustrating a state of capturing an initial input face image.

FIG. 6 is a flowchart showing processing of a face image display unit 26;

FIG. 7 is an explanatory diagram of face video processing.

FIG. 8 is a diagram showing a positional relationship between participants in display face direction determination.

FIG. 9 is an explanatory diagram of a method of displaying an avatar in a virtual space.

FIG. 10 is a diagram illustrating a difference in a method of displaying an avatar's face depending on an utterance situation of an operator.

FIG. 11 is a configuration diagram of a three-dimensional virtual space display device according to an embodiment of the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 10 Self-information transmission part of three-dimensional virtual space display apparatus 11 Gaze direction input part 12 Face display photographing part 13 Initial input face image processing part 14 Display face direction determination part 15 Data transmission part 16 Voice input part 20 Three-dimensional virtual space display device Other information reception unit 21 Data reception unit 22 Face display method determination unit 23 Display face direction determination unit 24 Sound pressure detection unit 25 Face model display unit 26 Face video display unit 31 Camera that captures a face from the right of the operator 32 Operation A camera that captures a face from the front of the operator 33 A camera that captures a face from the front of the operator 34 A camera that captures a face from the front of the operator 35 A camera that captures the left face of the operator 36 An operator 37 System virtual space display device 41 Video captured by camera 31 42 Video captured by camera 32 43 Video captured by camera 33 44 Video captured by camera 34 Image 45 Image taken by camera 35 46 Observer's position 47 Observed participant's position 61 Guidelines for adjusting the center position of the face at the time of shooting 62 Guidelines for adjusting the eye height at the time of shooting 81 Observed Position and line-of-sight direction of the operator 82 position and line-of-sight direction of the observer 91 an avatar displaying a face in a CG model 92 an avatar displaying a face in a video 93 a screen 101 for displaying a face video 101 the operator emits sound The avatar's face displayed on the other user's 3D virtual space display device when not present 102 The avatar's face displayed on the other user's 3D virtual space display device when the operator is emitting sound 111 Input Device 112 Imaging Device 113 Microphone 114 Storage Device 115 Transmitting Device 116 Recording Medium 117 Data Processing Device 121 Receiving Device 12 2 storage device 123 display device 124 recording medium 125 data processing device

Claims (15)

[Claims] 1. A three-dimensional virtual space participant display method in a three-dimensional virtual space system in which a plurality of users each communicate via a communication line using a computer terminal and share a three-dimensional virtual space, And a 3D CG face model and a 3D CG created from a synthesized face image and a 3D CG torso model or from a real shot face image according to the situation of the alter ego of the operator
A three-dimensional virtual space participant display method for displaying an alter ego of an operator in a virtual space by synthesizing a trunk model. 2. The three-dimensional virtual space participant display method according to claim 1, wherein a three-dimensional CG face model to which an image of the operator's own face is attached is created in real time from a captured operator's face image. 3. The apparatus according to claim 1, wherein a background portion other than the operator of the virtual space display device is removed from the captured image, and the head image of the operator is joined to the body of the three-dimensional CG model and displayed. 3D virtual space participant display method. 4. A face is photographed from a plurality of directions, and an image of the face is displayed in accordance with the swing of the participant's alter ego's head by the operation of the participant or in accordance with the direction in which the other participant views. The three-dimensional virtual space participant display method according to claim 1. 5. The three-dimensional virtual space participant display method according to claim 1, wherein a three-dimensional CG face model whose mouth is opened and closed is displayed according to the presence or absence of an utterance of the operator. 6. A three-dimensional virtual space display device in a three-dimensional virtual space system in which a plurality of users communicate through a communication line and share a three-dimensional virtual space, and inputs a line-of-sight direction of an operator in a virtual space. A gaze direction input unit, an initial input face image processing unit that creates a three-dimensional CG face model that resembles a captured operator in a virtual space, and a face that captures a real face image of the operator from different directions. An image capturing unit, an audio input unit for inputting an operator's voice, and an image of an operator's face taken from any of the facial image capturing units is displayed in a virtual space displayed on a terminal used by a user other than the operator. A display face direction determination unit that determines whether the gaze direction is input by the gaze direction input unit, a voice input by the voice input unit, a three-dimensional face model created by the initial input face image processing unit, Photographed by the face video camera Among the captured real face images, the real face image captured by the face image capturing unit determined by the display face direction determination unit or the real face image captured from all directions is displayed on the 3D virtual space display device. A three-dimensional virtual space display device, comprising a data transmission unit for transmitting to a person information reception unit. 7. A data receiving section for receiving data transmitted from the three-dimensional virtual space display device according to claim 6, and a live-action face image and a three-dimensional CG body according to the virtual space and the situation of the alter ego of the operator. Display the composite of the model or
A face display method determining unit for determining whether to display a 3D CG face model and a 3D CG torso model created from a live-action face image, and removing the operator's background when the former face display method is selected. A face image display unit for performing position correction of the face image for displaying the three-dimensional CG torso model and an appropriate position on the display, and displaying the face image; and a three-dimensional virtual space display device according to claim 6. When a real face image of an operator photographed from all directions is transmitted, based on the relative position and direction of another operator's alter ego in the virtual space, which real image of the other A display face direction determining unit for determining whether to display, and a face model for displaying a face model created by the initial input face image processing unit when the latter face display method is selected in the face display method determining unit. 3 characterized by having a display unit Based on the virtual reality display device. 8. A sound pressure detection unit for detecting an utterance of another operator participating in the same virtual space and transmitting the utterance to the face model display unit, wherein the face model display unit receives the utterance. The apparatus according to claim 7, wherein the apparatus displays an open face model. 9. The initial input face video processing unit captures an operator's face, extracts a face area from the captured video, specifies the positions of both eyes and a mouth in the extracted face area, Based on the distance between both eyes and the distance from the eyes to the mouth in the captured video, the size of the generalized three-dimensional head CG model is adjusted in advance by expanding and contracting to match the scale of the operator's face, 7. The apparatus according to claim 6, wherein the extracted face area is pasted on the adjusted face model. 10. The face image display unit according to claim 1, wherein the first input face image is a first frame, the face image taken immediately after the initial input face image is taken is a second frame, and the face extracted from the initial input face image is used as the first frame. The average hue of the region is recorded, a difference image is obtained by taking the difference between the first frame and the second frame, and the second frame is divided into regions by hue.
An area having a large overlap with the face area of the first frame, and
An area where the average hue of the area of the frame is closest to the average hue of the recorded first frame is extracted from the second frame.
The face area in the frame and the average hue of the extracted area are newly recorded as the average hue of the first frame for the next process. Extracting an area adjacent to the face area and displaying the extracted area after correcting the position and size of the extracted area means that the second frame is newly photographed immediately after the first frame and the second frame. The apparatus according to claim 7 or 8, wherein the reproduced image is repeated as a second frame. 11. A program for displaying a three-dimensional virtual space in a three-dimensional virtual space system in which a plurality of users communicate via a communication line and share a three-dimensional virtual space, comprising: Gaze direction input processing to input, initial input face image processing to create a 3D model that resembles the photographed operator in the virtual space, and the state of the operator's face taken from any face image photographing unit A display face direction determination process for determining whether to display in a virtual space displayed on a terminal used by the operator other than the operator; a gaze direction input in the gaze direction input process; A three-dimensional face model, a real face image photographed by the face image photographing unit determined by the display face direction determination processing, or a real face image photographed from all directions, among the photographed real face images A recording medium recording a three-dimensional virtual space display program for executing a data transmission process to the computer to be transmitted to another person information receiving unit of the three-dimensional virtual reality display device. 12. A data receiving process for receiving data transmitted from the data transmitting process according to claim 11, and synthesizing a real face image and a three-dimensional CG torso model according to a virtual space and a situation of an alter ego of an operator. Display what you did,
A face display method determination process for determining whether to display a 3D CG face model and a 3D CG torso model created from a live-action face image, and removing the operator's background when the former face display method is selected. 12. A face image display process for correcting a position of a face image for displaying an appropriate position on the display with the three-dimensional CG torso model, and displaying the face image; and all directions from the data transmission process of claim 11. Which live-action face image of the other party is displayed based on the relative position and direction of another operator in the virtual space when the live-action face image of the operator taken from is transmitted And a face model display process for displaying a face model created in the initial input face image process when the latter face display method is selected in the face display method determination process. Let it run A recording medium recording a three-dimensional virtual space display program of the eye. 13. The face model display process detects an utterance of another operator participating in the same virtual space, and, when an utterance is transmitted from a sound pressure detection unit to be transmitted to the face model display unit,
13. The recording medium according to claim 12, wherein an open face model is displayed. 14. The initial input face image processing includes photographing an operator's face, extracting a face region from the photographed image, specifying the positions of both eyes and a mouth in the extracted face region, and photographing. The size of the generalized 3D head CG model is adjusted and adjusted in advance based on the distance between the eyes and the distance from the eyes to the mouth in the image to fit the scale of the operator's face 12. The recording medium according to claim 11, wherein the extracted face area is pasted on the extracted face model. 15. The face image display process includes: setting an initial input face image as a first frame, a face image captured immediately after capturing the initial input face image as a second frame, and extracting a face extracted from the initial input face image. Record the average hue of the area,
Prepare a difference video obtained by taking the difference between the frame and the second frame, perform region division on the second frame by hue,
An area having a large overlap with the face area of the first frame, and
An area where the average hue of the area of the frame is closest to the average hue of the recorded first frame is extracted from the second frame.
The face area in the frame and the average hue of the extracted area are newly recorded as the average hue of the first frame for the next processing. After that, among the areas having a large overlap with the difference image, an area adjacent to the face area in the second frame is extracted, and the extracted area is displayed after correcting the position and size of the extracted area. 14. The recording medium according to claim 12, further comprising: repeating a new image as a first frame and a video shot immediately after the second frame as a new second frame.