JP2001228794A - Conversation information presenting method and immersed type virtual communication environment system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a means through which a user easily converses with other user while the users are freely walking in a virtual space in an immersed type virtual communication environment. SOLUTION: Plural display devices are arranged to surround the users in an immersed type virtual communication environment. In the conversation information presenting method, uttered contents of an uttering user are inputted, character information is generated from the inputted contents, three dimensional positions of the uttering user and the user receiving the conversation and their line of sight vectors in the environment are extracted, the distance between the uttering user and the receiving user is extracted by the three dimensional positions of the users, the field of vision of the receiving user is extracted by the user's line of sight vector and a beforehand determined field of vision angle, the presentation position of the uttered user's character information is determined based on the distance between the users, the field of vision of the receiving user, the three dimensional position of the uttering user ad the line of sight vector of the uttering user and the generated character information is outputted based on the determined presentation position.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a conversation information presentation method and an immersive virtual communication environment system, and more particularly to a conversation of a hearing impaired person in an immersive virtual environment in which a plurality of display devices are arranged so as to surround a user. The present invention relates to a technology that is effective when applied to a conversation information presentation technology for presenting a text image of the contents of a user's utterance for the purpose of support or the like.

[0002]

2. Description of the Related Art Conventionally, there is an immersive multi-sided display system in which an immersive virtual environment can be experienced. This immersive multi-surface display system was originally developed as a visualization environment for simulation and the like. In recent years, researches on connecting them via a network and using them as a communication environment have been actively conducted. The immersive multi-surface display system has a structure in which a plurality of screens (display devices) are arranged in front, rear, left, right, top, bottom, etc., and surrounds the user with images, so that a high sense of reality can be obtained.

[0003] In such an immersive virtual communication environment, a user becomes an avatar (self), can freely walk around a three-dimensional virtual world, can see the front, rear, up, down, left and right directions, and can use other avatars (other users). You can have a conversation place when you encounter. At this time, conversation information of the speaking user is presented by voice.

[0004] Such an immersive virtual communication environment is described in, for example, literature: IEICE Technical Report, MVE99.
-45, pp. 1-8, 1999 (Takashi Kono, Yuriko Suzuki, Norio Yamamoto, Shinichi Shiwa, Satoshi Ishibashi, titled "Immersive Virtual Communication Environment").

On the other hand, in a non-immersive display system in which a virtual environment can be experienced on a one-sided display, text information such as subtitles on a television can be used as an alternative to voice presentation of the content of the conversation to assist the hearing impaired. There is a way to present. In this case, since the display exists in the direction of the user's field of view, the user can acquire conversation information by presenting character information on a part of the display.

[0006]

However, the method of presenting a character such as a subtitle of a television employed in a non-immersive display system is not suitable for an immersive multi-sided display system. In the immersive multi-surface display system, a plurality of displays are installed so as to surround the user, and the user can see the front, rear, up, down, left, and right directions, so that the view direction of the user is not fixed to a specific display. For this reason, the method of presenting character information such as television subtitles on a part of a specific display has a problem that acquisition of conversation information fails when the user does not look at the specific display.

Further, even when the user is looking at the display, if the speaker is not in the field of view of the user, there is a problem that the user cannot grasp the position of the speaker and cannot communicate smoothly. . Furthermore, if the user attempts to place the specific display in view for acquiring conversation information, a problem arises that the user cannot freely roam in the virtual space.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems. In an immersive virtual communication environment, a user can freely walk around in a virtual space.
An object of the present invention is to provide a conversation information presentation technology that can easily have a place for conversation with another user.

The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

[0010]

SUMMARY OF THE INVENTION Among the inventions disclosed in the present application, the outline of a representative one will be briefly described.
It is as follows.

(1) In an immersive virtual communication environment in which a plurality of display devices (displays) are arranged so as to surround a user, an utterance input process of inputting the utterance content of the utterance user, and a character input from the input utterance content A character generating step of generating information, an extracting step of extracting a three-dimensional position and a line-of-sight vector of the uttering user and the receiving user in the virtual environment, and a three-dimensional position of the uttering user and the receiving user. A distance extracting step of extracting a distance, a view extracting step of extracting a view of the receiving user by a line-of-sight vector of the receiving user and a predetermined viewing angle, a distance between the talking user and the receiving user, and a view of the receiving user. And the presentation position of the character information of the speaking user based on the three-dimensional position of the speaking user and the gaze vector of the speaking user A presentation position determination process of determining a conversation information presentation method and an output step of outputting on the basis of character information the generated to the determined presentation position.

(2) In the conversation information presentation method according to the means (1), in the presentation position determination step, the distance between the speaking user and the receiving user is within a preset distance, and the three-dimensional position of the speaking user is determined. Is within the visual field of the receiving user, the surroundings of the speaking user are set as the presentation position of the character display image for a certain period of time from the utterance, and otherwise, the time from the utterance of the surroundings of the speaking user in the direction of the line-of-sight vector. This is the process of setting the position to be moved to the presentation position.

(3) An immersive virtual communication environment system in which a plurality of display devices (displays) are arranged so as to surround the user, and utterance input means for inputting the utterance content of the uttering user, and the input utterance Character generating means for generating character information from the contents, extracting means for extracting the three-dimensional positions of the speaking user and the receiving user and the line-of-sight vector, and distance between the speaking user and the receiving user according to the three-dimensional positions of the speaking user and the receiving user Distance extracting means for extracting the visual field of the receiving user by the line-of-sight vector of the receiving user and a predetermined visual field angle, and the distance between the speaking user and the receiving user and the visual field of the receiving user. The presentation position of the character information of the speaking user is determined based on the three-dimensional position of the speaking user and the gaze vector of the speaking user. And shows the position determining means, the character information the generated is to and an output means for outputting, based on the determined presentation position.

(4) In the immersive virtual communication environment system of the means (3), the presenting position determining means determines that the distance between the previous uttering user and the receiving user is within a preset distance, and that the uttering user has a third distance. When the three-dimensional position is within the field of view of the receiving user, the surroundings of the uttering user are set as the presentation position of the character display image for a fixed time from the utterance,
In other cases, the presentation position is a position that moves from the periphery of the utterance user in the direction of the line of sight with time from the utterance.

According to the above-described means, a user (for example, a hearing-impaired user) can grasp the position of the speaking user and the speaking content in the immersive virtual communication environment. This makes it possible to easily have a place for conversation with other users while freely walking around in the virtual space.

Hereinafter, the present invention will be described in detail with reference to the drawings together with embodiments (examples) according to the present invention.

[0017]

FIG. 1 is a block diagram showing a schematic configuration of an immersive virtual communication environment system according to an embodiment (example) of the present invention.

In FIG. 1, an utterance input means 1 inputs utterance contents of an uttering user and converts the utterance contents into data. As the utterance input unit 1, for example, an utterance voice input device such as a microphone or a motion capture operation input device for gesture utterance is used.

The character generation means 2 recognizes the utterance content data input from the utterance input means 1 and converts it into character information. As the character generating means 2, for example, a voice recognition device is used if the utterance content data is voice information, and a motion recognition device is used if the utterance content data is motion information.

Here, as the voice recognition software, for example, a voice recognition engine REX (NTT) is known. In addition, the motion recognition method is described in, for example, literature: IEICE Technical Report,
MVE99-36, 1999/7 (Tomoaki Yabe, et al., Entitled "Gesture Recognition Based on Network Structure Correspondence between Gesture Moving Image and Meaning Word Sequence").

The extracting means 3-1 and 3-2 extract a three-dimensional position and a line-of-sight vector of the speaking user and the receiving user in the virtual environment, respectively. As an example of the extracting method, there is a method of extracting a position in a virtual environment by attaching a position sensor for detecting a position to a user's body. Specifically, the actual position / direction information is provided by a magnetic sensor attached to three-dimensional (3D) glasses or a magnetic sensor provided in a three-dimensional wand (a bar-shaped interface device having a magnetic sensor and a switch button). It is conceivable to extract a user's three-dimensional extraction means in a virtual environment space using a processing personal computer (PC) or the like.
In particular, it is considered that it is better to attach a sensor to the head in order to faithfully extract the direction in which the user is looking at the line-of-sight vector. There is no need to stick to the position sensor device, and it goes without saying that a device with good detection accuracy is good.

Here, the three-dimensional position of the speaking user in the extracted virtual environment is A, the line-of-sight vector at the position A is vector a, the three-dimensional position of the receiving user is B, and the line-of-sight vector at the position B is B. Is a vector b. Also,
The line-of-sight vector a is a direction vector starting from the three-dimensional position A of the user. The line-of-sight vector b is also a direction vector starting from the three-dimensional position B.

The distance extracting means 4 inputs the three-dimensional position A of the speaking user and the three-dimensional position B of the receiving user obtained by the extracting means 3-1 and 3-2. The distance is calculated, and the distance d between the speaking user and the receiving user is extracted.

The view extracting means 5 inputs the line-of-sight vector b of the receiving user obtained by the extracting means 3-2, and sets a predetermined view angle R as shown in FIG. To extract a field of view W having an infinite conical shape. The view angle R is defined as the angle of the field of view of the receiving user considered necessary when talking in the virtual environment.
Angle. This view angle R can be freely adjusted, and the larger the view angle R, the wider the view W.

The presentation position determining means 6 includes the extracting means 3-
1. The three-dimensional position A and the line-of-sight vector a extracted in step 1, the distance d extracted by the distance extracting unit 4 and the field of view W extracted by the field extracting unit 5 are input, and the presentation position of the character information is determined based on the input. To decide.

The output means 7 is an output means for inputting the character information generated by the character generation means 2 and the presentation position determined by the presentation position determination means 6, and outputting the character information based on the presentation position.

FIG. 3 is a flowchart showing a processing procedure of the presentation position determining means 6. As shown in FIG. 3, the processing procedure of the presentation position determination unit 6 is as follows: the distance d between the speaking user and the receiving user from the distance extracting unit 4, and the three-dimensional position A and visibility of the speaker from the extracting unit 3-1. The field of view W is input from the extraction means 5. Then, the distance D is set.
The distance D set in advance is a distance considered to be suitable for conversation between users in the virtual environment. Distance D
The longer the conversation, the wider the conversation range, so that a clear conversation can be made with a distant user.
As shown in FIG. 4, the uttering user is at a distance D from the receiving user.
3 and the order of dividing into cases other than those within the field of view W as shown in FIG.
分類 Classify based on the judgment of W. When d ≦ D and A⊂W hold, the presentation position of the character information is determined to be around the speaking user for a fixed time T (set time) after the utterance.

On the other hand, when d ≦ D or A⊂W does not hold, the presentation position of the character information is set to be around the speaking user. This is set as the initial presentation position, and after a certain time T (set time) has elapsed, the presentation position of the character information is moved in the direction of the line-of-sight vector a of the speaking user.

FIGS. 5 and 6 are schematic diagrams showing examples of outputting character information. FIG. 5 is an output example when d ≦ D and A⊂W holds, in which a balloon image is output near the mouth and the utterance content is displayed in the balloon. When d ≦ D or A⊂W does not hold, as shown in FIG. 6, the speech balloon image is displayed at the mouth, which is the initial presentation position, and the speech content is displayed in the speech balloon.

When a predetermined time T has passed since the utterance, the user is moved from the initial presentation position to the presentation position a predetermined distance P from the utterance user in the direction of the vector a. Further, after a lapse of time twice as long as the time T, a position twice the distance P from the initial presentation position, and a destination that is increased n times from the initial presentation position every time the time increases n times (n is a natural number) is referred to as a presentation position. I do.
The time T and the distance P can be set freely, but it is necessary to balance them so that the characters in the balloon are easy to read. In addition to a balloon, a sphere, a floating body such as a cloud, or the like may be used for outputting character information.

That is, as shown in FIG. 1, the conversation information presenting method of the present embodiment is an immersive virtual communication environment in which a plurality of display devices (displays) are arranged so as to surround a user (for example, a hearing impaired user). , The utterance content of the utterance user is input to the utterance input means 1
Character generating means 2 generates character information from the input utterance contents.
The three-dimensional position and the line-of-sight vector of the speaking user and the receiving user in the immersive virtual communication environment are extracted by the extraction means 3-1 and the extraction means 3-2. The distance between the speaking user and the receiving user is extracted by the distance extracting means 4 based on the three-dimensional positions of the speaking user and the receiving user, and the field of view of the receiving user is extracted by the line-of-sight vector of the receiving user and a predetermined visibility angle. Extract at 5.
The presentation position determining unit 6 determines the presentation position of the character information of the speaking user based on the distance between the speaking user and the receiving user, the field of view of the previous receiving user, the three-dimensional position of the speaking user, and the line-of-sight vector of the speaking user. The generated character information is output by the output means 7 based on the determined presentation position and displayed on a display device (display).

In the presenting position determining step, as shown in FIG. 3, the distance between the speaking user and the receiving user is within a predetermined distance D, and the three-dimensional position of the speaking user is within the field of view of the receiving user. In some cases, the surroundings of the uttering user are set as the presentation position of the character display image for a certain time T from the utterance,
In other cases, a position that moves from the surroundings of the speaking user in the direction of the line of sight with time from the speaking is set as the presentation position.

As described above, the invention made by the present inventor is:
Although specifically described based on the embodiment (example),
The present invention is not limited to the above-described embodiment, and it is needless to say that various modifications can be made without departing from the spirit of the present invention.

[0034]

As described above, according to the present invention,
A user (for example, a hearing-impaired person) can easily grasp the position of the speaker and the contents of the utterance in the immersive virtual communication environment, and also easily understand the utterance act of the speaker. It is possible to have a place for conversation with other users while freely walking around in the space.

As described above, the invention made by the present inventor is:
Although specifically described based on the embodiment, the present invention
It is needless to say that the present invention is not limited to the above-described embodiment, but can be variously modified without departing from the scope of the invention.

[0036]

As described above, according to the present invention,
In an immersive virtual communication environment, a user (for example, a hearing-impaired user) can grasp the position of the speaking user and the content of the speaking. This makes it possible to easily have a place for conversation with other users while freely walking around in the virtual space.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of an immersive virtual communication environment system according to an embodiment (example) of the present invention.

FIG. 2 is a diagram illustrating an example of a field of view W of a receiving user in the present embodiment.

FIG. 3 is a flowchart illustrating a processing procedure of a presentation position determining unit in the embodiment.

FIG. 4 is a diagram illustrating an example of a conversation range classification in the embodiment.

FIG. 5 is a diagram showing an output example of character information in the embodiment.

FIG. 6 is a diagram illustrating an output example of character information in the embodiment.

[Explanation of symbols]

1 ... utterance input means, 2 ... character generation means, 3-1 and 3-2 ...
Extraction means, 4 ... Distance extraction means, 5 ... Visibility extraction means, 6 ...
Presentation position determining means 7, 7 output means.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Satoshi Ishibashi 2-3-1 Otemachi, Chiyoda-ku, Tokyo F-term in Nippon Telegraph and Telephone Corporation (reference) 5B050 BA09 BA20 CA07 EA07 EA20 EA28 FA02 FA10

Claims (4)

[Claims] 1. An immersive virtual communication environment in which a plurality of display devices are arranged to surround a user,
An utterance input process of inputting the utterance content of the utterance user, a character generation process of generating character information from the input utterance content, and an extraction of extracting a three-dimensional position and a line-of-sight vector of the utterance user and the reception user in the virtual environment A distance extracting step of extracting a distance between the speaking user and the receiving user based on a three-dimensional position of the speaking user and the receiving user; and extracting a field of view of the receiving user from a line-of-sight vector of the receiving user and a predetermined viewing angle. View extraction process,
A presentation position determining step of determining a presentation position of character information of the speaking user based on a distance between the speaking user and the receiving user, a field of view of the receiving user, a three-dimensional position of the speaking user, and a line-of-sight vector of the speaking user; Outputting the selected character information based on the determined presentation position. 2. The method according to claim 1, wherein the presenting position determining step includes: starting from the utterance when the distance between the uttering user and the receiving user is within a predetermined distance and the three-dimensional position of the uttering user is within the field of view of the receiving user. A process in which the position around the utterance user is set as the presentation position of the character display image for a certain period of time, and in other cases, the position that moves from the ambience of the utterance user in the direction of the line of sight with the time from the utterance is set as the presentation position. 2. The conversation information presentation method according to claim 1, wherein: 3. An immersive virtual communication environment system in which a plurality of display devices are arranged so as to surround a user, an utterance input means for inputting utterance content of an uttering user, and character information from the input utterance content. Character extracting means for extracting the three-dimensional positions of the speaking user and the receiving user and the line-of-sight vector, and a distance for extracting the distance between the speaking user and the receiving user based on the three-dimensional positions of the speaking user and the receiving user. Extracting means, view extracting means for extracting the field of view of the receiving user based on the line-of-sight vector of the receiving user and a predetermined field angle, and the distance between the speaking user and the receiving user, the field of view of the receiving user and the speaking user. A presentation position determining means for determining a presentation position of the character information of the speaking user based on the three-dimensional position and the line-of-sight vector of the speaking user; Output means for outputting the generated character information on the basis of the determined presentation position. 4. The presenting position determining means, if the distance between the previous uttering user and the receiving user is within a preset distance and the three-dimensional position of the uttering user is within the field of view of the receiving user, start the utterance. A means for presenting the character display image around the uttering user for a certain period of time, and otherwise, a position moving from the surrounding of the uttering user in the direction of the line of sight with the time from the utterance to the presenting position. The immersive virtual communication environment system according to claim 3, wherein: