JP3452348B2 - Speaker identification method in virtual space and recording medium storing the program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional virtual space or the like constructed on a computer (computer) by using computer graphic technology via a network.
Communicate using voice, facial images, texts, and complex communication means that express yourself,
In a system where multiple terminal users in remote areas can communicate at the same time in the same virtual space,
The present invention relates to a technique for achieving smooth communication by visually identifying a speaker.

[0002]

2. Description of the Related Art In a three-dimensional virtual space constructed on a computer via a network, one's own alter ego can be used for a walk, or voice can be heard in real time with another person who is also in the virtual space. Conventionally, there are several systems that can communicate with each other through images. In those traditional systems,
As a method of identifying a speaker (a person who has a conversation), a method in which the mouth of the body changes dynamically according to the utterance, or a display portion of a three-dimensional virtual space on a computer is called a three-dimensional display portion. A method of combining the whole world with a two-dimensional display part viewed from above, displaying the utterance situation in the form of ripples diffused around the speaker around the two-dimensional display part, and the virtual space itself A speaker is created by creating a three-dimensional sound field by determining the speaker by dividing the function into areas such as areas, areas that only receive audio, areas that only send audio, and the distances and directions between the subjects. There is a method to identify the position of.

[0003]

The above-mentioned conventional technique has three problems.
When the number of participants in the dimensional virtual space is very small,
It was an effective means to identify the speaker.

However, for example, in the method of changing the mouth of the alter ego according to the utterance, when the number of participants increases and there are a plurality of speakers, even if the mouth of the alter ego is dynamically changed according to the utterance. , It is difficult for oneself to identify the alter ego that he / she is interested in because there are multiple objects with similar movement patterns. So you don't know who is talking to whom. Further, if the distance between the speaker and the interlocutor is a certain distance in the virtual space, it is not possible to immediately identify where the speaker is, it takes time to form a conversation group, or it becomes an obstacle. There was also.

On the other hand, since the number of alternation models is finite on the computer, the number of users who have the same alternation shape increases as the number of participants increases. Therefore, the participants are who the speaker is on the screen. It is difficult to visually identify it, and it is difficult to enter the conversation.

Further, in the method of controlling the voice output according to the distance from the sound source, it is easy to aurally identify the speaker in a virtual space in which there are a plurality of participants and a plurality of speakers.
Visual matching is not easy, and even if you want to talk face-to-face with the person, it takes time to identify the other person.

The object of the present invention is to make it difficult to form a conversation group and take time to enter a conversation when the distance between the alternations is a certain distance or more in the three-dimensional virtual space of the multiplayer type. It is to solve the problem that the location of the speaker, which causes the problem of being unable to do so when he / she wants to meet immediately, cannot be immediately identified.

Further, the object of the present invention is to make it difficult to identify an individual when there are a plurality of speakers and a plurality of participants when they are close to each other within a certain distance, and to know who the speaker is speaking to. It is difficult to solve the problem of not being able to immediately enter the conversation.

[0009]

The present invention distinguishes between the processing of a speaker for an alter ego within a predetermined distance from the speaker (interlocutor) and the processing of a speaker for an alter ego outside the distance. In other words, the size of the speaker's alter ego model appears to be dynamically changed by the interlocutor who is more than a certain distance away from the speaker, and the speaker can speak from the interlocutor who is within the certain distance. Make a dynamic three-dimensional ripple object visible. The ripple object is specified according to a specific name (hereinafter referred to as a user ID) indicating that the speaker is participating in the virtual space.

When the interlocutor's alter ego is within a predetermined distance from the speaker, a three-dimensional ripple object is set from the speaker's alter ego according to the speaker's volume and the distance between the speaker and the interlocutor. It is possible to identify the speaker by dynamically changing the display. At this time, the visible color of the ripple object is changed between the opposite one and the opposite one so that it is possible to determine whether the sight line of the utterance is backward or forward. In addition, since the ripple object is displayed within a certain distance according to the direction of the speaker, the direction of the speaker's line of sight is visually determined by the direction of the ripple object displayed by the speaker within the range. Can be specified. The color of the ripple object is determined according to the user ID of the speaker.

On the other hand, when the altercation of the interlocutor is located outside the predetermined distance from the speaker, the size of the model of the altercation who is the speaker is dynamically changed. At this time, when the speaker's line of sight is not within the line of sight of the speaker, the direction of the speaker is changed to face the speaker. As a result, it is possible to visually identify the location of the speaker in other alternations that are distant from the speaker.

In both of the above two processes, the dynamic change process of the speaker is performed on the terminal side based on the data distributed in common from the virtual space control server to each terminal.

[0013]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described in detail below with reference to the drawings. FIG. 1 shows a schematic block diagram of a system to which a speaker identification method in a virtual space of the present invention is applied. In the figure, a virtual space control server 31 and a plurality of terminals 21, 41, etc. are connected via a network 100. Here, for convenience, the terminal 21 is a terminal on the speaker side and the terminal 41 is a terminal on the interlocutor side. 2 to 4 are detailed configuration diagrams of the speaker-side terminal 21, the virtual space control server 31, and the interlocutor-side terminal 41. Of course, it goes without saying that each terminal actually has both the configurations of FIG. 2 and FIG.

FIG. 5 shows that, in the multi-person participation type three-dimensional virtual space to which the speaker identification method in the virtual space of the present invention is applied, the alternation of the speaker dynamically changes after the speaker utters. Then, the processing flow until the interlocutor can specify the speaker is collectively shown. Hereinafter, with reference to FIGS. 2 to 4, the process at the terminal 21 on the speaker side, the server 3
The processing in 1 and the processing in the terminal 41 on the interlocutor side will be described in order.

First, the processing in the terminal 21 on the speaker side will be described with reference to FIG. The speaker inputs his voice using the microphone 8. The voice data 11 is the voice control unit 5,
The data is converted into data that can be interpreted by the communication module 2 and sent to the communication module 2. The communication module 2 adds its own user ID to the voice data 11 and transmits the voice data 11 to the virtual space control server 31 via the network 100. Further, the position coordinate data 9 is sent to the communication module 2 through the virtual world generation unit 4 and the virtual world control unit 3 by the input from the mouse 7, and the virtual world control unit 3 informs other participants of his / her existence. Together with the participant data 13 and the face image data 12 of the speaker taken from the camera 6, the virtual space control server 3 via the network 100.
Sent to 1.

Next, referring to FIG. 3, the virtual space control server 31
The processing in step will be described. The participant information management unit 32 of the server 31 has a function of holding participant information in the virtual space and a management function thereof. This participant information management unit 32
Unless the participant information is registered in advance, participation in the world of the virtual space of this system is not allowed. If registered, the data demultiplexing device 37 is used to deliver the participant data 10 of the virtual world to each terminal. The movement information control unit 33 determines and distributes the position coordinate data 9 to be transmitted to each terminal based on the movement information of each branch sent from each terminal. The face image processing unit 34 determines an image to be transmitted from the face image data 12 sent from each terminal based on the position determined by the movement information control information unit 33, and using the data demultiplexing device 37, Deliver to each terminal. The space control unit 35 holds information for managing the size of one virtual world and a group of other virtual space control servers used in that world. The voice mixer unit 36 performs a mixing process based on the voice data 11 sent from each terminal, and the information that specifies the utterers in the voice data 11 for all the participants in the virtual world as a result of the mixing process. Is added, and is transmitted to each terminal via the network 100 using the data demultiplexing device 37.

Finally, the processing at the terminal 41 on the interlocutor side will be described with reference to FIG. The interlocutor's terminal 41 receives the position coordinate data 9, the participant data 10 in the virtual space, and the voice data 11 of the speaker from the virtual space control server 31 via the network 100. The face image data 12 of the user is always transmitted to the virtual space control server 31, and the face image data 12 of another subject within a certain range is received. These data are processed by the communication module 2 and changed into a form that can be interpreted by the virtual world control unit 3. The result processed here is sent to the virtual world control unit 3, the contents of each sent data are interpreted here, and the virtual world generation unit 4
Is converted into data to be displayed on the monitor 8. As a result, a dynamic change of the speaker's alter ego is displayed on the monitor 14 viewed by a person who is physically separated. Further, the three-dimensional content data 13 displayed on those screens is stored in advance in the storage medium of the computer, so that the structure is read out from there.

Since the voice data 11 of the speaker sent by the mixing process from the virtual space control server 31 includes the voice data and the user ID, the communication module 2 separates the voice data and the user ID data. The voice data is sent to the voice control unit 5, and the voice of the speaker is output from the speaker 15. Further, the voice data separated at the same time is also sent to the virtual world control unit 3, and the voice amount is calculated there. Participant information is also sent from the communication module 2 to the virtual world control unit 3, and based on the distributed position coordinate data 9 of all the participants and the user ID of the sender of the sound in the virtual world control unit 3. The distance and direction to the speaker are calculated, and the display form change command 16 of the speaker is created from the calculated amount of voice, and is sent to the virtual world generation unit 4, and is displayed on the monitor 14 as the body of the speaker. Dynamic changes to the 3D ripple model are displayed. Further, by supplying a timing signal from the virtual world control unit 3 to the voice control unit 5, the display and the voice can be synchronized.

Next, a mechanism in which the size of the speaker's alter ego dynamically changes and the orientation also changes to the direction of the speaker will be described. The transmitted voice data 11 is divided into the voice data itself and the user ID by the communication module 2, the voice data itself is sent to the voice control unit 5, and the speaker 15 outputs the voice of the speaker. The voice data itself is also sent to the virtual world control unit 3, the amount of voice is calculated there, and is sent to the virtual world control unit 3 together with the user ID added to it, and utterance is made based on the position coordinates of the speaker. The person's alter ego and the display form change command 16 to the person are sent to the virtual world generation unit 4, and the person on the monitor 14 turns to the direction of the speaker's model, and the model of the speaker responds to the voice volume. It can be confirmed that it is dynamically changed. At this time, the position coordinate data 9 is sent to the virtual world control unit 3 via the communication module 2 to calculate the distance between the user and the speaker and the direction of the line of sight, and to change the line of sight of the user 1
Then, it is sent to the virtual world generation unit 4 as 6 and comes to face the speaker.

FIG. 6 shows an overall processing flow regarding the speaker identification in the virtual world control unit 3 of the terminal 41 on the side of the interlocutor. Further, FIGS. 7 to 10 are detailed flows of the process A, the process B, the process C, and the process D in FIG. Here, the process A (step 650) and the process C (step 680) are performed within the predetermined distance from the speaker's alter ego in the three-dimensional virtual space (this area is called a ripple display area). Processing in the case where the interlocutor's alter ego is outside the predetermined distance from the speaker's alter ego (this area is referred to as the alternator size area), processing B (step 660) and processing D (step 690) Is.

7 to 10, process A and process C, and process B and process D are slightly different depending on whether one or more speakers are present, and most of the processes are the same. Therefore, in the following, processing A and processing B will be taken as an example in the case where there is one speaker.
Will be specifically described.

FIG. 11 is a diagram for explaining the conditions under which the process branches into process A and process B in FIG. Figure 11
The area surrounded by a square is a virtual space on your computer.
Represents a delimited virtual world. This is a world where participants in the virtual space can see through the monitor 8. The difference in the dynamic change process of the speaker, the alter ego, is determined by the distance between the speaker's alter ego a51 and the interlocutor's alter ego b52.
The range is a radius r5 centered on the speaker's alter ego a51.
It is composed of a ripple display area 56 surrounded by a circle of 5 and an alternation size area 57 which is an area outside the circle. Radius r is
Depending on the hardware configuration of the system, it can be arbitrarily set in advance by the system administrator side, as in the range of displaying a face image described later.

First, the process (process A) for the altercation of the speaker when the interlocutor is in the ripple display area 56 will be described in detail with reference to FIGS.

FIG. 12 shows a situation before the utterer's alter ego a51 and the interlocutor's alter ego b52 in the ripple display area 56 speak. FIG. 13 shows a situation in which the speaker's alter ego a51 speaks and the interlocutor sees the ripples 71 to 73 that are leaving. At this time, the size of these ripple models is the radius r55 shown in FIG.
51 and the distance b52 of the interlocutor's alternation b52, and changes according to the ratio of the distance | V _ab | 54.
It is dynamically displayed one after another in the order of 2,73. At this time, the model data representing the ripples is stored in advance in the storage medium of the computer as the content data 13.

When the interlocutor's terminal 41 receives the position coordinate data 9, the participant data 10, and the voice data 11 for all the persons including the speaker from the virtual space control server 31 (step 610), the virtual world control unit 3 Calculates the distance and direction to the speaker and the amount of voice (step 620), and recognizes that the user is in the ripple display area (step 640: YE
S), the process A (FIG. 7) of step 650 for dynamically displaying the three-dimensional ripple model for the altercation of the speaker is entered.

FIG. 14 shows that the interlocutor displays the ripples from the speaker within a range of a certain angle with respect to the line-of-sight direction of the speaker who is the speaker. Left and right α / 2 degrees with respect to the line-of-sight direction 81 of the speaker's alter ego 51 (total α
The ripples 71, 72, 73 are sequentially displayed one by one in the direction of (degree). This is repeated during speech. At this time, the angle α can be arbitrarily determined by the side managing the virtual space control server according to the hardware configuration of the system. At this time, as shown in FIG. 14, the ripple model can be confirmed even from the altercation of the interlocutor who is not facing the speaker if the angle model is within the range of the angle α.

FIG. 15 shows that the colors of ripples are different between the altercation of the interlocutor facing the speaker's line of sight as shown in FIG. 14 and the altercation of the interlocutor on the opposite side. For example, the color of the back side of the ripples is common to all the alter-egoes. This allows the interlocutor to immediately recognize that he is on the opposite side of the speaker's line of sight.

FIG. 16 shows how to determine the ripple color. The code of the character string of the user ID of the speaker is extracted, and the virtual world control unit 5 of the terminal 41 uses the hash function to obtain the hash value based on the character code. With this value, a 16-bit color RGB pattern (65336 colors)
Decide one. The reason why the hash value is used is that the hash value rarely collides with the user ID, that the color range is wide, and that the color is appropriately scattered for each user ID.

FIG. 17 shows that the size of the three-dimensional model of ripples is changed according to the distance | V _ab | 54 with the speaker.

FIG. 18 shows that the number of ripples is displayed differently according to the voice volume of the speaker. That is, the virtual world control unit 4 interprets the size of the received voice data 14 and the ripples 71 appear to blink on the monitor 8 of the interlocutor when the volume 121 of the speaker is low. When the speaker's voice volume is 122, the ripple is 7
It appears to blink repeatedly in the order of 1, 72, and when the volume of the speaker is high 123, ripples 71, 72, 7
It can be seen that it blinks repeatedly in the order of 3. Thereby, the attenuation of the voice volume of the speaker is displayed three-dimensionally.

Next, referring to FIGS. 19 to 22, a detailed description will be given of the speaker's altercation and the process B (FIG. 8) for the speaker when the interlocutor is in the alternation size area 57 outside the ripple display area 56. .

FIG. 19 shows the situation before the interlocutor's alter ego c53 in the alternation size area 57 outside the ripple display area 56 created by the speaker's alter ego 51 hears the voice of the speaker. FIG. 20 shows the altercation c of the interlocutor in the alternation size area 57 outside the ripple display area 56 created by the speaker's alter ego 51.
Reference numeral 53 indicates a situation in which the voice of the speaker is being heard.

From the position coordinate data 9 received from the virtual space control server 31, the virtual world control section 3 calculates the distance and direction to the speaker, and the virtual world control section 3 calculates the voice volume from the voice data 11. A display form change command 16 for changing the size of the speaker and the orientation of the speaker is sent to the virtual world generation unit 4. In this way, the interlocutor is
20, the monitor 8 can confirm that the size of the speaker's alter ego is dynamically changed to 141, 142, 143 in order according to the voice volume. Thus, even if the uttered alter ego is far from the interacting alter ego, it is possible to immediately identify where the speaker is in the virtual space.

FIG. 21 shows that even if the line-of-sight of the interlocutor is not within the range in which the speaker can be confirmed, when the speaker's speech starts, the line-of-sight rotates so that the speaker can be seen within the range of his own line of sight. From this, the position of the speaker's alter ego is immediately specified.

FIG. 22 shows that the size of the speaker's alter ego a51 dynamically changes according to the voice volume of the speaker. When the sound volume 161 is small, it can be seen that it changes dynamically as 141, 142, 141, 142, ... Seen from the interlocutor's terminal. 162 in volume, 1 in volume
The same applies to the case of 63.

Although one embodiment of the present invention has been described above, the processing flow as shown in FIGS. 6 to 10 is described in a computer-executable format, and as a speaker identification program in the virtual space, It can be provided by recording it on a computer-readable recording medium such as a floppy disk (FD) or a compact disk (CD-ROM).

[0037]

As described above, according to the present invention,
By dynamically changing the display form according to the distance and direction from the speaker, other participants in the virtual space can identify where in the space the speaker is. In addition, the number of ripples and the number of alternations of the speaker change depending on the volume of the speaker.
Further, the color is displayed differently for each speaker as a ripple object. By these things, when the virtual space becomes large, it is possible to characterize where the ghosts are even if they are far apart, and even if there are many participants with the same shape and multiple participants, each movement It can be characterized for each speaker. Therefore, even in a virtual space where there are a large number of participants, it is easy to identify the speaker, it is possible to smoothly participate in the conversation, and it helps to form a conversation group.

[Brief description of drawings]

FIG. 1 shows a schematic block diagram of a system to which the present invention is applied.

FIG. 2 is a block diagram showing a configuration example of a terminal on the speaker side.

FIG. 3 is a block diagram showing a configuration example of a virtual space control server.

FIG. 4 is a block diagram showing a configuration example of a terminal on the interlocutor side.

FIG. 5 is a diagram showing an outline of processing from a speaker uttering to dynamically changing the alternation on another terminal according to the present invention.

FIG. 6 is a diagram showing an overall processing flow for identifying a speaker on a terminal on the side of an interlocutor.

FIG. 7 is a diagram showing a detailed processing flow of processing A in FIG. 6;

8 is a diagram showing a detailed processing flow of processing B in FIG.

9 is a diagram showing a detailed processing flow of processing C in FIG.

FIG. 10 is a diagram showing a detailed processing flow of processing D in FIG. 6;

FIG. 11 is a diagram showing a ripple display area and a separate size area in a virtual space.

FIG. 12 is a diagram showing a situation before the utterance of the speaker, in which the interlocutor's alter ego exists in the ripple display area.

FIG. 13 is a diagram showing a situation during utterance.

FIG. 14 is a diagram showing the relationship between the ripples of the speaker and the line of sight of the speaker.

FIG. 15 is a diagram showing that the ripples of the speaker are different on the facing / opposite sides of the speaker and the interlocutor.

FIG. 16 is a diagram illustrating that a ripple color is specified by a user ID.

FIG. 17 is a diagram showing that the size of a ripple changes depending on the distance between the interlocutor and the speaker.

FIG. 18 is a diagram showing that the size of a ripple changes depending on the volume of a speaker.

FIG. 19 shows the altercation of the interlocutor outside the ripple display area,
It is a figure which shows the condition before a speaker's speech.

FIG. 20 is a diagram showing a situation during utterance.

FIG. 21 is a diagram showing that the interlocutor rotates within a range in which the speaker can be confirmed.

FIG. 22 is a diagram showing that the size of a speaker's alter ego changes according to the speaker's volume.

[Explanation of symbols]

100 network 21 speaker side terminal 31 virtual space control server 41 interlocutor side terminal 51 speaker's alter ego 52-53 interlocutor's alter ego 54 distance between speaker and interlocutor in virtual space 55 speaker creates ripples Distance that can be displayed 56 Range in which the speaker can display ripples 57 Range in which the speaker dynamically changes the size of his or her own body 58 Virtual space 71 to 73 Three-dimensional ripple model 121 displayed by the body of the speaker 121 -123 Three-dimensional ripple model 141-144 corresponding to the voice volume of the speaker Speaker's alter ego 161 to 163 Speaker alternation model corresponding to the voice volume of the speaker

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP-A-10-207684 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 15/00-17/40 G06F 13 / 00

Claims (6)

(57) [Claims] 1. A plurality of terminals are connected via a network, and at least voice data of a speaker, position data of oneself or other participants, participant data are transmitted and received, and in a virtual space displayed on the terminal, A method for visually identifying a speaker's alter ego in a virtual space in a system in which participants communicate with each other by using voice and the expression of the participant's alter ego, and the distance between the speaker's alter ego and the interlocutor's alter ego Is calculated, it is determined whether the interlocutor's alter ego is within a predetermined distance from the speaker's alter ego, and if the interlocutor's alter ego is within a predetermined distance from the speaker's alter ego,
If a dynamic ripple-like object is sent out from the speaker's alter ego and displayed, and if the speaker's alter ego is outside a predetermined distance from the speaker's alter ego,
A method for identifying a speaker in a virtual space, which comprises rotating a dialogue person's alternation in a visual direction of the speaker's alternation and dynamically changing the size of the speaker's alter ego. 2. The speaker identifying method in the virtual space according to claim 1, wherein the color of the rippled object is changed in a direction in which the speaker's alter ego and the interlocutor's alter ego face each other. Speaker identification method. 3. The speaker identifying method in the virtual space according to claim 1, wherein the color of the rippled object is set to a different color for each speaker. 4. The method for identifying a speaker in a virtual space according to claim 1, 2, or 3, wherein the size of the rippled object is changed according to the distance between the speaker's alter ego and the interlocutor's alter ego. A speaker identification method in a characteristic virtual space. 5. The method for identifying a speaker in a virtual space according to claim 1, 2, 3, or 4, wherein the number of repetitions of a rippled object or the number of repetitions of a speaker's alternation is changed according to the amount of voice of the speaker. A method for identifying a speaker in a virtual space characterized by: 6. A plurality of terminals are connected via a network, and at least voice data of a speaker, position data of oneself and other participants, participant data are transmitted and received, and in a virtual space displayed on the terminal, A computer-readable recording medium in which a program for visually identifying a speaker's alter ego in a virtual space is recorded in a system in which participants communicate with each other by voice and expression of the participant's alternation. Process of calculating the distance between the person's alter ego and the speaker's alter ego, the process of determining whether the interlocutor's alter ego is within a predetermined distance from the speaker's alter ego, and the interlocutor within the predetermined distance from the speaker's alter ego If you have
The process of sending and displaying a dynamic ripple object from the speaker's alter ego, and if the speaker's alter ego is outside the predetermined distance from the speaker's alter ego,
A recording medium characterized by: a process of rotating a dialogue person's alternation in a visual direction of the speaker's alternation and dynamically changing and displaying the alternation of the speaker's alternation.