JP2006094315A - Stereophonic reproduction system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable sound source direction in a virtual space to be discriminated more accurately (recognized). <P>SOLUTION: A client 201 has accepting means 231 and 232 for accepting a swing indication by swinging a user's head from side to side or up and down in a virtual space, a client transmission means 222 for transmitting the swing indication accepted by the accepting means to an acoustic server 120, a client reception means 215 for receiving from the acoustic server 120 a stereophonic sound whose acoustic effect of sound sources is controlled, and an output means 217 for outputting the stereophonic sound received by the client reception means. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for reproducing sound three-dimensionally according to the position of a sound source in a virtual space.

  As a first background technology related to the present invention, there is a stereophonic sound generation technology based on digital signal processing (hereinafter, “three-dimensional audio technology”). In the 3D audio technology, a signal for multi-channel stereo reproduction using a plurality of speakers or binaural reproduction using stereo headphones is generated using a digital signal processing technology (for example, Non-Patent Document 1). reference).

  Among 3D audio technologies, there are technologies based on digital signal processing using the head-related transfer function (HRTF) or head-related impulse response (HRIR). HRTF expresses changes in sound due to ear shells and shoulders around the human head in the form of a transfer function (frequency response). HRIR expresses changes in sound due to the ear shells and shoulders around the human head in the form of impulse responses. HRTF is the result of Fourier transform of HRIR. Techniques using HRTF or HRIR allow listeners to identify the direction of the sound source (ie, left / right, front / back, top / bottom) in the playback sound.

  As a second background art related to the present invention, there is a conference system using a virtual space (see, for example, Patent Document 1, Patent Document 2, and Non-Patent Document 2). A conference system is a system in which a plurality of users share a virtual space and users in the same space can have a conversation.

US 6,408,327 B1 US 5,889,843 Begault, DR, -D Sound for Virtual Reality and Multimedia NASA / TM-2000-XXXX, NASA Ames Research Center, April 2000, http://human-factors.arc.nasa.gov/ihh/spatial/papers/pdfs_db/ Begault_2000_3d_Sound_Multimedia.pdf Singer, A. Hindus, D .; Stifelman, L .; , And White, S. "Tangible Progress: Less Is More In Somewire Audio Spaces", ACM CHI '99 (Conference on Human Factors in Computing Systems), pp. 104-112, May 1999.

  Now, when listening to reproduced sound output from a speaker or headphones using 3D audio technology, it is not easy for a person (listener) to accurately determine (recognize) the direction of the sound source.

  The first reason is the essential problem that people have only two left and right ears. In real space, a person identifies the direction of a sound source based on a volume difference and time difference between both ears, a change in frequency characteristics, and the like. However, since there are two ears on the left and right, a person originally has a low ability to discriminate front and rear or up and down directions. For this reason, in general, people often mistakenly determine the sound that can be heard directly in real space in the front-rear or up-down direction.

  The second reason is that the HRTF (or HRIR) that differs from person to person is not accurately reflected in the reproduced sound using the three-dimensional audio technology. That is, data used for HRTF or the like is measured using a standard human head shape and a dummy head having an ear shell. However, since the shape of the human head and the ear shell vary from person to person, listeners who are different from the head shape and ear shell of the dummy head, etc., accurately determine the direction of the sound source from the reproduced sound using 3D audio technology. It is not easy to distinguish (recognize).

  The present invention has been made in view of the above circumstances, and an object of the present invention is to enable more accurate determination (recognition) of the direction of a sound source in a virtual space.

  In order to solve the above-described problem, in the present invention, a swing instruction for swinging the user's neck left and right or up and down is received in a virtual space, and the sound of the sound source is controlled while the head is swung.

  For example, it is a stereophonic sound reproduction system that controls the sound of at least one sound source that exists in a virtual space, and includes a sound server that controls the sound effect of each sound source and a client used by the user. The client includes a reception unit that receives a swing instruction for swinging the user's neck left and right or up and down in the virtual space, a client transmission unit that transmits the swing instruction received by the reception unit to the acoustic server, and a sound effect of each sound source. Client receiving means for receiving the controlled stereo sound from the sound server, and output means for outputting the stereo sound received by the client receiving means.

  The acoustic server includes server storage means for storing positions and orientations of the user and the sound source in the virtual space, server reception means for receiving sound output from the sound source from each sound source, and a user stored in the server storage means, and Based on the position and orientation of each sound source, acoustic control means for controlling the acoustic effect applied to each of the sounds received by the server receiving means, and server transmission for transmitting the stereophonic sound whose acoustic effect is controlled by the acoustic control means to the client And when the swing instruction is received from the client, the sound control means applies each received sound based on the orientation changed from the user orientation stored in the server storage means to the left or right or up and down. Control sound effects.

  According to the present invention, in the virtual space, the user virtually swings his / her neck horizontally and vertically. Thereby, based on the change of the sound heard from the sound source, the direction of the sound source can be grasped more accurately.

  Embodiments of the present invention will be described below.

  FIG. 1 shows a system configuration diagram of a three-dimensional sound reproduction system to which an embodiment of the present invention is applied. As illustrated, the system includes a plurality of clients 201, 202, and 203 used by each of a plurality of users, a room management server 110, an acoustic server 120, and a registration server 130. These apparatuses 201, 202, 203, 110, 120, and 130 are connected via a network 101 such as the Internet.

  The room management server 110 manages the virtual space and the presence of users (location information, etc.) existing in the virtual space, and performs session control. Presence is the virtual space itself and position information (presence) of each user in the virtual space. The acoustic server 120 three-dimensionally mixes audio signals input from the clients 201, 202, and 203 using a three-dimensional audio technology. The registration server 130 registers or authenticates each user.

  In the present embodiment, three clients are provided, but the number of clients is not limited to three, and may be two or four or more. In this embodiment, the network 101 is configured by a single domain, but a network is configured by a plurality of domains, and it is also possible to perform communication across a plurality of domains by combining the domains. In that case, there are a plurality of room management servers 110, acoustic servers 120, and registration servers 130.

  Next, the hardware configuration of the stereophonic sound reproduction system will be described.

  FIG. 2 shows the hardware configuration of each of the clients 201, 202, 203, the room management server 110, the acoustic server 120, and the registration server 130.

  Clients 201, 202, and 203 are a CPU 301 that processes and operates data according to a program, a memory 302 that can be directly read and written by the CPU 301, an external storage device 303 such as a hard disk, and a communication device that performs data communication with an external system. A general computer system having 304, an input device 305, and an output device 306 can be used. For example, a portable computer system such as a PDA (Personal Digital Assistant), a wearable computer, or a PC (Personal Computer). The input device 305 and the output device 306 will be described later with reference to FIG.

  The room management server 110, the acoustic server 120, and the registration server 130 include at least a CPU 301 that processes and calculates data according to a program, a memory 302 that can be directly read and written by the CPU 301, an external storage device 303 such as a hard disk, an external system, and data A general computer system having a communication device 304 for communication can be used. Specifically, a server, a host computer, and the like.

  The functions described below of each of the above-described devices are a predetermined program loaded or stored in the memory 302 (a client program in the case of the clients 201, 202, and 203, and a room management server program in the case of the room management server 110). In the case of the acoustic server 120, the acoustic server program, and in the case of the registration server 130, the registration server program) is executed by the CPU 301.

  Next, the input device 305, the output device 306, and the functional configuration of the client 201 will be described with reference to FIG. The clients 202 and 203 have the same configuration.

  The client 201 includes a microphone 211, a pointing device 230, a left / right swing button 231, and an up / down swing button 232 as the input device 305. The pointing device 230 is a device for a user to input movement information (position information and direction information) in his / her own virtual space. The left / right swing button 231 is an input device for instructing the user to swing his / her head left / right in the virtual space (that is, to rotate the head left / right). The up / down swing button 232 is an input device for instructing the user to swing his / her head up / down in the virtual space (that is, to rotate the head upward or downward).

  In addition, the client 201 includes a headphone 217 compatible with a three-dimensional audio technology and a display 220 as the output device 306.

  The functional configuration includes an audio encoder 212, an audio decoder 216, an audio communication unit 215, a graphics renderer 219, a spatial modeler 221, a presence provider 222, and a session control unit 223.

  The audio encoder 212 converts the sound (analog signal) input from the microphone 211 into an audio signal (digital signal) and outputs the audio signal (digital signal) to the audio communication unit 215.

  The audio communication unit 215 transmits and receives audio signals to and from the acoustic server 120 in real time. That is, the audio communication unit 215 transmits the audio signal of the own user converted by the audio encoder 212 to the acoustic server 120. Also, the audio communication unit 215 receives, from the acoustic server 120, the audio signal of the other user that has been three-dimensionalized by performing processing resulting from the attributes of the virtual space such as reverberation and filtering using a three-dimensional audio technique.

  The audio decoder 216 converts the three-dimensional audio signal input from the audio communication unit 215 into sound (analog signal) and outputs it to the headphones 217.

  The graphics renderer 219 performs processing resulting from the attributes of the virtual space, and generates virtual space image data to be output to the display. The space modeler 221 receives the movement information input from the pointing device 230 and calculates the presence such as the position and orientation of the own user in the virtual space. The space modeler 221 receives an instruction input from the left / right swing button 231 or the up / down swing button 232, and calculates the direction when the user swings his / her head in the virtual space.

  The presence provider 222 transmits and receives position information and direction information of each user in the virtual space to and from the room management server 110. In addition, the presence provider 222 transmits a change in the orientation of the own user due to the pressing of the left / right swing button 231 or the up / down swing button 232 to the acoustic server 120. As a protocol for transmitting and receiving such information, it is conceivable to use an extended specification of SIP (Session Initiation Protocol) that is being standardized in the Internet Engineering Task Force (IETF).

  The session control unit 223 controls a communication session with the room management server 110. As a protocol for such session control, it is conceivable to use SIP standardized in IETF document RFC3261.

  Here, the virtual space is a space created by a plurality of users for a meeting or a conversation, or a space created for listening to music or Internet broadcasting. The room management server 110 manages the virtual space. When the user enters a certain virtual space, the room management server 110 transmits the attribute of the virtual space and the position information and direction information of the other users existing in the virtual space. The space modeler 221 stores the transmitted information and the position information and orientation information of the user in the virtual space input from the pointing device 230 in the memory 302 or the external storage device 303.

  The attributes of the virtual space include, for example, the size of the space, the height of the ceiling, the reflectance / color / texture of the walls and ceiling, the reverberation characteristics, and the sound absorption rate by the air in the space. Of these, the reflectance of walls and ceilings, reverberation characteristics, sound absorption by air in the space are auditory attributes, and the color and texture of walls and ceilings are visual attributes, and the size of the space The height of the ceiling is an attribute related to both hearing and vision.

  Next, the operation of each function will be described in the order of presence, audio, and video.

  Regarding the presence, the pointing device 230 receives input of movement information (position information or direction information) from the own user, converts the information into a digital signal, and inputs the digital signal to the space modeler 221. The space modeler 221 receives an input from the pointing device 230 and changes the position and orientation of the own user in the virtual space. That is, the space modeler 221 changes the position and orientation of the own user in the virtual space based on the attribute of the virtual space held in the memory 302 or the external storage device 303.

  Then, the space modeler 221 transmits the location information and orientation information of the user's virtual space to the room management server 110 via the presence provider 222. In addition, the space modeler 221 receives the position information and direction information of the virtual space of other users from the room management server 110 via the presence provider 222. The space modeler 221 holds position information and direction information in the virtual space of the user using the client, and position information and direction information in the virtual space of other users.

  Further, the left / right swing button 231 inputs a left / right swing instruction to the space modeler 221 when detecting pressing of the button. When the space modeler 221 receives a left / right swing instruction, the space modeler 221 performs the following operation in about one second.

  First, the space modeler 221 calculates the direction (azimuth information) of the user when the user's neck in the virtual space is swung to the left by a predetermined angle (for example, about 10 °). Then, the space modeler 221 sends the calculated orientation of the user to the graphics renderer 219 and the presence provider 222. The presence provider 222 transmits the orientation information of the own user to the acoustic server 120.

  Then, the space modeler 221 calculates the direction (direction information) of the user when the user's neck in the virtual space is swung to the right by a predetermined angle (for example, about 10 °) from the current time. Then, the space modeler 221 sends the calculated orientation of the own user to the graphics renderer 219 and the acoustic server 120. The space modeler 221 transmits the orientation information of the own user to the acoustic server 120 via the presence provider 222.

  Hereinafter, the case where the left / right swing button 231 is pressed will be further described with reference to FIGS. 4, 5, and 6.

  FIG. 4 is a diagram schematically showing the own user and a sound source (for example, another user of the communication partner) in the virtual space. FIG. 4 shows the own user 1 and the sound source 2 showing the own user from directly above. The own user 1 has a nose 11 to indicate the direction. That is, the user 1 is facing in the direction 3 in which the nose 11 is added. In FIG. 4, in the initial state (the state before pressing the left / right swing button 231), the own user 1 is facing the front direction 3, and the sound source 2 is diagonally right front of the own user 1 (n ° In the direction of angle 4).

  Now, since there are only two human ears on the left and right, the ability to discriminate whether the sound source 2 is in front or behind is low. Therefore, even in real space, the user 1 may recognize that the direction of the sound source 2 is erroneously present behind. Such human ear characteristics are described, for example, in the following documents.

BCJ Moore, Takeshi Ogushi Translated: Introduction to Auditory Psychology, Seishin Shobo, 1994. P. 220-221. (Original: BCJ Moore: An Introduction to the Psychology of Hearing, 3rd Ed., Academic Press, 1989.)
In such a situation, the user 1 hears the stereophonic sound (reproduced sound) of the sound source 2 reproduced by the three-dimensional audio technology from the headphones, and the sound source 2 exists diagonally right front or diagonally right rear. In some cases, it is difficult to determine whether it exists in 2a (if it is ambiguous). In this case, the user 1 swings his / her head to the left and right in the same manner as when the direction of the sound source is generally confirmed in real space. That is, in order to accurately determine (recognize) the direction of the sound source 2, the left / right swing button 231 is pressed. It should be noted that the sound source 2a that is easily misrecognized is present at a position symmetrical to the front and rear of the actual sound source 2 with respect to the plane 5 connecting the left and right ears of the user 1.

  FIG. 5 is a diagram schematically showing a state in which the space modeler 221 has swung his / her own user's neck to the left when the left / right swing button 231 is pressed. That is, the space modeler 221 changes the orientation of the user 1 to the left by a predetermined angle (α °). In this state, the sound source 2 moves further to the right (that is, the direction 4L of n + α °) from the initial state. Therefore, the user 1 can hear the three-dimensional sound (reproduced sound) of the sound source 2 reproduced by the three-dimensional audio technology by moving to the right from the initial state shown in FIG. When the sound source is present at the position of the sound source 2a that is easily misrecognized, the user 1 moves the stereophonic sound of the sound source 2 ′ reproduced by the three-dimensional audio technology to the left from the initial state shown in FIG. I hear it.

  FIG. 6 is a diagram schematically illustrating a state in which the space modeler 221 has swung the user's 1 neck to the right. That is, the space modeler 221 changes the orientation of the user 1 to the right by a predetermined angle (α °). In this state, the sound source 2 moves to the left (that is, the n-α ° direction 4R) from the initial state. Therefore, the user 1 can hear the three-dimensional sound (reproduced sound) of the sound source 2 reproduced by the three-dimensional audio technology by moving to the left from the initial state shown in FIG. When a sound source exists at the position of the sound source 2a that is easily misrecognized, the user 1 can hear the three-dimensional sound of the sound source 2a reproduced by the three-dimensional audio technology moving to the right from the initial state shown in FIG. .

  In this way, by using the left / right swing button 231, the user can grasp the accurate direction of the sound source whose direction is ambiguous. That is, when it is ambiguous whether the sound source is in front or behind, the user presses the left / right swing button 231. The user can correctly determine that the sound source is ahead if the sound source moves to the right and then to the left and is heard. On the other hand, the user can correctly determine that the sound source is behind when the sound source is heard moving first to the left and then to the right.

  Note that a discontinuous operation in which the user's orientation changes to the left (or right) by a predetermined angle at a time may lead to user confusion. For this reason, the spatial modeler 221 does not send the azimuth information when the predetermined angle is swung to the left and right at the same time to the graphics renderer 219 and the acoustic server 120, but to interpolate the azimuth information for each angle at a certain interval. It is sent to the graphics renderer 219 and the sound server 120. Thereby, it becomes a substantially continuous operation | movement like the operation | movement when a user shakes his head in real space, and can prevent a user's confusion.

  There are individual differences in the order of shaking the head and the angle of shaking the head. For this reason, it is assumed that a predetermined angle for first swinging to the left or to the right or swinging the neck can be adjusted (changed) for each user.

  Next, a case where the up / down swing button 232 is pressed will be described. The up / down swing button 232 inputs an up / down swing instruction to the space modeler 221 when detecting pressing of the button. When the space modeler 221 receives the up / down swing instruction, the space modeler 221 performs the following operation in about one second.

  First, the space modeler 221 calculates the orientation of the user (vertical orientation information) when the user's neck in the virtual space is shaken from a current position (horizontal state) to a predetermined angle (for example, about 10 °). To do. Then, the space modeler 221 sends the calculated orientation of the own user to the graphics renderer 219 and the acoustic server 120.

  Then, the space modeler 221 calculates the orientation of the user (vertical orientation information) when the user's neck in the virtual space is shaken down from the current time (horizontal state) by a predetermined angle (for example, about 10 °). To do. Then, the space modeler 221 sends the calculated user orientation to the graphics renderer 219 and the acoustic server 120. Hereinafter, the case where the up / down swing button 232 is pressed will be further described with reference to FIG.

  FIG. 7 is a diagram schematically showing the user and the sound source in the virtual space. In FIG. 7, the own user 1 and the sound source 2 showing the own user from the side are shown. The own user 1 has a nose 11 to indicate the direction. In the initial state (before pressing the up / down swing button 232), the own user 1 is directed in the horizontal direction 3, and the sound source 2 is present diagonally upward and forward of the own user 1 (upward direction 4 of n °). is doing.

  Now, since there are only two human ears on the left and right, the ability to discriminate whether the sound source 2 exists above or below is low, as in the previous and subsequent discrimination. Such characteristics of the human ear are described in the above-mentioned document (Introduction to Auditory Psychology).

  In such a situation, the user 1 hears the stereophonic sound (reproduced sound) of the sound source 2 reproduced by the three-dimensional audio technology from the headphones, and the sound source 2 exists in the front upper direction or the front lower direction. In some cases, it is difficult to determine whether it exists. In this case, in order to accurately determine (recognize) the direction of the sound source, the own user presses the up / down swing button 232 as in the case of confirming the direction of the sound source in the real space. Note that the sound source 2a that is easily misrecognized is present at a position that is vertically symmetrical with the actual sound source 2 with respect to the plane 5 connecting the left and right ears of the user 1.

  When the up / down swing button 232 is pressed, the space modeler 221 first changes the orientation of the user 1 to a predetermined angle (β °). That is, the space modeler 221 swings the neck of the own user 1 upward 3U. In this state, the sound source 2 is positioned below the initial state (that is, in the direction 4U of n ° −β °). Accordingly, the user 1 can hear the three-dimensional sound (reproduced sound) of the sound source 2 reproduced by the three-dimensional audio technology, moving downward from the initial horizontal state. When an actual sound source is present at the position of the sound source 2a that is easily misrecognized, the user 1 can hear the three-dimensional sound of the sound source 2a reproduced by the three-dimensional audio technology moving upward from the initial state.

  Next, the space modeler 221 changes the orientation of the user 1 below a predetermined angle (β °). That is, the space modeler 221 swings the neck of the user 1 in the downward direction 3D. In this state, the sound source 2 is positioned above the initial state (that is, the direction 4D of n ° + β °). Accordingly, the user 1 can hear the three-dimensional sound of the sound source 2 reproduced by the three-dimensional audio technology moving upward from the initial horizontal state. When an actual sound source exists at the position of the sound source 2a that is easily misrecognized, the user 1 can hear the three-dimensional sound of the sound source 4 reproduced by the three-dimensional audio technology moving upward from the initial state.

  Thus, by using the up / down swing button 232, the user can grasp the accurate direction of the sound source whose direction is ambiguous. That is, when it is ambiguous whether the sound source is above or below, the user presses the up / down swing button 232. The user can correctly determine that the sound source is at the top if the sound source is heard to move down and then up. On the other hand, if the user hears the sound source moving up first and then down, the user can correctly determine that the sound source is down.

  Note that a discontinuous operation in which the user's orientation changes at a predetermined angle (or below) at a time may lead to user confusion. Therefore, the space modeler 221 sends the azimuth information interpolated for each angle at a constant interval to the graphics renderer 219 and the acoustic server 120, as with the left and right swing buttons 231. Thereby, it becomes a substantially continuous operation | movement like the operation | movement when a user shakes his head in real space, and can prevent a user's confusion. In addition, the order of shaking the head and the angle of shaking the head have individual differences and can be adjusted (changed) for each user.

  In addition, the swinging motion by the left / right swing button 231 and the up / down swing button 232 described above can be input by using the pointing device 230. However, by separately providing the left / right swing button 230 and the up / down swing button 232, the user can input a swing instruction easily and accurately.

  Next, audio will be described.

  As for the voice, the microphone 211 collects the voice of the user using the client and sends it to the audio encoder 212. Then, the audio encoder 212 converts the voice of the own user into an audio signal (digital signal) and outputs the audio signal to the audio communication unit 215. The audio communication unit 215 transmits the audio signal of the own user input from the audio encoder 212 to the acoustic server 120 in real time.

  In addition, the audio communication unit 215 receives, in real time, an audio signal (stereo sound) of another user of another client that is three-dimensionalized using the three-dimensional audio technology from the sound server 120 and outputs the audio signal to the audio decoder 216. The audio decoder 216 outputs the audio signal (stereo sound) received from the sound server 120 to the headphones 217. The sound three-dimensional processing using the three-dimensional audio technology of the audio signal performed by the sound server 120 will be described later.

  For real-time communication of audio signals, RTP (Real-time Transport Protocol), which is a protocol described in document RFC 3550 issued by IETF (Internet Engineering Task Force), is used.

  Next, the image will be described.

  As for the image, how the graphics renderer 219 performs in the virtual space based on the visual virtual space attribute held by the space modeler 221, the position of the other user (communication partner) in the virtual space, and the position of the own user. Calculate (coordinate conversion) whether the user can see. Next, the graphics renderer 219 performs processing that results from the attribute of the virtual space from the viewpoint of the user's position by the above calculation on the image of another user determined in advance, and outputs the image data on the screen Create (video).

  The video generated by the graphics renderer 219 is reproduced as a video from the viewpoint of the user using the client, and is output to the display 220. The user refers to the video output to the display 220 as necessary.

  FIG. 8 is an example of a virtual space using a plan view. The display content shown in the figure is an example in which the own user who uses the client 201 shares a virtual space with the first and second other users who use the client 202 and the client 203. The virtual space shown in the figure is based on the attributes of the virtual space held by the space modeler 221 and the own user 411 arranged in the virtual space from directly above based on the position and orientation information of the own user and other users in the virtual space, A two-dimensional image obtained by viewing the first other user 412 and the second other user 413 is displayed. The illustrated user 411 and other users 412, 413 each have a head a, a nose b for indicating the direction in which each user faces in the virtual space, and a shoulder (shoulder width) c. .

  The graphics renderer 219 fixes the position and orientation of the user 411 and displays the virtual user and other users 412 and 413 in the virtual space relative to the user 411 so as to move and rotate. When the own user 411 moves or changes its direction using the pointing device 230, a screen in which the virtual space and other users in the virtual space are relatively moved and rotated is displayed on the display 220 in real time. By fixing the direction of the own user to the front, consistency between the voice and the graphics display is ensured, and the position and direction of the other user can be grasped as a physical sensation.

  In the illustrated virtual space, a scale bar 414 indicating a predetermined length (for example, 1 m) is displayed. In the illustrated virtual space, the shoulder (shoulder width) c of each user is displayed. By displaying the scale bar 414 and the shoulder (shoulder width) c, the user 411 can visually (intuitively) grasp the approximate distance of the other users 412 and 413 in the virtual space. Then, the user listens to the other user's stereophonic sound using the three-dimensional audio technology output from the headphones while referring to the image data (FIG. 8) on which the scale bar 414 and the shoulder (shoulder width) c are displayed.

  Note that the 3D audio technology includes a reverberation simulation technology that expresses the distance between the user and the sound source, the size of the room, and the like by simulating the reverberation of the room. The reverberation of a room is the sound (hereinafter referred to as “direct sound”) that is added to the sound of the sound source (hereinafter referred to as “direct sound”) due to the reflection and diffusion of the sound from the walls of the room (virtual space) where the sound source exists and objects in the room "Reflected sound"). In general, when determining (recognizing) a distance to a sound source, a human determines based on a ratio between a direct sound and reflected sound that is reverberation.

  Therefore, the user listens to the stereophonic sound of the other user, in which the direct sound and the reflected sound are mixed, while visually grasping the distance from the other user (sound source) by referring to the display. Thereby, the own user can learn how the ratio of the direct sound and the reflected sound is changed by changing the distance to the other user by moving in the virtual space and changing the distance. By repeating such learning, the user can know the approximate distance from other users without looking at the display.

  It is assumed that the predetermined length of the scale bar 414 can be changed (adjusted) according to the size of the virtual space or a user instruction. In the illustrated virtual space, both the scale bar 414 and the shoulder (shoulder width) c are displayed. However, only one of the scale bar 414 and the shoulder (shoulder width) c may be displayed.

  FIG. 9 is an example of a virtual space displayed when the left / right swing button 231 is pressed in the state of FIG. When the left / right swing button 231 is pressed, the user 411 first swings his / her head to the left. Therefore, in the plan view 9A of the virtual space displayed on the display, the other users 412 and 413 move to the right from the position displayed in FIG. 8 (that is, rotate to the right by a predetermined angle).

  Next, since the own user 411 swings his head to the right, in the plan view 9B of the virtual space displayed on the display, the other users 412 and 413 move to the left from the position displayed in FIG. Rotate to the left by an angle of. Then, the virtual space displayed on the display displays the original state shown in FIG. Note that the position and orientation of the user 411 are not different from the state of FIG. 8 even if the left / right swing button 231 is pressed.

  FIG. 10 is an example of the virtual space displayed when the up / down swing button 232 is pressed in the state of FIG. When the up / down swing button 231 is pressed, the user 411 first swings his / her head up. Therefore, in the plan view 10A of the virtual space displayed on the display, the upper side is deformed and displayed in a trapezoid smaller than the bottom side. Therefore, the other users 412, 413 existing in front of the own user 411 in the virtual space are displayed by a predetermined ratio smaller than the original size shown in FIG.

  Next, since the user 411 swings his / her head down, in the plan view 10A of the virtual space displayed on the display, the base is deformed and displayed in a trapezoid smaller than the top. Therefore, the other users 412 and 413 existing in front of the user 411 in the virtual space are displayed by a predetermined ratio larger than the original size. Then, the virtual space displayed on the display displays the original state shown in FIG. Note that the position and orientation of the user is not changed from the state shown in FIG. 8 even when the up / down swing button 232 is pressed.

  FIG. 11 is an example of a virtual space when the virtual space displayed in the plan view shown in FIG. 8 is rendered as a perspective view using a three-dimensional graphics technique. That is, the graphics renderer 219 includes the size of the space stored in the memory 302 or the external storage device 303, the attributes of the virtual space such as the wall and ceiling materials, the location information of the own user and other users in the virtual space, etc. A two-dimensional image is created from the three-dimensional data and displayed on the display 220. In the illustrated example, a two-dimensional image obtained by viewing a wall surface, a ceiling, a floor surface, and other users 412 and 413 arranged in the virtual space from a viewpoint determined from the position of the own user 411 in the virtual space is displayed. Yes.

  In FIG. 11, a mesh (for example, 1 m × 1 m) indicating a predetermined distance is displayed on the floor surface as the scale bar. As a result, the same effect as the scale bar described in FIG. 8 occurs. That is, the own user can learn how the ratio of the direct sound and the reflected sound changes due to a change in the distance to other users.

  Note that it is difficult for the own user to intuitively grasp the distance to other users who are far away only with a mesh indicating a predetermined distance (for example, 1 m × 1 m). Therefore, the mesh may be displayed with a thicker line for each distance larger than a predetermined distance (for example, 5 m × 5 m or 10 m × 10 m). Alternatively, the user may be made to grasp the distance by displaying only a part of the mesh or displaying only the intersection of the mesh. Moreover, in the top view shown in FIG. 8, although the scale bar 414 is displayed, it is good also as displaying a mesh.

  In the present embodiment, a flat display 220 is used. However, it is also possible to display the distance more directly without displaying a scale bar or mesh by using a head-mounted display or the like capable of stereo viewing.

  This is the end of the description of the client in FIG. Among the clients, the microphone 211, the pointing device 230, the left / right swing button 231, the up / down swing button 232, the headphones 217, and the display 220 are realized by hardware. The audio encoder 212, the audio decoder 216, and the graphics renderer 219 are realized by software, hardware, or a combination thereof. The audio communication unit 215, the space modeler 221 and the session control unit 223 are usually realized by software.

  Note that the client 201 may be a computer having a size and function similar to those of a PDA or handheld computer as shown in FIG. That is, the client main body 230 is connected to the display 220, the operation unit 240 for inputting the position and orientation of the user as the pointing device 230, the left / right swing button 231, the up / down swing button 232, and the network 101. An antenna 237.

The headset connected to the main body 230 includes a headphone 217 and a microphone 211. The headset shown is wired to the main body 230, but can be wirelessly connected by Bluetooth or IrDA (infrared rays). The client 201 may use a general personal computer.
Next, the acoustic server 120 will be described.

  The acoustic server 120 uses audio data transmitted from each of the audio communication units 215 of the client, the auditory virtual space attribute held by the spatial modeler of the acoustic server 120 using the three-dimensional audio technology, and the virtual space Based on the positions of the own user and other users (sound sources) existing in the above, how to hear the voices of other users in the virtual space is calculated.

  FIG. 16 is a configuration diagram of the acoustic server 120. As illustrated, the acoustic server 120 includes at least one audio receiving unit 121, an audio renderer 122, and an audio transmitting unit 123. That is, the acoustic server 120 includes these processing units 121 to 123 as many as the number of clients (that is, for each client). The acoustic server 120 may be realized by using a single program or device in a time-sharing manner, without having the audio reception unit 121, the audio renderer 122, and the audio transmission unit 123 as many as the number of clients. .

  The acoustic server 120 includes a space modeler 124 and a communication session 125. The space modeler 124 receives the position of each user in the virtual space and the attribute of the virtual space from the room management server 110, performs the same processing as the client space modeler 221 shown in FIG. Deploy. The session control unit 125 controls a communication session with the room management server 110 in the same manner as the session control unit 223 of the client shown in FIG.

Each audio receiving unit 121 associated with each client receives an audio signal (voice) from the audio communication unit 215 of each client. Then, each of the audio reception units 121 sends the signal data synchronized (corresponded) between the audio signals from all clients to each audio renderer 122 by buffering the received audio signal. This buffering (playout buffering) method is described, for example, in the following document.
Colin Perkins: RTP: Audio and Video for the Internet, Addison-Wesley Pub Co; 1st edition (June 11, 2003).
Each audio renderer 122 associated with each client three-dimensionalizes each audio signal (sound) input from each audio receiving unit 121 based on the position of each user in the virtual space arranged by the space modeler 124. . Then, the audio renderer 122 outputs the signal data (signal sequence) of two channels (left channel and right channel) corresponding to the client to the audio transmission unit 123 of the client. The audio transmission unit 144 associated with each client transmits 2-channel signal data to the corresponding client.

  Next, the audio renderer 122 will be specifically described.

  In 3D audio technology, it is mainly generated by HRIR (Head Related Impulse Response), which expresses how the sound changes around the human head (hereinafter “human head”) (impulse response), and a virtual environment such as a room. The direction and distance of the sound is expressed by the simulated reverberation. HRIR is determined by the distance between the sound source and the human head and the angle (horizontal angle and vertical angle) between the human head and the sound source. It is assumed that the memory 302 of the acoustic server 120 or the external storage device 303 stores HRIR values measured in advance for each distance and each angle using a dummy head (person's head). Also, by using different values for the left channel (measured with the left ear of the dummy head) and the right channel (measured with the right ear of the dummy head), the HRIR values Express a sense of direction in the front-rear or up-down direction.

  FIG. 14 is a diagram showing processing of the audio renderer 122.

  The audio renderer 122 performs the following calculation for each packet (usually every 20 ms) received by RTP (Real-time Transport Protocol) for each sound source (other users).

First, the audio renderer 122 inputs the sound source signal sequence s _i [t] (t = 1, ...) and coordinates (x _i , y _i, z _i ) in the virtual space of each sound source. Accept (S61). Note that the coordinates of each sound source in the virtual space are input from the space modeler 124. The space modeler 124 arranges each sound source (each user) in the virtual space, and then inputs the coordinates of each sound source to the audio renderer 122. In addition, the signal sequence of each sound source is input from each audio receiving unit 121.

  Then, the audio renderer 122 calculates the direct sound of the sound source and the reflected sound that is reverberation.

  For direct sound, the audio renderer 122 calculates the distance and angle (azimuth) between the user and the sound source for each sound source using the input coordinates (S62). Then, the audio renderer 122 identifies the HRIR corresponding to the distance and angle (azimuth) with the user from the HRIR values stored in advance in the memory 302 or the external storage device 303 (S63). The audio renderer 122 may use the HRIR value calculated by interpolating the HRIR value stored in the memory 302 or the like.

  The audio renderer 122 performs a convolution calculation using the signal sequence input in S61 and the HRIR for the left channel specified in S63, and generates a left channel signal (S64). The audio renderer 122 performs convolution calculation using the signal sequence input in S61 and the HRIR for the right channel specified in S63, and generates a right channel signal (S65).

  For the reflected sound, the audio renderer 122 calculates reverberation to be added using the input coordinates (S66, S67). That is, the audio renderer 122 calculates the reverberation based on the way of changing the sound (impulse response) according to the attribute of the virtual space. Hereinafter, calculation of reverberation will be described.

  The reverberation is composed of early reflection and late reverberation. And it is generally thought that the early reflection is more important than the late reverberation in the formation (recognition) of the sense regarding the distance to other users and the size of the room. In a room in real space, after hearing the sound directly emitted from the sound source (direct sound), several tens of initial reflections from walls, ceilings, floors, etc., depending on conditions, within a few ms to 100 ms. It is said that you can hear. If the shape of the room is a rectangular parallelepiped, there are only six initial reflections at a time. However, in a room with a more complicated shape or furniture, the number of reflected sounds increases, and sounds reflected multiple times by walls or the like can be heard.

  There is an image source method as a method of calculating the initial reflection, which is described in the following document, for example.

Allen, J .; B. and Berkley, A.A. “Image Method for Efficiently Simulating Small-Room Acoustics”, J. Am. Acoustical Society of America, Vol. 65, no. 4). , Pp. 943-950, April 1979.
In a simple image source method, the wall, ceiling, and floor of a room are regarded as mirror surfaces, and the reflected sound is calculated as sound from an image of a sound source on the opposite side of the mirror surface.

  FIG. 15 illustrates a two-dimensional image source method with the ceiling and floor omitted for the sake of simplicity. That is, there is an original sound chamber 1 in the center, and the sound chamber 1 includes a sound source and a user who is a listener. Around the sound chamber 1, twelve mirror images including the wall 2 of the room are drawn. Note that the number of mirror images is not necessarily 12 and can be increased or decreased.

The audio renderer 122 calculates the distance and direction from the image of each sound source to the listener, assuming that the sound from the position of the image of each sound source existing in each mirror image goes straight to the listener (own user). (S66). Since the sound intensity is inversely proportional to the distance, the audio renderer 122 attenuates each volume according to the distance. However, if the reflectance of the wall is α (0 ≦ α ≦ 1), the sound sample reflected n times by the wall is multiplied by α ⁿ to further attenuate the volume.

  Note that the value of the reflectance α is about 0.6. The reason why the value is set to about 0.6 is to obtain reverberation sufficient for the listener to recognize the distance to the sound source (that is, the ratio between the direct sound and the reflected sound). Another reason is that if the value of α is excessively large, the listener's sense of direction is disturbed.

  Then, the audio renderer 122 specifies the HRIR corresponding to the distance and angle (azimuth) with the user for each sound source image from the HRIR values stored in advance in the memory 302 or the external storage device 303. (S67). Since the reflected sounds reach the human head from different directions, it is necessary to apply an HRIR different from the HRIR of the direct sound specified in S63.

  If a convolution calculation (S67, S68), which will be described later, is performed on each of a large number of reflected sounds using different HRIRs, a huge amount of calculation is required. In order to prevent an increase in the amount of calculation, the HRIR when the sound source is in front may be applied to the calculation of the reflected sound regardless of the actual sound source direction. By calculating only the time difference (ITD, internal time difference) and intensity difference (IID) when the sound reaches the left and right ears, the calculation of HRIR can be replaced with a small amount of calculation.

  Then, the audio renderer 122 performs convolution calculation using the signal sequence input in S61 and the HRIR for the left channel specified in S67, and generates reverberation of the left channel signal (S68). . In addition, the audio renderer 122 performs convolution calculation using the signal sequence input in S61 and the HRIR for the right channel of the HRIR specified in S67, and generates the reverberation of the right channel signal (S69). .

  Then, the audio renderer 122 adds all the left channel signals from the respective sound sources (S70). Note that the left channel signal includes the direct sound calculated in S64 and the reflected sound calculated in S68.

  The audio renderer 122 adds all the right channel signals from the respective sound sources (S71). Note that the left channel signal includes the direct sound calculated in S65 and the reflected sound calculated in S69.

  The HRIR calculation (S63, S67) is performed for each packet as described above. However, in the convolution calculation (S64, S65, S68, S69), a portion to be carried over to the next packet is generated. Therefore, it is necessary to hold the specified HRIR or the input signal string until the next packet processing.

  In this way, the audio renderer 122 performs processing such as volume adjustment, superimposition of reverberation and reverberation, filtering, etc. on the voice of each user transmitted from the audio communication unit 215 of each client, The sound effect is controlled on the sound to be heard at the position in the virtual space of the own user. That is, the audio renderer 122 generates stereophonic sound in which the sound is localized by a process resulting from the relative position between the attribute of the virtual space and another user.

  Note that the audio renderer 122 provided for each client may perform processing resulting from virtual space attributes such as reverberation and filtering on the voice of the user using the client as necessary. The sound of the own user generated by the audio renderer 122 is output to the headphones 217, and the own user listens to the sound. When the user directly listens to the direct sound of the user's voice, the user may have a strange impression. In particular, if the delay is large, the user's utterance will be hindered. However, it is possible to listen to only the reflected sound (reverberation) with a delay in the range of several tens of ms without listening to the direct sound. As a result, the position of the user in the virtual space or the size of the virtual space can be recognized by the user as a physical sensation.

However, when the acoustic server 120 calculates reverberation as in the present embodiment, it takes 100 ms or more until the reflected sound calculated by the acoustic server 120 reaches the user after the user speaks. There are many cases. Since this delay is much larger than the delay time of the reflected sound in a normal room, it can cause the user's perception to be confused. In order to solve this problem, an audio renderer having the same function as the audio renderer 122 of the acoustic server 120 is mounted on the client. For the user's voice, the audio renderer installed in the client calculates the reverberation, reducing the delay, and realizing a reverberation with the same degree of delay as when the user is in the actual room. It is done. As described above, the acoustic server 120 performs reverberation calculation for the voices of other users.
Next, client processing will be described.

Hereinafter, client processing will be described in the order of network connection processing, entrance processing, exit processing, own user movement processing, other user movement processing, and swing processing.
The network connection process is a process procedure when connecting to the network 101, and is executed when the client is turned on. First, the session control unit 223 transmits a login message including user identification information and authentication information to the registration server 130. The registration server 130 authenticates the user identification information and authentication information, and sends a login message to the room management server 110. Then, the presence provider 222 receives the room list from the room management server 110 and displays it on the display 220.

  Note that it is possible to use a SIP (Session Initiation Protocol) REGISTER message for communication between the client and the registration server 130. For communication between the client and the room management server 110, an SIP INVITE message, BYE message, SUBSCRIBE message, and NOTIFY message can be used.

  The entrance process is a client process when the user selects a room to be entered from the room list displayed on the display 220. The presence provider 222 receives a room selection instruction input using the input device 305 and transmits an entrance message (enter) to the room management server 110. The admission message includes identification information of the own user and position information and direction information (hereinafter, “position information etc.”) in the virtual space of the own user. It is assumed that the position information of the own user is stored in advance in the memory 302 or the external storage device 303 (hereinafter referred to as “memory etc.”).

  The SIP INVITE message can also be used to send the admission message. The INVITE message declares the start of voice communication between the client and the room management server 110. When receiving the INVITE message, the room management server 110 instructs the acoustic server 120 to start voice communication between the client and the acoustic server 120. The usage of this INVITE message may be in accordance with IETF document RFC3261.

  The client can also send a SIP PUBLISH message to notify the user's location at the same time as sending the admission message. The PUBLISH message is a message for notifying an event (for example, movement of a user) occurring in the virtual space of the selected room even when there is no notification request.

  In addition, the client can transmit a SUBSCRIBE message in order to request the room management server 110 for event notification in the virtual space including a change in the position of another user. The SUBSCRIBE message is a message requesting notification of an event (for example, user movement) that has occurred in the virtual space of the selected room. When the room management server 110 receives the SUBSCRIBE message, when an event occurs in the virtual space, the room management server 110 notifies the client of the content of the event by a NOTIFY message. The NOTIFY message is a message for notifying an event (for example, a change in the user's position) occurring in the virtual space in accordance with a notification request.

  The usage of these PUBLISH messages, SUBSCRIBE messages, and NOTIFY messages is described in IETF document RFC 2543 (Roach, AB, “Session Initiation Protocol (SIP) —Specific Event Notification”) and the Internet draft “Session Initiation Protocol (SIP)”. You can follow “Extension for Event State Publication” (Niemi, A.).

  Then, the presence provider 222 receives the attendee list of the selected room from the room management server 110 in the form of a NOTIFY message, for example. The attendee list includes identification information of other users who have entered the room, position information in the virtual space, and the like, and virtual space attributes of the selected room.

  The exit process is a process when the user leaves the room. The presence provider 222 receives the exit instruction of the own user and transmits an exit message including the identification information of the own user to the room management server 110. When the SIP INVITE message is used as the visitor message, it is appropriate to use the BYE message as the exit message. Further, when the SUBSCRIBE message is transmitted in the admission process, the client should transmit an UNSUBSCRIBE message for canceling the notification request by the SUBSCRIBE message to the room management server 110.

  The movement process of the own user is a process when the own user changes the presence, that is, when the position or orientation in the virtual space is changed.

  FIG. 16 is a process flow diagram of the movement process of the own user. First, the space modeler 221 receives input of movement information from the pointing device 230 (S1101). That is, when the user operates the pointing device 230, movement information is input to the space modeler 221.

  When the space modeler 221 detects an input from the pointing device 230, the position after the user's movement is detected using the position and orientation before the user's movement and the movement information input from the pointing device 230. And the orientation is calculated (S1102). Note that the pointing device 230 may directly input position information after movement. Then, the space modeler 221 stores the calculated position information after movement in a memory or the like.

  Next, the space modeler 221 notifies the calculated position information and the like after movement to the graphics renderer 219 and the presence provider 222 (S1103). The graphics renderer 219 changes the viewpoint of the own user based on the notified position and orientation after movement in the virtual space, and calculates (coordinate conversion) how other users can see in the virtual space. Then, the graphics renderer 219 creates image data to be output on the screen with the view from the position and orientation, and updates the display screen.

  The presence provider 222 notifies (sends) the notified location information of the own user after moving to the room management server 110 in the form of a NOTIFY message, for example (S1104). The NOTIFY message is normally transmitted as a result of receiving a SUBSCRIBE message. Therefore, when the room management server 110 receives an admission message from the client 201, it may be possible to send back a SUBSCRIBE message corresponding to the NOTIFY message as well as returning a visitor list.

  The room management server 110 receives the location information notified from the presence provider 222 and updates the location information of the user in the attendee list. Then, the room management server 110 transmits the notified position information and the like to the acoustic server 120 and other users' clients. The acoustic server 120 calculates how the other user's voice can be heard at the notified position and orientation in the virtual space of the own user (see FIG. 14). Then, the acoustic server 120 performs processing such as volume adjustment, reverberation, filtering, and the like on the other user's voice, controls the acoustic effect on the sound to be heard at the position in the user's virtual space, and Update the sound. Then, the acoustic server 120 transmits the updated stereophonic audio signal to the client.

  The movement process of another user is a process when the room management server 110 notifies the client of position information and the like in the virtual space of the other user.

  FIG. 17 is a process flow diagram of another user's movement process.

  The space modeler 221 receives location information of other users in the virtual space from the room management server 110 via the presence provider 222 (S1201). Note that the room management server 110 notifies (transmits) the position information and the like transmitted from the client in S1104 of FIG. 16 to clients other than the transmission source client. Then, the space modeler 221 stores the notified virtual position information or the like in a memory or the like.

  Then, the space modeler 221 changes the position and orientation of the other user in the virtual space using the notified position information or the like (S1202). Then, the space modeler 221 notifies the graphics renderer 219 of the changed position information and the like (S1203). As described in S1103 of FIG. 16, the graphics renderer 219 updates the display screen based on the notified position and orientation of the other user.

  When the room management server 110 notifies the client of location information or the like of another user in the virtual space, the room management server 110 also notifies the acoustic server 120 of the location information or the like of the other user. The acoustic server 120 transmits a stereophonic audio signal updated based on the position information of the other user to the client.

  The swing process is a process when the left / right swing button 231 or the up / down swing button is pressed.

  FIG. 18 is a process flowchart of the swing process. The space modeler 221 receives an input of a swing process from the left / right swing button 231 or the up / down swing button 232 (S1301). That is, when it is difficult for the user to determine the direction of the communication partner, the user instructs to swing his / her head virtually in the virtual space by pressing the buttons 231 and 232.

  When the space modeler 221 detects an input from the buttons 231 and 232, the spatial modeler 221 calculates the orientation of the user when the user's neck is swung left (or up) by a predetermined angle. Then, the space modeler 221 notifies the calculated orientation information of the user to the graphics renderer 219 and the presence provider 222 (S1302).

  Then, the space modeler 221 calculates the orientation of the user when the user's neck is swung to the right (or down) by a predetermined angle. Then, the space modeler 221 notifies the calculated orientation information of the user to the graphics renderer 219 and the presence provider 222 (S1303).

  As shown in FIG. 9 or FIG. 10, the graphics renderer 219 creates image data when the user shakes his / her head from the viewpoint of the user, and updates the display screen.

  The presence provider 222 sequentially notifies (transmits) the direction information of the user notified in S1302 and S1303 to the acoustic server 120 (S1204). The acoustic server 120 receives the direction information notified from the presence provider 222, and sequentially updates the direction information of the user in the space modeler 124. Then, the acoustic server 120 calculates how the voice of the communication partner can be heard in the notified user's orientation in the virtual space (see FIG. 14). Then, the acoustic server 120 performs processing such as volume adjustment, reverberation, filtering, and the like on the voice of the other user who is the communication partner, and the sound to be heard at the position and orientation in the user's virtual space. Control effects and update stereophony. The acoustic server 120 transmits the updated stereophonic audio signal to the client whose buttons 231 and 232 are pressed.

  The swing process shown in FIG. 18 is performed at a predetermined time (for example, for about 1 second), and then the client and the acoustic server 120 return to the state before the buttons 231 and 232 are pressed.

  Next, the functional configuration and processing procedure of the room management server 110 will be described. Since the registration server 130 is the same as the conventional communication using SIP, the description is omitted.

  FIG. 19 shows a functional configuration of the room management server 110. The room management server 110 includes an interface unit 111 for transmitting and receiving various types of information to and from the client and the acoustic server 120, a determination unit 112 that determines a message type from the client, and a processing unit 113 that performs processing according to the determination result. And a storage unit 114 that manages and stores the attributes of the virtual space, events (user entry / exit, movement, etc.) that occurred in the virtual space, room lists, visitor lists, and the like.

  In the storage unit 114, attributes of some virtual spaces managed by the room management server 110 are stored in advance. In the admission process described above, the user selects a virtual space to enter from these virtual spaces (virtual rooms). Thereafter, the client transmits various events of the user who entered the virtual space to the room management server 110. As a result, various events occur in each virtual space. The storage unit 114 stores such information in the memory 302 or the external storage device 303.

  FIG. 20 shows the processing procedure of the room management server 110. The room management server 110 receives a request from a client and performs processing for this until the room management server 110 stops. First, the interface unit 111 waits for a message from the client (S1411). When receiving the message, the determination unit 112 determines the type of message received by the interface unit 111 (S1412).

  In the case of a login message, the processing unit 113 instructs the interface unit 111 to transmit the room list to the message transmission source client (S1421). The interface unit 111 transmits the room list to the message transmission source client, and then returns to S1411 to wait for the next message.

  In the case of an entrance message, the processing unit 113 adds the user of the message transmission source client to the entry list of the designated room (S1431). That is, the processing unit 113 adds the identification information of the user and the position information and direction information of the user in the virtual space included in the entrance message to the attendee list. Next, the processing unit 113 instructs the interface unit 111 to transmit the virtual space attribute of the designated room and the visitor list to the message transmission source client. The attendee list includes identification information of all users who have entered the designated room, position information in the virtual space, and direction information. The interface unit 111 transmits the visitor list to the transmission source client according to the instruction (S1432). And it progresses to S1436 mentioned later.

  When the message is a moving message, the processing unit 113 updates the position information and the direction information in the virtual space of the message transmission source client (user) in the attendee list (S1435). Note that position information and orientation information in the virtual space are included in the movement message. Then, the processing unit 113 sends the identification information of the user of the message transmission source client and the position in the virtual space to all the visitors' clients (excluding the message transmission source client) and the acoustic server 120 in the target room. The interface unit 111 is instructed to notify the information and the direction information (S1436). The interface unit 111 transmits to each client and the acoustic server 120 in accordance with the instruction, and returns to S1411.

  Also in the case of an entrance message, the process of S1436 is performed after the process of S1432. That is, when the admission message is received, the processing unit 113 notifies the notification acoustic server 120 and the client of the identification information of the user of the message transmission source client, the position information in the virtual space, and the like (S1436). Thus, when the client enters the room, the client performs voice communication with a predetermined communication port of the acoustic server 120 (or with a port notified from the room management server 110 at the time of entry). That is, the audio communication unit 215 of each client transmits a one-channel audio stream to the acoustic server 120 and receives a two-channel audio stream from the acoustic server 140.

  In the case of an exit message, the processing unit 113 deletes the user of the message transmission source client from the visitor list (S1441). Then, the processing unit 113 is an interface that notifies all the visitors' clients (except the message transmission source client) in the target room and the acoustic server 120 that the user has left the room. The unit 111 is instructed (S1442). The interface unit 111 transmits to the client in accordance with the instruction, and returns to S1411.

  The embodiment of the present invention has been described above.

  In the present embodiment, when it is difficult to determine the direction of the sound source simply by listening to the sound output from the headphones 217, the user uses the left / right swing button 231 or the up / down swing button 232 to virtually Shake your neck left and right or up and down. Thereby, the sound of the sound source reproduced from the headphones 217 changes (vibrates) left and right or up and down.

  In the case of the left / right swing button 231, the way the sound of the sound source changes depends on whether the sound source exists in front of or behind the user, so that the user can grasp the exact direction of the sound source. . In addition, in the case of the up / down swing button 232, the way of changing the sound of the sound source differs depending on whether the sound source exists above or below the user, so that the user must grasp the exact direction of the sound source. Can do. That is, by using the left / right swing button 231 or the up / down swing button 232, the user can more accurately determine (recognize) the direction of the sound source in the virtual space.

In the present embodiment, an image of the virtual space (see FIGS. 8 and 11) displaying the distance such as a scale bar or a mesh is output to the display 220. As a result, the user visually discriminates (recognizes) the distance from the sound source audibly discriminated (recognized) from the three-dimensional stereophonic sound output from the headphone 217 and the image displayed on the display 220. Learn the relationship with the distance to the sound source. Thereby, the user can grasp | ascertain the distance with the sound source in virtual space more correctly only by listening to the three-dimensional stereophonic sound stand. That is, by referring to the image data of the virtual space displaying the distance, the user can improve the ability to grasp the distance (sound source position) to the sound source in the virtual space.
In addition, the following effects generate | occur | produce by raising the capability to grasp | ascertain the distance with the sound source in virtual space.

  First, in a conference system using a virtual space, a richer communication environment can be realized by accurately recognizing the distance to a communication partner (another user who is an attendee of the conference). For example, in a meeting where there are other users who are communication partners, when determining whether or not a statement made by a certain other user is directed to him (your user) It is important to know the distance to the user. That is, when another user who is a speaker is approaching the user in the virtual space, the possibility of speaking to the user becomes higher.

  Also, in a one-on-one conversation, when another user talks from a close position (for example, approaching about 1 m), the user is trying to establish a more intimate relationship with the other user or a more important (secret) Recognize that you are talking about content. However, if the other user is speaking from a remote location (for example, about 5 m), the user tries to talk about content that the other user is hostile or less important (secret). Recognize that

  In a conventional conference system, it has been difficult to transmit such non-verbal communication information. However, by learning the distance to the sound source in the virtual space, such non-verbal information can be transmitted, and richer communication can be realized.

  Secondly, a game with higher reality can be realized by incorporating a sense of distance in a game of fighting enemies in a virtual space. In other words, in a game in which a three-dimensional stereophonic sound is used to fight an enemy in a virtual space, is it possible to accurately grasp the distance from the game player (own user) to the opponent (other user, predetermined object)? There are cases where the game is divided. For example, it is unlikely that an opponent at a long distance will hit the game performer (own user), but if an opponent at a short distance fires, it will hit the game performer (self user). Probability is high. Therefore, the game performer (own user) needs to pay more attention when the opponent is at a short distance. Also, even in visual games using computer graphics, when the opponent approaches from behind, the presence or distance of the opponent cannot be displayed on the display. It is important to learn.

  In addition, this invention is not limited to said embodiment, Many deformation | transformation are possible within the range of the summary.

  For example, in the client of the present embodiment, the graphics renderer 219 outputs virtual space image data (see FIGS. 8 to 11) to the display 220. However, since the present invention is a system mainly for communication by stereophonic sound using the three-dimensional audio technology, the client 201 may not output the virtual space image data to the display 220. In this case, the client 201 does not have the graphics renderer 219 and the display 220.

  In the present embodiment, the acoustic server 120 three-dimensionalizes the user's voice (audio signal) transmitted from each client for each client using the tertiary audio technology. Then, the acoustic server 120 transmits each user's voice, which is three-dimensional for each client, to each client. However, each client may directly transmit and receive audio (audio signals) in a one-to-one manner without going through the acoustic server 120 and three-dimensionalize audio input from other clients.

  In this case, each client differs from the client configuration shown in FIG. 3 in the following points. That is, the audio decoder 216 has the same function as the audio renderer 122 (see FIGS. 13 and 14) of the acoustic server 120. The audio communication unit 215 communicates directly with other clients instead of communicating with the acoustic server 120. In this case, the acoustic server 120 is not necessary.

  In addition, the virtual space in the present embodiment is a space that is virtually created for a plurality of users to hold a conference or conversation. However, the present invention is not limited to this, and may be a space created virtually by a user to listen to various sound sources such as music and Internet broadcasts.

  Further, the stereophonic sound reproduction system described in the present embodiment includes a registration server 130. However, if communication is not performed using the SIP protocol, the registration server is not necessary.

It is a whole block diagram in this embodiment. It is a hardware block diagram of each apparatus in this embodiment. It is a block diagram of the client in this embodiment. It is the figure which showed typically the direction of the sound source in this embodiment. It is the figure which showed typically the direction of the sound source in the state which faced the left in this embodiment. It is the figure which showed typically the direction of the sound source in the state which faced the right in this embodiment. It is the figure which showed typically the direction of the sound source in the state which faced up or down in this embodiment. It is an example of the display display screen of the virtual space by the top view in this embodiment. It is an example of a display display screen of a virtual space in a state of facing left and right in the present embodiment. It is an example of the display display screen of the virtual space of the state which faced up and down in this embodiment. It is an example of the display display screen of the virtual space by the perspective view in this embodiment. The type of the client in this embodiment is illustrated. It is a block diagram of the acoustic server in this embodiment. It is the figure which showed typically the process of the audio renderer in this embodiment. It is the figure which showed typically the mirror image for demonstrating the initial stage reflection in this embodiment. It is a movement process flowchart of the self user of the client in this embodiment. It is a movement processing flowchart of the other user of the client in this embodiment. It is a swing process flowchart of the client in this embodiment. It is a function block diagram of the presence server in this embodiment. It is a processing flow figure of the presence server in this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Network, 110 ... Presence server, 120 ... Acoustic server, 130 ... Registration server, 201, 202, 203 ... Client, 211 ... Microphone, 212 ... Audio encoder, 215 ... Audio communication part, 216 ... Audio decoder, 217 ... Headphone 219 ... Graphics renderer, 220 ... Display, 221 ... Spatial modeler, 222 ... Presence provider, 223 ... Session control unit, 230 ... Pointing device, 231 ... Left / right swing button, 232 ... Up / down swing button

Claims (10)

A stereophonic sound reproduction system for controlling the sound of at least one sound source existing in a virtual space,
An acoustic server for controlling the acoustic effect of each of the sound sources, and a client used by the user,
The client
Accepting means for accepting a swing instruction to swing the user's neck left and right or up and down arranged in the virtual space;
Client transmission means for transmitting the swing instruction received by the reception means to the acoustic server;
Client receiving means for receiving, from the acoustic server, a three-dimensional acoustic signal in which the acoustic effect of each of the sound sources is controlled;
Output means for outputting the stereophonic sound of the stereoacoustic signal received by the client receiving means,
The acoustic server is
Server storage means for storing the position and orientation of each of the user and the sound source in the virtual space;
Server receiving means for receiving an acoustic signal output by the sound source from each of the sound sources;
Acoustic control means for controlling an acoustic effect applied to each acoustic signal received by the server reception means based on the position and orientation of each of the user and the sound source stored in the server storage means;
Server transmission means for transmitting to the client a stereophonic sound signal in which the acoustic control means has controlled the acoustic effect;
When the swing instruction is received from the client, the acoustic control unit applies the received acoustic signal to each of the received acoustic signals based on the direction changed from the user orientation stored in the server storage unit to the left, right, or up and down. A three-dimensional sound reproduction system characterized by controlling sound effects. The three-dimensional sound reproduction system according to claim 1,
A management server for managing positions of the user and the sound source in the virtual space,
The management server
Management server receiving means for receiving the position and orientation of the user and each of the sound sources in the virtual space from the client or the sound sources,
Management server storage means for storing the position and orientation received by the management server reception means;
Management server transmission means for transmitting the position and orientation of each of the user and the sound source stored in the management server storage means to the acoustic server,
The client
Further comprising position information transmitting means for transmitting the position and orientation of the user in the virtual space to the management server;
The acoustic server is
Further comprising position information receiving means for receiving the position and orientation of each of the user and the sound source in the virtual space from the management server and storing them in the server storage means;
The three-dimensional sound reproduction system, wherein the sound control means controls sound effects applied to each sound signal received by the server receiving means based on the position and orientation received by the position information receiving means. The three-dimensional sound reproduction system according to claim 1,
The client
Client storage means for storing the position and orientation of each of the user and the sound source in the virtual space;
Image creation means for creating image data to be output to a display device based on the position and orientation stored in the client storage means;
The image creating means displays a scale for indicating a distance in the virtual space between the user and the sound source in the image data,
The acoustic control means of the acoustic server includes:
Based on the position and orientation of each of the user and the sound source stored in the server storage unit, a distance in the virtual space between the user and each of the sound sources is calculated,
A stereophonic sound reproduction system characterized in that, for each sound signal of the sound source received by the server receiving means, the sound effect is controlled by increasing or decreasing the ratio of the reflected sound to the direct sound according to the calculated distance. The three-dimensional sound reproduction system according to claim 3,
A management server for managing positions of the user and the sound source in the virtual space,
The management server
Management server receiving means for receiving the position and orientation of the user and each of the sound sources in the virtual space from the client or each of the sound sources;
Management server storage means for storing the position and orientation received by the management server reception means;
Management server transmission means for transmitting the position and orientation of each of the user and the sound source stored in the management server storage means to the acoustic server,
The client
Further comprising position information transmitting means for transmitting the position and orientation of the user in the virtual space to the management server;
The acoustic server is
Further comprising position information receiving means for receiving the position and orientation of each of the user and the sound source in the virtual space from the management server and storing them in the server storage means;
The acoustic control unit calculates a distance in the virtual space between the user and each of the sound sources based on the position and orientation received by the position information receiving unit, and applies the calculated distance to each acoustic signal received by the server receiving unit. A three-dimensional sound reproduction system characterized by controlling sound effects. The three-dimensional sound reproduction system according to claim 1,
The client
Client storage means for storing the position and orientation of each of the user and the sound source in the virtual space;
Image creation means for creating image data to be output to a display screen based on the position and orientation stored in the client storage means;
When the swing instruction is received, the image creation means fixes the position and orientation of the user in the virtual space, and image data obtained by moving the sound sources relatively left and right or up and down around the user. A stereophonic sound reproduction system characterized by creation. The three-dimensional sound reproduction system according to claim 5,
The sound source is a user other than the user arranged in the virtual space,
The image creation means of the client displays the shoulder width of the user and the shoulder width of each of the other users for indicating the distance in the virtual space with the user and each of the other users in the image data,
The acoustic control means of the acoustic server includes:
Based on the positions and orientations of the user and each of the other users stored in the server storage unit, a distance in the virtual space between the user and each of the other users is calculated,
A stereophonic sound reproduction system characterized by controlling the sound effect by increasing or decreasing the ratio of reflected sound to direct sound according to the calculated distance for each of the other user's sound signals received by the server receiving means. A stereophonic sound reproducing device for controlling the sound of at least one sound source existing in a virtual space,
Storage means for storing positions and orientations of the user using the stereophonic device and each of the sound sources in the virtual space;
Receiving means for receiving an acoustic signal output from the sound source from each of the sound sources;
Acoustic control means for controlling an acoustic effect applied to each acoustic signal received by the reception means based on the position and orientation of each of the user and the sound source stored in the storage means;
Accepting means for accepting a swing instruction to swing the user's neck left and right or up and down arranged in the virtual space;
Output means for outputting stereophonic sound controlled by the sound control means,
When the swing instruction is received, the sound control means controls the sound effect applied to each of the received sound signals based on the direction changed from the user orientation stored in the storage means to the left or right or up and down. A stereophonic sound reproducing device characterized by: A three-dimensional sound reproduction method in a three-dimensional sound reproduction system for controlling the sound of at least one sound source existing in a virtual space,
The three-dimensional sound reproduction system includes a sound server that controls sound effects of each of the sound sources, and a client that is used by a user.
The processing unit of the client is
An instruction accepting step for accepting a swing instruction to swing the user's neck placed in the virtual space left and right or up and down;
A client transmission step of transmitting the swing instruction received in the instruction reception step to the acoustic server;
A client receiving step of receiving, from the acoustic server, a three-dimensional acoustic signal in which the acoustic effect of each of the sound sources is controlled;
Outputting the stereophonic sound of the stereoacoustic signal received by the client reception step, and
The acoustic server includes a processing unit, and a storage unit that stores positions and orientations of the user and the sound source in the virtual space,
The processing unit of the acoustic server is
A server receiving step of receiving an acoustic signal output from the sound source from each of the sound sources;
An acoustic control step for controlling an acoustic effect applied to each acoustic signal received in the server reception step based on the position and orientation of each of the user and the sound source stored in the storage unit;
A server transmission step of transmitting a stereophonic sound signal whose acoustic effect is controlled in the acoustic control step to the client, and
When the swing instruction is received from the client, the sound control step is configured to apply sound to each of the received sound signals based on a direction changed from the user's direction stored in the storage unit to the left / right or up / down. A stereophonic sound reproduction system characterized by controlling the effect. A stereophonic sound reproduction method for controlling the sound of at least one sound source existing in a virtual space, performed by an information processing device,
The information processing device includes a processing unit, a storage unit that stores a position and orientation of each of the sound source and the user who uses the information processing device in the virtual space, and the user's neck arranged in the virtual space. Or an instruction receiving unit that receives a swing instruction to swing up and down,
The processor is
A receiving step of receiving an acoustic signal output from the sound source from each of the sound sources;
An acoustic control step for controlling an acoustic effect applied to each acoustic signal received in the reception step based on the position and orientation of each of the user and the sound source stored in the storage unit;
An output step for outputting the stereophonic sound controlled in the acoustic control step,
The acoustic control step is applied to each of the received acoustic signals based on a direction changed from the user orientation stored in the storage unit to the left or right or up and down when the instruction receiving unit receives the swing instruction. A three-dimensional sound reproduction method characterized by controlling an acoustic effect to be performed. A three-dimensional sound reproduction program for controlling sound of at least one sound source existing in a virtual space, performed by an information processing device,
In the information processing apparatus,
Storage means for storing positions and orientations of the user using the stereophonic device and each of the sound sources in the virtual space;
Receiving means for receiving an acoustic signal output from the sound source from each of the sound sources;
Acoustic control means for controlling an acoustic effect applied to each acoustic signal received by the receiving means based on the position and orientation of each of the user and the sound source stored in the storage means;
Accepting means for accepting a swing instruction to swing the user's neck left and right or up and down arranged in the virtual space; and
An output unit that outputs stereophonic sound controlled by the sound control unit;
When receiving the swing instruction, the sound control means controls the sound effect applied to each of the received sound signals based on the orientation changed from the user orientation stored in the storage means to the left or right or up and down. A stereophonic sound reproduction program characterized by:

Images