JP4426484B2 - Audio conference system, conference terminal and audio server - Google Patents

Description

  The present invention relates to a technology of an audio conference system using 3D audio technology.

  Patent Document 1 discloses a conference room connection type audio conference system. In this audio conference system, a microphone and a speaker are installed in each conference room, the conference rooms are connected to each other, and the voice of the participant in the conference room picked up by the microphone is installed in each other conference room. Output from.

  Patent Document 2 discloses an audio conference system using 3D audio technology. In this audio conference system, audio data of each participant subjected to 3D audio processing (stereoscopic audio processing) is output from the conference terminal of each participant participating in the audio conference.

US Pat. No. 5,365,583 US Pat. No. 6,327,567

  In the conference room connection type audio conference system described in Patent Document 1, when there are a plurality of participants in the conference room, it is easy to determine who speaks from the audio data collected by the microphone installed in the conference room. There is a problem that cannot be determined.

  On the other hand, in the audio conference system using the 3D audio technology described in Patent Document 2, each participant's audio is subjected to 3D audio processing to express distance and direction. For this reason, it is possible to easily determine who speaks from the voice data. However, when two or more of the audio conference participants use the audio conference system in the same conference room, the distance and direction expressed by the audio data of the participants in the same conference room, If the actual position and orientation are different, the distance and direction of the participant's voice that can be heard via the audio data are different from the distance and direction of the participant's voice that can be heard directly, and this causes a sense of incongruity.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to prevent a sense of incongruity in the direction and distance of each user represented by audio data in an audio conference system using 3D audio technology. Is to make it.

  In order to solve the above-described problems, the present invention provides a presence server that manages the location of each of a plurality of conference terminals participating in an audio conference in the virtual space and the location in the real space. Then, for each conference terminal, using the presence server, the voice data of the users of the other conference terminals other than the conference terminal existing at the same location in the same real space as the conference terminal is transmitted to each of the other conference terminals. Based on the position of the user in the virtual space, the sound is synthesized and synthesized.

For example, an audio conference system according to the present invention includes a plurality of conference terminals, a presence server that manages the location of each user of the plurality of conference terminals in the virtual space and the location in the real space, A plurality of relay devices installed at a location and used for communication of a conference terminal existing at the location with the presence server;
Each of the plurality of conference terminals includes:
Virtual position information transmitting means for transmitting to the presence server virtual position information of the own user including the position and direction of the own user in the virtual space, which is the user of the self-conference terminal;
Position information receiving means for receiving, from the presence server, virtual position information of a user of each conference terminal and location information indicating a location in real space;
Based on the location information of each conference terminal user received by the location information receiving means, another conference room user detection for detecting a user in another conference room who is a user in a location different from the location in the real space of the own user Means,
And audio data transmission means for transmitting the audio data of its own user, to the conference terminal each of the other conference users the other conference room user detection means detects,
Voice data receiving means for receiving voice data of other conference room users from each conference terminal of the other conference room users detected by the other conference room user detecting means,
The other conference room user in the virtual space specified by the virtual position information of the other conference room user and the virtual position information of the own user for each voice data of the other conference room user received by the voice data receiving means A speech synthesis unit that performs stereophonic sound processing according to the relative position between the user and the user, synthesizes each of the audio data of other conference room users subjected to the stereophonic sound processing, and generates stereo-synthesized sound data;
Voice control means for outputting the three-dimensional synthesized voice data generated by the voice synthesis means from a speaker;
The presence server
While managing the virtual position information of the users sent from each of the plurality of conference terminals, and based on the relay device through which the information sent from each of the plurality of conference terminals has passed, Management means for managing user location information;
For each of the plurality of conference terminals, there is provided location information transmission means for transmitting the virtual location information and location information of the user of each conference terminal managed by the management means to the conference terminal.

In addition, another audio conference system of the present invention includes a plurality of conference terminals, a presence server that manages a location of each of the plurality of conference terminals in a virtual space and a location in real space, and the plurality of conferences A voice server that transmits voice data to each of the terminals, and a plurality of relay devices that are installed at each location in the real space and are used by the conference terminal that exists at the location to communicate with the presence server; Have
Each of the plurality of conference terminals includes:
Virtual position information transmitting means for transmitting to the presence server virtual position information of the own user including the position and direction of the own user in the virtual space, which is the user of the self-conference terminal;
Voice data transmitting means for transmitting the voice data of the own user to the voice server;
Three-dimensional synthesized voice data receiving means for receiving three-dimensional synthesized voice data from the voice server;
Voice control means for outputting the stereo synthesized voice data received by the stereo synthesized voice data receiving means from a speaker;
The voice server is
Position information receiving means for receiving, from the presence server, virtual position information of each user of the plurality of conference terminals and location information indicating a location in real space;
For each of the plurality of conference terminals , based on the location information of the user of each conference terminal received by the location information receiving means , a user who exists in the same location in the real space as the target user who is the user of the conference terminal Other meeting room user detecting means for detecting other meeting room users,
Voice data receiving means for receiving voice data of a user of the conference terminal from each of the plurality of conference terminals;
For each of the plurality of conference terminals, with respect to the audio data of each other conference room user detected by the other conference room user detection means, the virtual location information of the other conference room user and the target user who is the user of the conference terminal 3D acoustic processing is performed in accordance with the relative position between the other conference room user and the target user in the virtual space specified by the virtual position information, and the audio data of the other conference room users subjected to the 3D acoustic processing are synthesized. Voice synthesis means for generating three-dimensional synthesized voice data;
For each of the plurality of conference terminals, there is stereo synthesized speech data transmitting means for transmitting to the conference terminal stereo synthesized speech data for the target user of the conference terminal generated by the speech synthesizer,
The presence server
While managing the virtual location information of the users sent from each of the plurality of conference terminals, and based on the relay device through which the information sent from each of the plurality of conference terminals has passed, each of the plurality of conference terminals Management means for managing location information;
Position information transmission means for transmitting the virtual position information and location information of the user of each conference terminal managed by the management means to the voice server.

  According to the present invention, the conference terminal does not output the voice data of the users of other conference terminals existing at the same location in the real space. Only the audio data of the user of another conference terminal existing at another location in the real space is subjected to stereophonic processing and output. Therefore, it is possible to prevent a sense of incongruity from occurring in the direction and distance of each user expressed by the audio data.

  Hereinafter, embodiments of the present invention will be described.

<< first embodiment >>
FIG. 1 is a schematic configuration diagram of an audio conference system to which the first embodiment of the present invention is applied. As shown in the figure, the audio conference system of the present embodiment includes a presence server 1, a plurality of conference terminals 2, and a plurality of wireless LANs (Local Area Networks) connected to the presence server 1 via an IP (Internet Protocol) network 4. ) -AP (Access Point) 3A-3C.

  The wireless LAN-APs 3A to 3B are installed in different conference rooms A to C, respectively, and are used for the conference terminal 2 existing in the conference rooms A to C to communicate with the presence server 1. Although FIG. 1 shows three wireless LAN-APs, the number of wireless LAN-APs is not limited to this number.

  FIG. 2 is a schematic configuration diagram of the wireless LAN-APs 3A to 3C.

  As illustrated, the wireless LAN-APs 3 </ b> A to 3 </ b> C include an IP network interface unit 301 for connecting to the IP network 4, a wireless LAN interface unit 302, and a location information transmission unit 303.

  The wireless LAN interface unit 302 is an interface for connecting to the conference terminal 2 existing in the conference rooms A to C where the own wireless LAN-APs 3A to 3C are installed via the wireless LAN.

  When the location information transmission unit 303 receives a location information registration request with a user ID from the conference terminal 2 via the wireless LAN interface 302, the location information transmission unit 303 includes the user ID and an APID that is identification information of the own wireless LAN-AP 3A to 3C. The location information is transmitted to the presence server 1 via the IP network interface unit 301.

  The presence server 1 manages location information and location information (APID) of the user of each conference terminal 2 in the virtual space. Here, the virtual space is a space created virtually for the user of each conference terminal 2 to hold a conference. The attributes of the virtual space include, for example, the size of the space, the height of the ceiling, the reflectance / color / texture of the walls and ceiling, the reverberation characteristics, and the sound absorption rate by the air in the space.

  FIG. 3 is a schematic configuration diagram of the presence server 1.

  As illustrated, the presence server 1 includes an IP network interface unit 101 for connecting to the IP network 4, a location information management unit 102, a SIP server processing unit 103, and a location information storage unit 104.

  FIG. 4 is a diagram schematically showing registration contents in the position information storage unit 104. As shown in the figure, the position information storage unit 104 stores a record 1040 for each user of the conference terminal 2. The record 1040 includes a field 1041 for registering a user ID for uniquely identifying the user of the conference terminal 2, a field 1042 for registering the SIP-URI (Uniform Resource Identifier) of the conference terminal 2, and the conference terminal 2 A field 1043 for registering an IP address, a field 1044 for registering virtual position information indicating the position (coordinates) and orientation (direction of the line of sight) of the user of the conference terminal 2 in the virtual space, and the user of the conference terminal 2 A field 1045 for registering location information indicating the location (conference room).

  The location information management unit 102 searches and updates the record 1040 registered in the location information storage unit 104.

  The SIP server processing unit 103 uses the correspondence relationship between the SIP-URI and the IP address registered in the location information storage unit 104 to send the INVITE message received from the conference terminal 2 on the callee side to the conference terminal 2 on the callee side. Send to.

  FIG. 5 is a diagram for explaining the operation flow of the presence server 1.

  When the location information management unit 102 receives virtual location information together with the user ID from the conference terminal 2 via the IP network interface unit 101 (S1001), the location information storage unit stores a record 1040 in which the user ID is registered in the field 1041. The virtual position information registered in the field 1044 of the searched record 1040 is updated to the received virtual position information (S1003).

  Further, when the location information management unit 102 receives the location information (APID) together with the user ID from the wireless LAN-APs 3A to 3C via the IP network interface unit 101 (S1004), the user ID is registered in the field 1041. The record 1040 is searched from the position information storage unit 104 (S1005), and the location information registered in the field 1045 of the searched record 1040 is updated to the received location information (S1006).

  In addition, when the location information management unit 102 receives a location information transmission request with a user ID from the conference terminal 2 via the IP network interface unit 101 (S1007), it reads all the records 1040 from the location information storage unit 104 (S1008). ), It is sent back to the conference terminal 2 that sent the request (S1009).

  In addition, when the SIP server processing unit 103 receives an INVITE message with designation of a destination SIP-URI from the conference terminal 2 via the IP network interface unit 101 (S1010), the SIP-URI is registered in the field 1042. The current record 1040 is searched from the position information storage unit 104 (S1011). Then, the INVITE message is transferred with the IP address registered in the field 1043 of the retrieved record 1040 as the destination (S1012).

  Returning to FIG. 1, the description will be continued. The conference terminal 2 uses the voice data of the users of the other conference terminals 2 existing in the conference room other than the conference room in which the self-conference terminal 2 exists, the location information in the virtual space of each user, and the user of the self-conference terminal 2 Stereophonic processing is performed based on the relative positional relationship with the position information and output. FIG. 6 is a schematic configuration diagram of the conference terminal 2.

  As illustrated, the conference terminal 2 includes an audio input unit 201, an audio output unit 203, a video output unit 204, an operation reception unit 205, an audio encoder 206, an audio renderer 208, a presence provider 210, a space, Modeler 211, IP processing unit 212 for processing IP packets, RTP (Real-time Transport Protocol) processing unit 213, SIP control unit 214, seating information creation unit 217, other conference room user detection unit 218, A wireless LAN interface unit 219 for connecting to the wireless LAN-APs 3A to 2C via the wireless LAN.

  The audio input unit 210 is an input terminal for an audio signal collected by the microphone 221. The audio output unit 203 is an audio output terminal connected to a headphone (or speaker) 223 compatible with 3D audio (for example, pseudo 5.1 channel). Then, the operation reception unit 205 receives a user operation on the pointing device 225. The audio encoder 206 encodes the audio signal input to the audio input unit 201 and outputs audio data.

  The RTP processing unit 213 stores the audio data output from the audio encoder 206 in an RTP packet, and the RTP packet is transmitted to the destination notified from the SIP processing unit 214 via the IP processing unit 212 and the wireless LAN interface unit 219. Send to IP address. In addition, the RTP processing unit 213 extracts voice data from the RTP packet received from the other conference terminal 2 via the wireless LAN interface unit 219 and the IP processing unit 212, and the other conference room together with the transmission source address of the RTP packet. The data is output to the user detection unit 218.

  The space modeler 211 follows the attributes of the virtual space set in advance, and the user's position (coordinates) and line-of-sight direction (azimuth) in the virtual space according to the operation of the user's pointing device 225 received by the operation receiving unit 205. ) And the position information of the user including the determined position and line-of-sight direction is output to the presence provider 210. In addition, the space modeler 211 receives and holds the record 1040 including the location information and location information of the user of each conference terminal 2 from the presence provider 210 and outputs the record 1040 to the seating information generation unit 217 and the other conference room user detection unit 218. .

  The presence provider 210 periodically transmits the location information of the own user received from the space modeler 211 to the presence server 1 via the IP network interface unit 210. In addition, the presence provider 210 periodically transmits a location information transmission request to the presence server 1 via the IP processing unit 212 and the wireless LAN interface unit 219, and participates in the audio conference from the presence server 1 as a response. A record 1040 for each user is received. Then, the received record 1040 of each user is notified to the space modeler 211.

  The other conference room user detection unit 218 identifies the record 1040 of the own user from the record 1040 of each user received from the space modeler 211 based on the user ID of the own user. Then, another user having location information different from the location information (APID) of the own user, that is, a user record 1040 existing in a conference room different from the own user is extracted as a record 1040 of another conference room user. Then, the other conference room user detection unit 218 sends a source that matches the IP address included in the record 1040 of any other conference room user from the voice data received together with the source address from the RTP processing unit 213. Audio data having an address is extracted as audio data of the other conference room user. Then, the virtual position information included in the record 1040 of the other conference room user is added to each of the extracted audio data of the other conference room user and output to the audio renderer 208. Also, the other conference room user detection unit 218 outputs the virtual position information included in the record 1040 of the own user to the audio renderer 208.

  Based on the position information of each user record 1040 received from the space modeler 211, the seating room information creation unit 217, for example, as shown in FIG. The seating information display data indicating 2162 is generated. Then, the seating information display data is displayed on the display 224 via the video output unit 204.

  The audio renderer 208 receives the virtual position information of the own user from the other conference room user detection unit 218. Also, each of the audio data of other conference room users is received together with the virtual position information. Then, the received audio data of each other conference room user is buffered to synchronize (correlate) the audio data. This buffering (playout buffering) method is described in, for example, the document “Colin Perkins: RTP: Audio and Video for the Internet, Addison-Wesley Pub Co; 1st edition (June 11, 2003)”. Yes. In addition, the audio renderer 208 identifies each of the synchronized audio data of the other conference room users by the virtual position information of the other conference room users and the virtual position information of the own users, and the other users and the own users in the virtual space. Based on the relative position of the three-dimensional. The audio renderer 208 then outputs signal data (signal sequence) of two channels (left channel and right channel) to the 3D audio-compatible headphone 223 connected to the audio output unit 203.

  The audio renderer 208 will be described in more detail. In 3D audio technology, it is mainly generated by HRIR (Head Related Impulse Response) representing how the sound changes (impulse response) around the human head (hereinafter “human head”) and a virtual environment such as a room. The direction and distance of the sound is expressed by the simulated reverberation. HRIR is determined by the distance between the sound source and the human head and the angle (horizontal angle and vertical angle) between the human head and the sound source. It is assumed that the audio renderer 208 stores HRIR values measured in advance for each distance and each angle using a dummy head. Also, by using different values for the left channel (measured with the left ear of the dummy head) and the right channel (measured with the right ear of the dummy head), Express a sense of direction in the front-rear or up-down direction.

  FIG. 8 is a diagram for explaining the processing of the audio render 208. The audio renderer 208 performs the following calculation for each other conference room user for each piece of audio data sent from the other conference room user detection unit 218 together with the virtual position information of the other conference room user.

First, the audio renderer 208 receives the signal sequence s _i [t] (t = 1,...) Of the audio data of the other conference room user from the other conference room user detection unit 218 for each other conference room user. It is received together with the virtual position information of other conference room users. Then, the virtual position information of the other conference room user and the virtual position information of the user are used, and the signal sequence s _i [t] (t = 1,...) Of the voice data of the other conference room user is 3D audio. It sets to the parameter used for a process (S3001).

  Next, the audio renderer 208 calculates the direct sound of the audio data and the reflected sound that is reverberation for each other conference room user. For the direct sound, the distance and angle (azimuth) in the virtual space between the other conference room user and the user are calculated using the virtual position information set as parameters (S3002). Then, the audio renderer 208 specifies the HRIR corresponding to the distance and angle with the user from the previously stored HRIR values (S3003). The audio renderer 208 may use the HRIR value calculated by interpolating the HRIR value stored in advance.

  Next, the audio renderer 208 performs convolution calculation using the signal sequence input in S3001 and the HRIR left channel HRIR specified in S3003 to generate a left channel signal (S3004). Similarly, convolution calculation is performed using the signal sequence input in S3001 and the HRIR right channel HRIR specified in S3003 to generate a right channel signal (S3005).

  For reverberant sound, reverberation to be added is calculated using the position information set as a parameter in S3001 (S3006, S3007). That is, the audio renderer 208 calculates the reverberation based on the way of changing the sound (impulse response) according to the attribute of the virtual space. Hereinafter, calculation of reverberation will be described.

  Reverberation is composed of early reflections and late reverberation. Then, it is generally considered that the early reflection is more important than the late reverberation in the formation (recognition) of the sensation regarding the distance from other conference room users and the size of the room (virtual space). In a room in real space, after hearing the sound directly emitted from the sound source (direct sound), several tens of initial reflections from walls, ceilings, floors, etc., depending on conditions, between several ms to 100 ms. It is said that you can hear. If the shape of the room is a rectangular parallelepiped, there are only six initial reflections at a time. However, in a room with a more complicated shape or furniture, the number of reflected sounds increases, and sounds reflected multiple times by walls or the like can be heard.

  There is an image source method as a method for calculating the initial reflection. For example, “Allen, JB and Berkley, A.,“ Image Method for efficiently Simulating Small-Room Acoustics ”, J. Acoustic Society of America, Vol. 65, No. 4 , Pp.943-950, April 1979. ”. In the simple image source method, the wall, ceiling, and floor of the room are regarded as mirror surfaces, and the reflected sound is calculated as sound from the sound source image on the opposite side of the mirror surface.

  FIG. 9 is a diagram schematically showing a two-dimensional image source method in which the ceiling and floor are omitted for the sake of simplicity. That is, there is a virtual conference room 2081 that is an original virtual space in the center, and the virtual conference room 2081 includes its own user and other conference room users. In addition, twelve mirror images including the wall 2082 of the room are drawn around the virtual conference room 2081. The number of mirror images is not necessarily 12 and can be increased or decreased.

The audio renderer 208 assumes that the sound from each image of the other conference room user existing in each mirror image goes straight to the own user (listener), and the distance from each image of the other conference room user to the own user. The direction is calculated (S3006). Since the sound intensity is inversely proportional to the distance, the audio renderer 208 attenuates each volume according to the distance. However, if the reflectance of the wall is α (0 ≦ α ≦ 1), the sound sample reflected n times by the wall is multiplied by α ⁿ to further attenuate the volume.

  Note that the value of the reflectance α is about 0.6. The reason why the value is set to about 0.6 is to acquire reverberation (that is, the ratio of direct sound and reflected sound) sufficient for the user to recognize the distance to the other conference room user. Another reason is that if the value of α is excessively large, the user's sense of direction is disturbed.

  Next, the audio renderer 208 specifies the HRIR corresponding to the distance and angle with the own user from the stored HRIR numerical values for each image of other conference room users (S3007). Since the reflected sounds reach the human head from different directions, it is necessary to apply HRIR different from the HRIR of the direct sound specified in S3003.

  If a convolution calculation (S3007, S3008), which will be described later, is performed on each of a large number of reflected sounds using different HRIRs, an enormous calculation is required. In order to prevent an increase in the amount of calculation, HRIR when a sound source is in front may be applied to the calculation of reflected sound regardless of the actual direction of the sound source. By calculating only the time difference (ITD: interaural time difference) and intensity difference (IID: interaural intensity difference) when the sound reaches the left and right ears, the calculation of HRIR can be replaced with a small amount of calculation.

  Next, the audio renderer 208 performs convolution calculation using the signal sequence input in S3001 and the HRIR left channel HRIR specified in S3007 to generate reverberation of the left channel signal (S3008). Similarly, convolution calculation is performed using the signal sequence input in S3001 and the HRIR right channel HRIR specified in S3007 to generate the reverberation of the right channel signal (S3009).

  When the audio renderer 208 calculates the left channel signals of all the other conference room users as described above, all of them are added (S3010). Note that the left channel signal includes the direct sound calculated in S3004 and the reflected sound calculated in S3008.

  Similarly, if the audio renderer 208 has calculated the right channel signals of all the other conference room users as described above, all of them are added (S3011). The right channel signal includes the direct sound calculated in S3005 and the reflected sound calculated in S3009.

  The HRIR calculation (S3003, S3007) is performed for each voice data of one RTP packet. However, in the convolution calculation (S3004, S3005, S3008, S3009), a portion to be carried over to the next one packet of audio data occurs. For this reason, it is necessary to hold the specified HRIR or the input signal sequence until processing for the next one packet of audio data.

  As described above, the audio renderer 208 adjusts the volume by the above-described calculation, superimposition of reverberation and reverberation sound on the audio data of each other conference room user sent from the other conference room user detection unit 218, and Then, processing such as filtering is performed, and an acoustic effect is applied to the sound to be heard at the position in the virtual space of the user. That is, the audio renderer 208 generates stereophonic sound in which the sound is localized by processing resulting from the attribute of the virtual space and the relative position of the other conference room user with respect to the user.

  Returning to FIG. 6, the description will be continued. The SIP control unit 214 holds a table in which the user ID and SIP-URI of each conference terminal 2 are registered, and establishes a connection with another conference terminal 2 using this table as necessary. To do.

  FIG. 10 is a diagram for explaining the operation flow of the SIP control unit 214.

  When the conference terminal 2 is activated, the SIP processing unit 214 establishes a connection with each conference terminal 2 having the user ID of the other conference room user notified from the other conference room user detection unit 218. First, the SIP processing unit 214 extracts the SIP-URIs of unextracted other conference room users from the SIP-URIs registered in its own table (S4001). Next, the SIP processing unit 214 transmits an INVITE message destined for the extracted SIP-URI to the presence server 1 via the IP processing unit 212 and the wireless LAN interface unit 219, and the conference terminal 2 having the SIP-URI. In response to this, an attempt is made to establish a connection (S4002). Next, the SIP processing unit 214 checks whether or not the SIP-URIs of all other conference room users registered in its own table have been extracted (S4003). If not extracted, the process returns to S4001 and extracted. In such a case, the connection establishment process at the time of activation is terminated, and a transition is made to a wait state for various events.

  When the SIP processing unit 214 receives an INVITE message from the IP network 4 via the wireless LAN interface unit 219 and the IP processing unit 212 (YES in S4101), the conference terminal 2 that is the transmission source (originating side) of the INVITE message. A call control sequence according to SIP is executed with the conference terminal 2 to establish a connection with the conference terminal 2 (S4102).

  In addition, when the SIP processing unit 214 receives a BYE message from the conference terminal 2 of the communication partner that has established a connection via the wireless LAN interface unit 219 and the IP processing unit 212 (YES in S4201), the SIP communication unit 214 A call control sequence according to SIP is executed between the terminal 2 and the connection with the conference terminal 2 is released (S4202).

  In addition, when the user ID of the other conference room user is newly notified from the other conference room user detection unit 218 (YES in S4301), the SIP control unit 214 selects the SIP-URI registered in its own table. The SIP-URI associated with each user ID of the other conference room user who has been notified is extracted, and it is checked whether or not a connection has been established with each SIP-URI. If there is a SIP-URI that has not established a connection (YES in S4302), the SIP processing unit 214 sends an INVITE message addressed to the SIP-URI via the IP processing unit 212 and the wireless LAN interface unit 219. It transmits to the server 1 and tries to establish a connection to the conference terminal 2 having the SIP-URI (S4303).

  On the other hand, when connections have been established with all the SIP-URIs associated with the user IDs of other conference room users newly notified (NO in S4302), the SIP control unit 214 Using the table, it is associated with the user ID of the user other than the other conference room user newly notified from the other conference room user detection unit 218 in the SIP-URI of the other party whose connection is being established. Check whether there is a SIP-URI. If a connection is being established with a SIP-URI associated with a user ID other than another conference room user (YES in S4304), a BYE message destined for the SIP-URI is sent to the IP processing unit. The connection is transmitted to the conference terminal 2 having the SIP-URI via the 212 and the wireless LAN interface unit 219, and the connection with the conference terminal 2 is released (S4305).

  As shown in FIG. 11, the presence server 1 configured as described above includes a CPU 401 that processes and operates data according to a program, a memory 402 that the CPU 401 can directly read and write, an external storage device 403 such as a hard disk, and an IP network 3. A general computer system having a communication device 404 for performing data communication with an external system, an input device 405, and an output device 406 can be used. Specifically, a server, a host computer, and the like.

  The wireless LAN-APs 3A to 3C having the above configuration can use a computer system in which a wireless communication device for connecting to the wireless LAN is added to the configuration shown in FIG.

  In addition, the conference terminal 2 having the above configuration can use a computer system equipped with a wireless communication device for connecting to a wireless LAN in place of the communication device 404 in the configuration shown in FIG. For example, a PDA (Personal Digital Assistant), a handheld computer, and a wearable computer.

  FIG. 12 shows an example in which a PDA or a handheld computer is used for the conference terminal 2. The apparatus main body 230 is provided with a display 224, a pointing device 225, and a wireless LAN antenna 231. The headset connected to the apparatus main body 230 includes a microphone 221 and a 3D audio compatible headphone 223.

  The pointing device 225 includes a forward button 2251, a backward button 2252, a left movement button 2253, a right movement button 2254, and a selection button 2255. For example, pressing the forward button 2251 moves forward in the virtual space, and pressing the backward button 2252 moves backward in the virtual space. Note that the pointing device 225 may be a touch panel. That is, the surface of the display 224 may be a touch screen that is covered with a transparent screen (touch panel) on which elements for detecting contact with a finger or the like are arranged. The user can easily perform an input operation by touching the display 224 with a finger or a dedicated pen.

  Further, although the illustrated headset is connected to the apparatus main body 230 by wire, it may be connected by short-range wireless communication such as Bluetooth (registered trademark) or IrDA.

  Each function of each of the above devices is a predetermined program loaded or stored in the memory 302 (a program for a presence server in the case of the presence server 1, a program for a wireless LAN-AP in the case of the wireless LAN-APs 3A to 3C) In the case of the conference terminal 2, the program is implemented by the CPU 301.

  Next, a schematic operation of the video conference system having the above configuration will be described.

  FIG. 13 is a diagram for explaining the schematic operation of the audio conference system shown in FIG. In this example, first, users A and D are in the same conference room A, users B and C are in another conference room B, and then user A moves to the same conference room B as users B and C. By the way, the general operation when the user A participates in the audio conference will be described.

  First, the conference terminal 2 of the user A transmits the virtual position information of the user A to the presence server 1. In response, the presence server 1 registers the virtual position information of the user A (S5001). Further, the conference terminal 2 of the user A transmits a location information transmission request to the wireless LAN-AP 3A used by the conference terminal 2 for wireless communication (S5002). In response to this, the wireless LAN-AP 3A transmits the APID of its own wireless LAN-AP to the presence server 1 as the location information of the user A. In response, the presence server 1 registers the location information of the user A (S5003). In addition, the conference terminal 2 of the user A transmits a position information transmission request to the presence server 1 (S5004). In response, the presence server 1 transmits the virtual position information and location information of the users A to D (S5005).

  Next, the conference terminal 2 of the user A detects another conference room user (S5006). Here, the users A and D are initially in the same conference room A, and the users B and C are in another conference room B. Therefore, here, users B and C are detected as other conference room users. Therefore, the conference terminal 2 of the user A transmits an INVITE message addressed to the SIP-URI of each of the conference terminals 2 of the users B and C to the presence server 1. The presence server 1 transmits these INVITE messages to the conference terminals 2 of the users B and C (S5007). Thereby, the conference terminal 2 of the user A establishes a connection with each of the conference terminals 2 of the users B and C, and performs a voice conference through these connections (S5008). Since the user D is in the same conference room A, no connection is established with the user D. User A and user D have a direct conversation without going through the conference system of this embodiment.

  Now, it is assumed that the user A has moved to the same conference room B as the users B and C. Also in the conference room B, the conference terminal A of the user A performs the same processing as S5001 to S5006 (S5009 to S5014). As a result, only the user D is detected as another conference room user. For this reason, the conference terminal 2 of the user A transmits a BYE message to each of the conference terminals 2 of the users B and C that are establishing connections, and releases the connection with these conference terminals 2 (S5015). The conference terminal 2 of the user A transmits an INVITE message addressed to the SIP-URI of the conference terminal 2 of the user D to the presence server 1. The presence server 1 transmits this INVITE message to the conference terminal 2 of the user D (S5016). Thereby, the conference terminal 2 of the user A establishes a connection with the conference terminal 2 of the user D, and performs a voice conference through this connection (S5017). Since the users B and C are in the same conference room B, no connection is established between the users B and C. User A, user B, and user C have a direct conversation without going through the conference system of this embodiment.

  The first embodiment of the present invention has been described above. In the present embodiment, the conference terminal 2 does not output voice data of users of other conference terminals 2 existing in the same conference room in the real space. Only the audio data of the user of the other conference terminal 2 existing in another conference room in the real space is subjected to the stereophonic processing and output. Therefore, it is possible to prevent a sense of incongruity from occurring in the direction and distance of each user expressed by the audio data.

<< Second Embodiment >>
FIG. 14 is a schematic configuration diagram of an audio conference system to which the second embodiment of the present invention is applied. As shown in the figure, the video conference system of this embodiment includes a presence server 1 ′, an audio server 5, a plurality of conference terminals 2 ′, and a plurality of wireless LANs connected to the presence server 1 via the IP network 4. AP (Local Area Network) 3A-3C. In the present embodiment, components having the same functions as those in the first embodiment are denoted by the same reference numerals.

  The presence server 1 ′ manages the virtual position information and location information of the user of each conference terminal 2 ′. In response to the position information transmission request from the voice server 5, the virtual position information and location information of the user of each conference terminal 2 ′ are transmitted to the voice server 5. Note that the presence server 1 ′ of the present embodiment is obtained by omitting the SIP processing unit 103 from the presence server 1 of the first embodiment shown in FIG. The processing flow of the presence server 1 ′ according to the present embodiment is the same as that obtained by omitting the SIP processing (S1010 to S1012) from the processing flow of the presence server 1 according to the first embodiment shown in FIG.

  The voice server 5 receives the voice data of the user of each conference terminal 2 ′. Moreover, the audio | voice server 5 produces | generates the meeting audio | voice data (3D audio data) for users of the said conference terminal 2 'for every conference terminal 2', and transmits to the said conference terminal 2 '. FIG. 15 is a schematic diagram of the voice server 5.

  As illustrated, the voice server 5 includes an IP network interface unit 501 for connecting to the IP network 4, an RTP processing unit 502, an SIP processing unit 503, a presence provider 504, a spatial modeler 505, and user information generation. A unit 506, an audio distribution unit 508, and an audio renderer 509 provided for each conference terminal 2 ′.

  The SIP control unit 503 establishes a connection with each conference terminal 2 ′ via the IP network interface unit 501.

  For each conference terminal 2 ′, the RTP processing unit 502 receives the user's voice data from the conference terminal 2 ′ via the connection established with the conference terminal 2 ′. Along with the transmission source address of the voice data, the voice data is output to the voice distribution unit 508. In addition, the RTP processing unit 502 establishes, for each conference terminal 2 ′, conference audio data output from the audio renderer 509 associated with the conference terminal 2 ′ with the conference terminal 2 ′. It transmits to the said conference terminal 2 'via a connection.

  The presence provider 504 periodically sends a location information transmission request to the presence server 1 via the IP network interface unit 501, and as a response, the presence server 1 records the user's record (virtual location information, location). Information) 1040 is received. Then, the received record 1040 of each user is notified to the space modeler 505.

  The space modeler 505 receives and holds the user record 1040 of each conference terminal 2 ′ from the presence provider 504 and outputs it to the other conference room user detection unit 506.

  The user information generation unit 506 identifies, for each conference terminal 2 ′, a record 1040 including the user ID of the conference terminal 2 ′ from among the records 1040 of each user received from the space modeler 211. Then, own user information including the user ID, IP address, and position information included in the specified record 1040 is generated and transmitted to the voice distribution unit 508. In addition, the user information generation unit 506 searches for the record 1040 having location information different from the location information of the identified record 1040 from the records 1040 other than the identified record 1040 for each conference terminal 2 ′. Other conference room user information including the user ID, IP address, and virtual position information included in each record 1040 is generated, and each generated other conference room user information is included in the own user information of the conference terminal 2 ′. The voice ID is transmitted to the voice distribution unit 508 in association with the user ID.

  For each conference terminal 2 ′, the audio distribution unit 508 extracts audio data used for conference audio data to be transmitted to the conference terminal 2 ′ from the audio data of each user received from the RTP processing unit 502. Specifically, the following processing is performed for each conference terminal 2 ′. That is, the local user information including the user ID of the conference terminal 2 ′ is detected as the local user information of the conference terminal 2 ′ from the local user information received from the user information generation unit 506. Then, the user information of the conference terminal 2 ′ is output to the audio renderer 509 associated with the conference terminal 2 ′. Further, among the other conference room user information received from the user information generation unit 506, the other conference room user information associated with the user ID of the own user information of the conference terminal 2 ′ is changed to the other conference room 2 ′. Detect as conference room user information. In addition, voice data having the source address as the IP address of the other conference room user information of the conference terminal 2 ′ is detected from the voice data of each user received from the RTP processing unit 502. Then, the detected audio data is output to the audio renderer 509 associated with the conference terminal 2 ′ together with other conference room user information of the conference terminal 2 whose IP address is the transmission source address of the audio data.

  The audio renderer 509 receives each audio data from the audio distribution unit 508 together with other conference room user information. Also, the user information is received from the voice distributor 508. Then, the received audio data is buffered to synchronize (correlate) the audio data. In addition, the audio renderer 509 identifies each synchronized audio data based on the virtual location information of the other conference room user information given to each audio data and the virtual location information of the own user information. Three-dimensionalization is performed based on the relative position between the room user and the user. The audio renderer 509 outputs conference audio data including signal data (signal sequence) of two channels (left channel and right channel) to the RTP processing unit 502. Note that the method of three-dimensionalizing audio data is basically the same as that of the audio renderer 208 of the first embodiment (see FIGS. 8 and 9).

  The conference terminal 2 ′ establishes a connection with the voice server 5 and transmits the voice data of the own user to the voice server 5 through the connection. Also, the audio conference data is received from the audio server 5 through the connection and output. FIG. 16 is a schematic configuration diagram of the conference terminal 2 ′.

  As shown in the figure, the conference terminal 2 ′ includes an audio input unit 201, an audio output unit 203, a video output unit 204, an operation reception unit 205, an audio encoder 206, an audio decoder 248, a presence provider 210, The space modeler 211, the IP processing unit 212, the RTP processing unit 243, the SIP control unit 244, and the seating information creation unit 217 are included. Here, components having the same functions as those of the conference terminal 2 of the first embodiment shown in FIG.

  The SIP control unit 244 establishes a connection with the voice server 5 via the IP processing unit 212 and the wireless LAN interface unit 219.

  The RTP processing unit 243 transmits the audio data output from the audio encoder 206 to the audio server 5 via the connection established between the audio servers 5. Further, the conference audio data is received from the audio server 5 through the connection, and the received conference audio data is transmitted to the audio decoder 248.

  The audio decoder 248 decodes the conference audio data received from the RTP processing unit 243 and outputs an audio signal to the audio output unit 204.

  Similarly to the presence server 1 of the first embodiment, the presence server 1 ′ and the voice server 5 having the above configuration can also use a computer system as shown in FIG. 11. Specifically, a server, a host computer, and the like. Further, the conference terminal 2 ′ having the above configuration can also use a computer system as shown in FIG. 11, similarly to the conference terminal 2 of the first embodiment. For example, a PDA, a handheld computer, and a wearable computer.

  Next, a schematic operation of the voice conference system having the above configuration will be described.

  FIG. 17 is a diagram for explaining the schematic operation of the voice conference system shown in FIG. Here, an example is given in which the users E and H are initially in the same conference room A, the users F and G are in a different conference room D, and then the user E moves to the same conference room C as the users F and G. By the way, the general operation when the user E participates in the audio conference will be described. It is assumed that each of the conference terminals 2 ′ of the users E to H has established a connection with the voice server 5.

  First, the conference terminal 2 ′ of the user E transmits the virtual position information of the user E to the presence server 1. In response, the presence server 1 'registers the virtual position information of the user E (S6001). Further, the conference terminal 2 ′ of the user E transmits a location information transmission request to the wireless LAN-AP 3A used by the conference terminal 2 ′ for wireless communication (S6002). In response to this, the wireless LAN-AP 3A transmits the APID of its own wireless LAN-AP to the presence server 1 ′ as the location information of the user E. In response to this, the presence server 1 'registers the location information of the user E (S6003).

  On the other hand, the voice server 5 transmits a position information transmission request (S6004). In response, the presence server 1 ′ transmits the virtual position information and location information of the users E to H to the voice server 5 (S6005). Then, the voice server 5 detects other conference room users (S6006). Here, users E and H are initially in the same conference room A, and users F and G are in another conference room B. Accordingly, the users F and G are detected as other conference room users of the user E. For this reason, the audio server 5 generates and generates audio conference data by performing stereophonic processing on the audio data received from the conference terminals 2 of the users F and G based on the relative position with the user E and synthesizing them. Then, the generated voice conference data is transmitted to the conference terminal 2 ′ of the user E (S6007). Since the user H is in the same conference room A as the user E, the voice data of the user H is not included in the voice conference data for the user E. The user E and the user H have a direct conversation without going through the conference system of the present embodiment.

  Now, it is assumed that the user E moves to the same conference room B as the users F and G. In the conference room B, the same processing as S6001 to S6006 is performed (S6008 to S6013). As a result, only the user H is detected as the other conference room user of the user E. For this reason, the voice server 5 performs voice processing on the voice data received from the conference terminal 2 of the user H based on the relative position with the user E to generate voice conference data. Then, the generated audio conference data is transmitted to the conference terminal 2 ′ of the user E (S6014). Since the users F and G are in the same conference room B as the user E, the voice data of the users F and G is not included in the voice conference data for the user E. User E, user F, and user G have a direct conversation without going through the conference system of the present embodiment.

  The second embodiment of the present invention has been described above. Also in the present embodiment, as in the first embodiment, the conference terminal 2 ′ does not output the voice data of users of other conference terminals 2 ′ existing in the same conference room in the real space. Only the audio data of the user of the other conference terminal 2 ′ existing in another conference room in the real space is subjected to the stereophonic processing and output. Therefore, it is possible to prevent a sense of incongruity from occurring in the direction and distance of each user expressed by the audio data.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist.

  For example, in each of the above embodiments, each conference terminal 2, 2 ′ determines the virtual position information (position and orientation) of the user according to the operation content received from the user via the pointing device 225. However, the present invention is not limited to this. For example, the virtual position information of the user of the conference terminal 2, 2 ′ is determined based on the current position and orientation in the conference room where the conference terminal 2, 2 ′ is actually located Also good.

  FIG. 18 is a diagram for explaining a modification of the conference terminal 2 shown in FIG. In this modification, instead of the operation accepting unit 205, an azimuth measuring unit 253 that measures the azimuth of the conference terminal 2, and a current position calculating unit that calculates the current position of the conference terminal 2 in the conference room where the conference terminal 2 is located. 252.

  For the azimuth measuring unit 253, for example, a magnetic azimuth sensor can be used. Usually, the magnetic azimuth sensor has a Wheatstone bridge and a thin film coil composed of magnetoresistive elements. A magnetoresistive element changes its resistance value when a magnetic field is applied in a direction orthogonal to the direction of current flowing through the magnetoresistive element. The magnetic orientation sensor detects geomagnetism using this characteristic.

  For example, the current value calculation unit 252 determines the signal strength of wireless signals transmitted from at least three wireless transmitters installed in the conference room, and the installation positions (coordinates from the origin provided in the conference room) of each wireless transmitter. And the current position of the self-conference terminal 2 is measured by the principle of trilateral surveying. Here, for the position detection system using the wireless communication system, for example, “Ogino, Tsunehara et al./B-5-203, Wireless LAN integrated access system (1): Examination of position detection system, IEICE General Conference Proceedings of the Lecture, Vol. 2003_Communication Num. 1 pp.662 (2003.03) ”,“ Tsunehara, Sugano et al./B-5-204, Wireless LAN Integrated Access System (2): Study on location detection accuracy Proceedings of the IEICE General Conference, Vol. 2003_Communications Num. 1 pp.663 (2003.03) ”. In the present embodiment, the origin of each conference room is set at the center of each conference room.

  The space modeler 211 includes the current position in the conference room of the self-conference terminal 2 calculated by the current position calculation unit 252 and the orientation of the self-conference terminal 2 measured by the orientation measurement unit 253, and the position and orientation of the own user in the virtual space. Generate virtual location information. In the present embodiment, as described above, the origin of each conference room is set at the center of each conference room. Accordingly, the positions and orientations of the conference terminals 2 managed by the presence server 1 in the virtual space are as shown in FIG. That is, since the position and orientation of the conference terminal 2 in the real space are reflected in the position and orientation in the virtual space, it is possible to realize a voice conference that is more comfortable. Note that the origin of each conference room does not necessarily have to be the center of each conference room. In this case, the origin of each conference room has an offset value to the center of each conference room. A value obtained by correcting the measured value of the current position with the offset value is set as a position in the virtual space.

  The user of each conference terminal 2 confirms the position in his / her virtual space participating in the audio conference based on the seating information generated by the seating information generating unit 217 (see FIG. 7), and does not overlap with other users in the virtual space. Thus, the current position in its own conference room (real space) can be adjusted.

  Further, in each of the above embodiments, the wireless LAN-APs 3A to 3C transmit the location information of the request transmission source to the presence server 1 in response to the location information transmission request from the conference terminals 2 and 2 ′. . However, the present invention is not limited to this.

  For example, in response to the location information transmission request from the conference terminals 2 and 2 ', the wireless LAN-APs 3A to 3C return the location information of the request transmission source to the request transmission source, and the conference terminals 2 and 2' The location information received from the wireless LAN-APs 3A to 3C may be transmitted to the presence server 1.

  Alternatively, the wireless LAN-AP 3A to 3C or a network device provided for each conference room, which always passes when data of the conference terminals 2 and 2 'located in the conference room is transmitted to the IP network 4 (for example, A SIP proxy function may be added to the LAN switch, and a SIP header representing location information may be added to the SIP message transmitted from the conference terminals 2, 2 ′ to the presence server 1. As the SIP header to be added, for example, “Via: SIP / 2.0 / UDP room-301@aa.co.jp: 5060; type = room” can be considered. In this example, it is indicated by “type = room” that the SIP header represents location information, and the identifier of the conference room is indicated by “room-301@aa.co.jp: 5060”.

  Alternatively, the wireless LAN-AP 3A to 3C or a network device provided for each conference room, which always passes when data of the conference terminals 2 and 2 'located in the conference room is transmitted to the IP network 4 (for example, The SIP proxy function may be added to the LAN switch), and the location information may be added to the SIP registration request message (REGISTER packet) transmitted from the conference terminals 2, 2 ′ to the presence server 1. In this way, the location information of the conference terminals 2 and 2 ′ can be registered in the presence server 1 and 1 ′ using a normal SIP sequence as it is.

  Alternatively, the wireless LAN-AP 3A to 3C or a network device provided for each conference room, which always passes when data of the conference terminals 2 and 2 'located in the conference room is transmitted to the IP network 4 (for example, The LAN switch may have a router function with a DS (Differentiated Services) function, and the location information may be marked on the IP packet by using the DS field of the IP packet. If the DS field of the IP packet is used, 64 types of markings are possible, so that up to 64 conference rooms can be distinguished.

  In each of the above embodiments, the case where SIP is used for establishing a connection has been described as an example. However, the present invention is not limited to this. For example, a call control protocol other than SIP such as H.323 may be used. When it is assumed that communication is always performed between the conference terminal 2 ′ and the voice server 5 as in the second embodiment, the call control sequence according to the call control protocol can be omitted.

FIG. 1 is a schematic configuration diagram of an audio conference system to which the first embodiment of the present invention is applied. FIG. 2 is a schematic configuration diagram of the wireless LAN-APs 3A to 3C. FIG. 3 is a schematic configuration diagram of the presence server 1. FIG. 3 is a diagram schematically showing the registered contents of the position information storage unit 104. FIG. 5 is a diagram for explaining the operation flow of the presence server 1. FIG. 6 is a schematic configuration diagram of the conference terminal 2. FIG. 7 is a diagram showing a display example of seating information display data. FIG. 8 is a diagram for explaining the processing of the audio renderer 208. FIG. 9 is a diagram schematically showing a two-dimensional image source method with the ceiling and floor omitted. FIG. 10 is a diagram for explaining the operation flow of the SIP control unit 214. FIG. 15 is a diagram illustrating a hardware configuration example of each device constituting the audio conference system. FIG. 12 is a diagram illustrating an example of the appearance of the conference terminal 2. FIG. 13 is a diagram for explaining the schematic operation of the voice conference system shown in FIG. FIG. 14 is a schematic configuration diagram of an audio conference system to which the second embodiment of the present invention is applied. FIG. 15 is a schematic diagram of the voice server 5. FIG. 16 is a schematic configuration diagram of the conference terminal 2 ′. FIG. 17 is a diagram for explaining the schematic operation of the voice conference system shown in FIG. FIG. 18 is a diagram for explaining a modified example of the conference terminal 2. It is a figure for demonstrating an example of the determination method of a virtual present position.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1, 1 '... Presence server 2, 2' ... Conference terminal, 3A-3C ... Wireless LAN-AP, 4 ... IP network, 5 ... Voice server, 101 ... IP network interface part, 102 ... Location information management part, 103 DESCRIPTION OF SYMBOLS ... SIP server process part 104 ... Position information storage part 201 ... Audio | voice input part 203 ... Audio | voice output part 204 ... Video | video output part 205 ... Operation reception part 206 ... Audio encoder 208 ... Audio renderer 210 ... Presence Provider 211, Spatial modeler, 212 IP processor, 213 RTP processor, 214 SIP controller, 217 Seating information generator, 218 Other meeting room user detector, 219 Wireless LAN interface, 243 RTP processing unit, 244 ... SIP control unit, 248 ... audio decoder, 252 ... current location calculation unit, 253 ... direction meter 301: IP network interface unit, 302 ... Wireless LAN interface unit, 303 ... Location information transmission unit, 501 ... IP network interface unit, 502 ... RTP processing unit, 503 ... SIP control unit, 504 ... Presence provider, 505 ... Space Modeler, 506 ... user information generation unit, 508 ... audio distribution unit, 509 ... audio renderer

Claims (10)

An audio conference system,
A plurality of conference terminals, a presence server for managing a location of each of the plurality of conference terminals in a virtual space and a location in the real space, and each location in the real space. A plurality of relay devices used for existing conference terminals to communicate with the presence server;
Each of the plurality of conference terminals includes:
Virtual position information transmitting means for transmitting to the presence server virtual position information of the own user including the position and direction of the own user in the virtual space, which is the user of the self-conference terminal;
Position information receiving means for receiving, from the presence server, virtual position information of a user of each conference terminal and location information indicating a location in real space;
Based on the location information of each conference terminal user received by the location information receiving means, another conference room user detection for detecting a user in another conference room who is a user in a location different from the location in the real space of the own user Means,
And audio data transmission means for transmitting the audio data of its own user, to the conference terminal each of the other conference users the other conference room user detection means detects,
Voice data receiving means for receiving voice data of other conference room users from each conference terminal of the other conference room users detected by the other conference room user detecting means,
The other conference room user in the virtual space specified by the virtual position information of the other conference room user and the virtual position information of the own user for each voice data of the other conference room user received by the voice data receiving means A speech synthesis unit that performs stereophonic sound processing according to the relative position between the user and the user, synthesizes each of the audio data of other conference room users subjected to the stereophonic sound processing, and generates stereo-synthesized sound data;
Voice control means for outputting the three-dimensional synthesized voice data generated by the voice synthesis means from a speaker;
The presence server
While managing the virtual position information of the users sent from each of the plurality of conference terminals, and based on the relay device through which the information sent from each of the plurality of conference terminals has passed, Management means for managing user location information;
For each of the plurality of conference terminals, there is provided location information transmission means for transmitting the virtual location information and location information of the user of each conference terminal managed by the management means to the conference terminal. Voice conference system. The audio conference system according to claim 1,
The relay device is
In accordance with a request from a conference terminal that uses the own relay device, information on the own relay device is transmitted to the presence server as location information of the user of the conference terminal. The audio conference system according to claim 1 or 2,
Each of the plurality of conference terminals includes:
It further has position information detecting means for detecting the position and orientation of the own user with respect to the origin provided in the location where the own conference terminal exists,
The virtual position information transmission is
The audio conference system, wherein the position and orientation detected by the position information detection means are set as the position and orientation of the user in the virtual space, and the user's virtual position information is transmitted to the presence server. An audio conference system,
A plurality of conference terminals; a presence server that manages a location of each of the plurality of conference terminals in a virtual space and a location in real space; a voice server that transmits voice data to each of the plurality of conference terminals; A plurality of relay devices installed at each location in real space, and used by a conference terminal existing at the location to communicate with the presence server,
Each of the plurality of conference terminals includes:
Virtual position information transmitting means for transmitting to the presence server virtual position information of the own user including the position and direction of the own user in the virtual space, which is the user of the self-conference terminal;
Voice data transmitting means for transmitting the voice data of the own user to the voice server;
Three-dimensional synthesized voice data receiving means for receiving three-dimensional synthesized voice data from the voice server;
Voice control means for outputting the stereo synthesized voice data received by the stereo synthesized voice data receiving means from a speaker;
The voice server is
Position information receiving means for receiving, from the presence server, virtual position information of each user of the plurality of conference terminals and location information indicating a location in real space;
For each of the plurality of conference terminals, based on the location information of the user of each conference terminal received by the location information receiving means, exists in a location different from the location in the real space of the target user who is the user of the conference terminal Other meeting room user detecting means for detecting another meeting room user who is a user to perform,
Voice data receiving means for receiving voice data of a user of the conference terminal from each of the plurality of conference terminals;
For each of the plurality of conference terminals, the virtual location information of the other conference room user with respect to the audio data of each other conference room user for the target user who is the user of the conference terminal detected by the other conference room user detecting means And the audio of the other conference room user subjected to the stereophonic sound processing, which is specified by the virtual position information of the target user and subjected to the stereophonic sound processing according to the relative position between the target user and the other conference room user in the virtual space. A voice synthesis means for synthesizing each of the data to generate three-dimensional synthesized voice data;
For each of the plurality of conference terminals, there is stereo synthesized speech data transmitting means for transmitting to the conference terminal stereo synthesized speech data for the target user of the conference terminal generated by the speech synthesizer,
The presence server
While managing the virtual location information of the users sent from each of the plurality of conference terminals, and based on the relay device through which the information sent from each of the plurality of conference terminals has passed, each of the plurality of conference terminals Management means for managing location information;
A voice conference system comprising: position information transmitting means for transmitting virtual position information and location information of a user of each conference terminal managed by the management means to the voice server. The audio conference system according to claim 4,
The relay device is
In accordance with a request from a conference terminal that uses the own relay device, information on the own relay device is transmitted to the presence server as location information of the user of the conference terminal. The audio conference system according to claim 4 or 5,
Each of the plurality of conference terminals includes:
It further has position information detecting means for detecting the position and orientation of the own user with respect to the origin provided in the location where the own conference terminal exists,
The virtual position information transmitting means includes
The audio conference system, wherein the position and orientation detected by the position information detection means are set as the position and orientation of the user in the virtual space, and the user's virtual position information is transmitted to the presence server. A conference terminal,
The user's virtual location information, including the location and orientation of the user who is the user of the conference terminal, in the virtual space, and the location and real space of the user of each of the plurality of conference terminals participating in the audio conference Virtual location information transmitting means for transmitting to a presence server for managing the location on
Position information receiving means for receiving, from the presence server, virtual position information of a user of each conference terminal and location information indicating a location in real space;
Based on the location information of each conference terminal user received by the location information receiving means, another conference room user detection for detecting a user in another conference room who is a user in a location different from the location in the real space of the own user Means,
And audio data transmission means for transmitting the audio data of its own user, to the conference terminal each of the other conference users the other conference room user detection means detects,
Voice data receiving means for receiving voice data of other conference room users from each conference terminal of the other conference room users detected by the other conference room user detecting means,
The other conference room user in the virtual space specified by the virtual position information of the other conference room user and the virtual position information of the own user for each voice data of the other conference room user received by the voice data receiving means A speech synthesis unit that performs stereophonic sound processing according to the relative position between the user and the user, synthesizes each of the audio data of other conference room users subjected to the stereophonic sound processing, and generates stereo-synthesized sound data;
And a voice control means for outputting the three-dimensional synthesized voice data generated by the voice synthesis means from a speaker. A computer-readable program,
The program is a computer,
The user's virtual location information, including the location and orientation of the user who is the user of the conference terminal, in the virtual space, and the location and real space of the user of each of the plurality of conference terminals participating in the audio conference Virtual location information transmitting means for transmitting to a presence server for managing the location on
Position information receiving means for receiving, from the presence server, virtual position information of a user of each conference terminal and location information indicating a location in real space;
Based on the location information of each conference terminal user received by the location information receiving means, another conference room user detection for detecting a user in another conference room who is a user in a location different from the location in the real space of the own user Means,
And audio data transmission means for transmitting the audio data of its own user, to the conference terminal each of the other conference users the other conference room user detection means detects,
Voice data receiving means for receiving voice data of other conference room users from each conference terminal of the other conference room users detected by the other conference room user detecting means,
The other conference room user in the virtual space specified by the virtual position information of the other conference room user and the virtual position information of the own user for each voice data of the other conference room user received by the voice data receiving means A speech synthesis unit that performs stereophonic sound processing according to the relative position between the user and the user, synthesizes each of the audio data of other conference room users subjected to the stereophonic sound processing, and generates stereo-synthesized sound data;
A computer-readable program that functions as a conference terminal having voice control means for outputting the three-dimensionally synthesized voice data generated by the voice synthesis means from a speaker. An audio server that transmits audio data to each of a plurality of conference terminals,
The virtual location information of each user of the plurality of conference terminals and the location in the real space from the presence server that manages the location in the virtual space and the location in the real space of each of the conference terminals participating in the audio conference Position information receiving means for receiving location information indicating
For each of the plurality of conference terminals, based on the location information of the user of each conference terminal received by the location information receiving means, exists in a location different from the location in the real space of the target user who is the user of the conference terminal Other meeting room user detecting means for detecting another meeting room user who is a user to perform,
Voice data receiving means for receiving voice data of a user of the conference terminal from each of the plurality of conference terminals;
For each of the plurality of conference terminals, the virtual location information of the other conference room user with respect to the audio data of each other conference room user for the target user who is the user of the conference terminal detected by the other conference room user detecting means And the audio of the other conference room user subjected to the stereophonic sound processing, which is specified by the virtual position information of the target user and subjected to the stereophonic sound processing according to the relative position between the target user and the other conference room user in the virtual space. A voice synthesis means for synthesizing each of the data to generate three-dimensional synthesized voice data;
Stereo-synthesized speech data transmitting means for transmitting, to each of the plurality of conference terminals, stereo-synthesized speech data for the target user of the conference terminal generated by the speech synthesizer, to the conference terminal. Voice server. A computer-readable program,
The program is a computer,
The virtual location information of each user of the plurality of conference terminals and the location in the real space from the presence server that manages the location in the virtual space and the location in the real space of each of the conference terminals participating in the audio conference Position information receiving means for receiving location information indicating
For each of the plurality of conference terminals, based on the location information of the user of each conference terminal received by the location information receiving means, exists in a location different from the location in the real space of the target user who is the user of the conference terminal Other meeting room user detecting means for detecting another meeting room user who is a user to perform,
Voice data receiving means for receiving voice data of a user of the conference terminal from each of the plurality of conference terminals;
For each of the plurality of conference terminals, the virtual location information of the other conference room user with respect to the audio data of each other conference room user for the target user who is the user of the conference terminal detected by the other conference room user detecting means And the audio of the other conference room user subjected to the stereophonic sound processing, which is specified by the virtual position information of the target user and subjected to the stereophonic sound processing according to the relative position between the target user and the other conference room user in the virtual space. A voice synthesis means for synthesizing each of the data to generate three-dimensional synthesized voice data;
As a voice server having, for each of the plurality of conference terminals, stereo synthesized voice data transmitting means for transmitting to the conference terminal stereo synthesized voice data for the target user of the conference terminal generated by the voice synthesizing means. A computer-readable program characterized by causing it to function.

Images