JP2018067157A - Communication device and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate communication connection setting between users in a virtual space.SOLUTION: A communication device used for a user to communicate with one or more other users who the user shares a virtual space with includes: voice acquisition means for acquiring voice uttered by the user; acquisition means for acquiring positions of the user and the one or more other users in the virtual space; determination means for determining other users whose distance from the user in the virtual space is equal to or less than a prescribed threshold value; and control means for performing communication connection setting to set the other users determined by the determination means as transmission destinations of voice acquired by the voice acquisition means.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a communication control technique between users sharing a virtual space.

  Mixed reality (MR) technology and augmented reality (AR) technology are known as technologies that fuse real space and virtual space in real time and present them to the user. In these techniques, a composite image is displayed by superimposing a virtual space image (CG: Computer Graphic) corresponding to the position and orientation of the display device on the real space video. An MR / AR system equipped with a microphone and speakers has also been developed. For example, by sharing the same virtual space with multiple users, it is possible for even users at remote locations to communicate as if they are in the same room, and application to various fields is expected .

  Patent Document 1 discloses a voice communication system that can grasp the relative position and direction of a communication partner in real space as a body sensation. Specifically, a technique for controlling the acoustic effect based on the position of the user in the virtual space calculated based on the position of the user in the real space is described.

Japanese Patent Laid-Open No. 2005-341092

  However, in a system in which a plurality of users share the same virtual space as described above, communication connection settings for communicating with a specific partner are time-consuming or difficult for the user. Therefore, as a result of setting communication connection with an unintended partner sharing the same virtual space, smooth communication is hindered. Such a problem cannot be solved by the technique disclosed in Patent Document 1 described above.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a technique for facilitating communication connection settings between users in a virtual space.

In order to solve the above-described problems, a communication apparatus according to the present invention has the following configuration. That is, a communication device used for a user to communicate with one or more other users who share a virtual space is:
Voice acquisition means for acquiring voice emitted by the user;
Obtaining means for obtaining the positions of the user and the one or more other users in the virtual space;
Determining means for determining other users whose distance from the user is equal to or less than a predetermined threshold in the virtual space;
Control means for performing communication connection setting with the other user determined by the determining means as the transmission destination of the voice acquired by the voice acquiring means;
Have

  ADVANTAGE OF THE INVENTION According to this invention, the technique which makes easy the communication connection setting between the users in virtual space can be provided.

It is a block diagram which shows the structure of the image audio processing system in 1st Embodiment. It is a key map showing the position and posture of a plurality of users in virtual space. It is an operation | movement flowchart of the communication connection setting in 1st Embodiment. It is a block diagram which shows the structure of the image audio | voice processing system in 2nd Embodiment. It is a key map showing the position and posture of a plurality of users in virtual space. It is an operation | movement flowchart of the communication connection setting in 2nd Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of a communication device according to the present invention, a user device (display device 100 and control device 101) used in an audio / video processing system will be described below as an example.

<Device configuration>
FIG. 1 is a block diagram showing a configuration of an image / audio processing system according to the first embodiment. In particular, a user device prepared for each user is a device used for a user to communicate with one or more other users who share a virtual space, and includes a display device 100 and a control device 101. The Each user apparatus is connected to be communicable with each other via a network. In addition, a position and orientation server 111 and an audio server 112 described later are arranged on the network.

  The display device 100 includes a position / orientation detection unit 102, a display unit 103, an audio input unit 104, and an audio output unit 105. The display device 100 is configured as an HMD (Head Mounted Display), for example. For example, the position and orientation of the user wearing the display device 100 is detected, and a CG (Computer Graphics) video in a virtual space (virtual reality space) is displayed based on the position and orientation. Thereby, the user wearing the display device 100 can observe the virtual reality image in which the CG in the virtual space corresponding to the line-of-sight direction in the real space is superimposed and displayed. In order to generate a stereo image, two display units for the right eye and the left eye may be mounted. The display unit 103 includes, for example, a display element, an optical system, and an image processing unit, and the image processing unit includes a DSP (Digital Signal Processor) or the like.

  In the following description, the display device 100 is assumed to be an HMD, but may be a tablet terminal or other display device. That is, any display device that can be carried and can display an image corresponding to the visual field of the user can be used. The position / orientation detection unit 102 is a functional unit for acquiring the position of the user, and acquires the position of the user in the real space. For example, a position / orientation sensor using an acceleration sensor and an angular velocity sensor, an orientation sensor using geomagnetism, and an attitude detection sensor using GPS can be used. Furthermore, it may be configured to calculate a position and orientation by detecting a marker placed in the real space from an image captured by an imaging sensor (not shown). In this case, it is preferable to configure the display device 100 with a video see-through HMD.

  As the display unit 103, a small liquid crystal display, an organic EL display, or a retinal scan type device using MEMS is used. The voice input unit 104 is a functional unit for voice acquisition such as a microphone. Here, the configuration is shown as the configuration within the display device 100, but any configuration may be used as long as it can acquire the sound generated by the user wearing the display device 100, and the configuration may be configured separately. Here, the sound output unit 105 is assumed to be an earphone-type speaker. This is because, when there are a plurality of users wearing the display device 100 in the same real space (for example, a room), if a normal speaker is used, the direct sound from the speaker may be heard by other nearby users. Because there is. Of course, other types of speakers may be used as long as the user can hear the sound directly in the vicinity, for example, when there is only one user in the room.

  The control device 101 includes a position / orientation transmission unit 106, a virtual reality object calculation unit 107, a control unit 108, an audio transmission unit 109, and a relative distance calculation unit 110. Here, it is assumed that the control device 101 uses a device having a high-performance arithmetic processing function or graphic display function such as a personal computer or a workstation. By connecting the control device 101 and the display device 100, a user wearing the display device 100 can view a video in the virtual space. The video of the virtual space may be a mixed real world video in which the real world and the virtual world are seamlessly fused in real time. In addition, you may comprise so that a user can carry the control apparatus 101 and the display apparatus 100 as an integral structure.

  The position and orientation transmission unit 106 transmits the position and orientation information to the position and orientation server 111 that manages the position and orientation of the user in the virtual space in order to share the position and orientation information acquired by the position and orientation detection unit 102 with other users. . In addition, the position and orientation information of other users is received. The virtual reality object calculation unit 107 performs CG drawing of the virtual reality object viewed from a specific direction based on the information indicating the position and orientation of the user acquired from the position and orientation detection unit 102.

  The control unit 108 performs control for setting a communication partner of the audio transmission unit 109. The control unit 108 operates by setting one of the following three modes as the transmission mode. Details of the operation will be described later.

A first mode (individual transmission mode) for transmitting / receiving only to some users in the virtual space (for example, short-distance users)
Second mode for transmitting / receiving to / from all users in the virtual space (all-user transmission mode)
・ Third mode that does not transmit / receive (non-transmission mode)
As a transmission mode, another mode, for example, a mode that preferentially transmits to a short-distance user in a virtual space may be further provided.

  When receiving an instruction from the control unit 108, the voice transmission unit 109 transmits the voice data input from the voice input unit 104 to the voice server 112. Also, the other user's voice data acquired from the voice server 112 is output to the voice output unit 105.

  The relative distance calculation unit 110 calculates a relative distance between the own user and another user in the virtual space. For example, the relative position of the other user and the position of the own user from the same virtual reality object are acquired from the position / orientation server 111 via the position / orientation transmission unit 106. Then, the calculated distance between users is notified to the control unit 108. When there are a plurality of other users, the distance from each other user is calculated.

  The position / orientation server 111 is a device that manages position / orientation information of each user sharing the same virtual space. For example, the position and orientation information in the real space is received from the control device 101 of each user, and the position and orientation in the virtual space corresponding to the position and orientation in the real space are calculated and managed. For example, a position and orientation based on a location (room) in the real space where the user exists is mapped to a position and orientation based on a predetermined virtual space designated by the user. In addition, based on the acquisition request from the control device 101 of each user, the position and orientation information of each user in the managed virtual space is returned.

  The voice server 112 is a device that manages voice data of users who share the same virtual space. The voice data is transmitted to another user designated based on the transmission request from the control device 101 of each user. Each user's voice data may be stored and managed in a storage area assigned to each user, or the source / destination information may be added to the voice data and stored and managed. .

<Operation of the device>
Below, the transmission process of the audio | speech data between the some users who share the same virtual space is demonstrated. FIG. 2 is a conceptual diagram showing the positions and orientations of a plurality of users in the virtual space.

  Users 200, 201, and 202 are users (avatars) who share the same virtual space and are each wearing the display device 100 in the real space. In the virtual space, the object 203 exists in addition to the three users.

  The users in the real space corresponding to the users 200, 201, and 202 may exist in the same real space (for example, the same room), or may exist in a remote place. Hereinafter, the users in the real space corresponding to the users 200, 201, and 202 in the virtual space are expressed as users A, B, and C, respectively. In addition, the explanation will be made assuming that the users A, B, and C exist in remote places.

  Here, it is assumed that these three users share a virtual object or the like through video and make a call. Specifically, it is assumed that the user 200 talks to the user 201 in the virtual space (situation in FIG. 2). That is, the relative position from the object 203 is similar between the user 200 and the user 201 in the virtual space. In addition, the user 200 and the user 201 exist in close positions in the virtual space.

  Thus, when the relative position of the user 200 and the user B201 is closer than a certain threshold (distance is short), the voice uttered by the user A (user 200) is transmitted only to the user B (user 201). That is, audio data is not transmitted to the user 202 whose relative position from the user 200 is far (long distance).

  In the first embodiment, when there are a plurality of other users in addition to the user 201 in the vicinity of the user 200, the voice data is transmitted to all users whose user distance is equal to or less than a certain threshold. As described above, the audio data is transmitted only to the user whose relative distance derived from the relative position from the virtual reality object is closer than a certain threshold.

  Note that the above-described operation is an operation when the first mode is set as the transmission mode of the control unit 108. Here, since only the user 201 is spoken, it is assumed that the user A (user 200) sets in advance such that the control unit 108 operates in the “first mode”. However, when it is desired to transmit voice information to all users, the user A himself / herself changes the transmission mode to the “second mode”, so that the voice is transmitted to all users regardless of the distance between users. Further, when it is not desired to perform voice transmission / reception with any user, all transmission / reception is suppressed by changing the transmission mode to “third mode” by the user A himself / herself. The transmission mode may be changed by the user operating a button or switch (not shown) arranged on the display device 100, or may be changed by other operations.

  FIG. 3 is an operation flowchart of communication connection setting in the first embodiment.

  In step S <b> 300, the control apparatus 101 transmits the position / orientation information detected by the position / orientation detection unit 102 to the position / orientation transmission unit 106. The voice information input from the voice input unit 104 is transmitted to the voice transmission unit 109.

  In step S <b> 301, the control apparatus 101 transmits the position / orientation information acquired in step S <b> 300 to the position / orientation server 111. In step S <b> 302, the control apparatus 101 acquires position / orientation information of another user from the position / orientation server 111. Here, relative position and orientation information from the virtual reality object 203 of all other users (user B and user C) is acquired.

  In S303, the control apparatus 101 calculates the distance between the own user (user A) and each of the other users (user B, user C) based on the position and orientation information of each user.

  In step S <b> 304, the control device 101 acquires the transmission mode currently set in the control unit 108. here. As described above, there are three transmission modes: the first mode to the third mode.

  In S305, the control apparatus 101 determines whether the control unit 109 performs the transmission / reception settings individually or the transmission / reception settings for the entirety based on the transmission mode acquired in S304. When the first mode is set and the transmission / reception setting is individually performed, the process proceeds to S306. If the second mode is set and transmission / reception settings are to be made for the entirety, the process advances to step S311.

  In S <b> 306, the control apparatus 101 determines whether another user exists at a short distance equal to or less than a predetermined threshold with respect to the user 200 based on the distance between users acquired in S <b> 303. Here, if there are one or more other users at a short distance, the process proceeds to S307, and settings for individually transmitting and receiving audio information are performed. On the other hand, if there is no other user at a short distance, the process proceeds to S311.

  In step S307, the control apparatus 101 performs communication connection settings for other users determined as transmission destinations. In addition, when transmitting to several other users, it is good to perform communication connection setting separately. The contents of the communication connection settings may include various settings such as audio data compression, time control using a time stamp, etc., and transmission control according to communication conditions. In S308, the control apparatus 101 transmits the audio data to the audio server 112 based on the communication connection setting set in S307.

  In step S309, the control apparatus 101 draws and generates the virtual reality object 203 as a CG viewed from a specific position and orientation based on the position and orientation acquired in step S300. In S310, the control apparatus 101 displays the CG generated in S309 on the display unit 103 of the HMD. Here, the virtual reality image displayed on the display unit 103 and the audio data output from the audio output unit 105 may be synchronized using a time stamp or the like. Of course, the output may be performed without performing the synchronization control. Moreover, you may comprise so that the information regarding the other user who transmits audio | voice data may be displayed collectively.

  In step S311, the control apparatus 101 determines whether or not to transmit audio data to all users based on the set transmission mode. When the second mode is set and transmission is to all users, the process proceeds to S312. On the other hand, if the third mode is set and no audio data is transmitted to anyone, the process proceeds to S309.

  In step S312, the control apparatus 101 performs communication connection settings for all users (user B and user C). Note that multicast / broadcast transmission may be set to be used. Since the specific processing is the same as that in S307, description thereof is omitted. In S313, the control apparatus 101 transmits the audio data to the audio server 312 based on the communication connection setting set in S312.

  As described above, according to the first embodiment, the user 200 can easily perform communication connection setting for transmitting voice data only to a user at a short distance. That is, the user 200 only needs to set the first mode as the transmission mode of the control unit 108 and approach the user 201 in the virtual space.

  In the above description, the position / orientation of the user in the real space is acquired by the position / orientation detection unit 102 arranged in the display device 100. However, the position and orientation of the user in the real space may be acquired by an external device (for example, a camera installed in a room), and the position and orientation of the user may be registered in the position and orientation server 111. In the above description, the configuration for transmitting voice data has been described. However, the configuration for transmitting text message data may be used.

(Second Embodiment)
In the second embodiment, a mode in which communication connection setting is performed in consideration of not only the position of each user but also the posture (line-of-sight direction) will be described. That is, in the first embodiment, in a situation where there are a plurality of other users at a short distance, the audio data is transmitted to all the plurality of other users. On the other hand, depending on the user's intention, it may be possible to transmit / receive audio data only to a specific user. Therefore, in the second embodiment, a mode will be described in which another user having the same gazing point as the own user is determined based on the position and orientation, and the audio data is transmitted only to the other user.

<Device configuration>
FIG. 4 is a block diagram showing the configuration of the audio / video processing system in the second embodiment. It is the structure which added the relative attitude | position calculating part 400 with respect to 1st Embodiment (FIG. 1). A description of functional units similar to those in the first embodiment will be omitted.

  The relative posture calculation unit 400 calculates the relative position and posture of each user with respect to the same virtual reality object. Further, the control unit 108 determines another user having a gazing point having a high degree of coincidence with the gazing point of the own user as a transmission destination.

<Operation of the device>
Below, the transmission process of the audio | speech data between the some users who share the same virtual space is demonstrated. FIG. 5 is a conceptual diagram showing the positions and orientations of a plurality of users in the virtual space.

  Users 200, 201, and 202 are users wearing the display device 100 that shares the same virtual space, as in the first embodiment. In the second embodiment (FIG. 5), a user 500 facing the same direction as the user 200 is further added.

  That is, in the second embodiment, an example is shown in which the virtual reality object 203 observed by the user 200 is the other party who observes the virtual reality object 203 in the same manner near the user 200 as a communication partner. Here, it is assumed that the user 500 is specified as a communication partner.

  In the first embodiment, the user 200 starts a call just by approaching the user 201. In the second embodiment, the user 200 approaches the user 201 and the user 500, and the user 200 and the user 500 have the same virtual reality object 203 as the point of sight. In this way, the audio data is transmitted only to other users having the same gazing point.

  The user 201 is close to the user 200 but has a different relative posture. Therefore, the voice data of the user 200 is not transmitted to the user 201. The user 202 uses the virtual reality object 203 as a gazing point, but is relatively far from the user 200 and has a different relative posture. Therefore, the voice data of the user 200 is not transmitted to the user 202.

  FIG. 6 is an operation flowchart of communication connection setting in the second embodiment. Note that S300 to S302 and S304 to S313 in FIG. 6 are the same processes as S300 to S302 and S304 to S313 in FIG. 6 is different from S303 of FIG. 3 in that S600 exists and the processing of S601 is added.

  In S600, the control apparatus 101 analyzes the position / orientation information of the other user acquired in S302. Specifically, in addition to the relative distance calculated by the relative distance calculation unit 110, the relative posture calculation unit 400 calculates a relative posture. Thereafter, a transmission method is set based on the position and orientation information, the audio data acquired in S300, and the current transmission setting acquired in S304.

  In S <b> 601, the control apparatus 101 uses the position / orientation information calculated in S <b> 600 to select a user who is closer than a threshold with a relative position / orientation with respect to the virtual reality object as another user of the transmission destination. That is, here, a user whose distance (positional difference) is equal to or smaller than a predetermined threshold and whose posture difference is equal to or smaller than the predetermined difference is regarded as a user having the same gazing point.

  As described above, according to the second embodiment, the user 200 can easily perform communication connection setting for transmitting audio data only to users having the same point of sight. That is, the user 200 only needs to set the first mode as the transmission mode of the control unit 108, approach the user 500 in the virtual space, and face the same direction (match the posture). In the above description, other users having the same gazing point are determined by regarding users who are close in position and close in posture as having the same gazing point. However, a specific gaze point may be derived for each user from the position and orientation of each user and information in the virtual space, and the degree of coincidence may be compared.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 100 Display apparatus; 101 Control apparatus; 102 Position and orientation detection part; 103 Display part; 104 Voice input part; 105 Sound output part; 106 Position and attitude transmission part; 107 Virtual reality object calculation part; 108 Control part; 110 Relative distance calculation unit; 111 Position and orientation server; 112 Voice server

Claims (10)

A communication device used for a user to communicate with one or more other users sharing a virtual space,
Voice acquisition means for acquiring voice emitted by the user;
Obtaining means for obtaining the positions of the user and the one or more other users in the virtual space;
Determining means for determining other users whose distance from the user is equal to or less than a predetermined threshold in the virtual space;
Control means for performing communication connection setting with the other user determined by the determining means as the transmission destination of the voice acquired by the voice acquiring means;
A communication apparatus comprising: The communication device according to claim 1, wherein the acquisition unit acquires the position of each user in the virtual space in the virtual space from a first server that manages the position. The communication device
Position acquisition means for acquiring the position of the user in real space;
Transmitting means for transmitting the position acquired by the position acquiring means to the first server;
Further comprising
The first server derives a position of the user in the virtual space based on the position of the user in the real space received from the user, and manages the position as the position of the user in the virtual space. The communication device according to claim 2. The first server is further configured to manage the posture of each user in the virtual space in the virtual space;
The acquisition means further acquires postures of the user and the one or more other users in the virtual space,
The determining means determines another user whose distance from the user in the virtual space is equal to or smaller than a predetermined threshold and whose posture difference from the user is equal to or smaller than a predetermined difference.
The control means determines the other user whose distance to the user is equal to or smaller than the predetermined threshold and whose difference in attitude with the user is equal to or smaller than the predetermined difference by the determination means. The communication apparatus according to claim 2, wherein communication connection setting as a transmission destination is performed. Generating means for generating an image of the virtual space based on the position and posture of the user and the one or more other users in the virtual space acquired by the acquiring means;
Display means for displaying the video generated by the generating means;
The communication device according to claim 4, further comprising: It further has an imaging means for imaging the real space,
The communication device according to claim 5, wherein the generation unit generates a video of the virtual space by superimposing a video of an object arranged in the virtual space on a video captured by the imaging unit. . The control means transmits the voice data acquired by the voice acquisition means and information on the destination of the voice to a second server that manages voice transmission in the virtual space. Item 7. The communication device according to any one of Items 1 to 6. The control means operates in a first mode in which data is transmitted to some users in the virtual space, or operates in a second mode in which data is transmitted to all users in the virtual space. And a selection means for selecting
When the operation in the second mode is selected by the selection unit, the control unit is configured to acquire all the users in the virtual space by the voice acquisition unit regardless of the determination by the determination unit. The communication apparatus according to any one of claims 1 to 7, wherein communication connection setting as a transmission destination is performed. A method of controlling a communication device used for a user to communicate with one or more other users sharing a virtual space,
An audio acquisition step of acquiring audio emitted by the user;
An acquisition step of acquiring the positions of the user and the one or more other users in the virtual space;
A determination step of determining other users whose distance from the user is equal to or less than a predetermined threshold in the virtual space;
A control step of performing communication connection setting with the other user determined in the determination step as a transmission destination of the voice acquired in the voice acquisition step;
A method for controlling a communication apparatus, comprising:   The program for functioning a computer as each means of the communication apparatus of Claim 1.

Images