JP6526879B1 - Data transmission device and program - Google Patents

Abstract

When sharing a virtual experience among a plurality of users, the traffic volume of voice data is suppressed or damage to the virtual experience of the users is prevented. A data transmission apparatus includes a data acquisition unit, a condition determination unit, and a transmission control unit. The data acquisition unit acquires first position data indicating a first position of the first avatar in the virtual space, and audio data of the first avatar. The condition determination unit determines whether a predetermined condition regarding the first position is satisfied. The transmission control unit determines to transmit voice data to the destination terminal when it is determined that the predetermined condition is satisfied, and directs the voice data to the destination terminal when it is determined that the predetermined condition is not satisfied. Decide not to send. [Selected figure] Figure 1

Description

  The present invention relates to VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality) technology.

  Conventionally, it is known to share a virtual experience, that is, a VR experience, via a network among a plurality of users wearing a head mounted display (HMD). Specifically, it is known to realize voice chat (VR chat) between users (between avatars) in a virtual space ([0081] of Patent Document 1).

  In Patent Document 1, "send voice data (voice data of user A) indicating voice of user A to server 2" ([0076]), and "control unit 23 of server 2 receives from user terminal 1A The volume parameter L related to the voice data of the user A is set (step S15), where the volume parameter L is the volume of the voice of the user A output to the headphones 116 (voice output unit) of the user terminal 1B (sound ([0078]), "if the avatar 4B is located in the high transmission area in the virtual space, set the volume parameter L to the volume parameter L1", and "the avatar If the 4B is not located in the high transfer area in the virtual space, set the volume parameter L to the volume parameter L2 "(summary) Transmitting the voice data of the user A and the volume parameter L to the user terminal 1B "([0079])," the control unit 121 of the user terminal 1B transmits the voice data of the user A and the volume parameter L, Outputting the voice of the user A to the headphone 116 ([0081]) and the like are disclosed.

Patent No. 6289703 gazette

  In order to realize sharing of an auditory virtual experience such as voice chat among a plurality of users in a virtual space, voice data may be transmitted from a terminal of each user to another terminal. When voice data is exchanged between terminals in this manner, the traffic volume of voice data increases as the number of users sharing the virtual experience increases. In other words, the traffic volume of voice data may be one factor that limits the maximum number of users that can be accommodated in the virtual space. This can occur in all experiences including virtual elements such as AR or MR (hereinafter referred to as virtual experiences) as well as VR experiences.

  In addition, if the number of users sharing the virtual experience is too large, for example, the voice of the avatar of a certain user may be buried in the voice of the avatar of another user who uttered at the same time, which may make it difficult to hear. This is more serious, for example, when holding an event such as live in a virtual space. That is, the talks and singing voices of major avatars such as performers, which are being watched by many users, are buried in the reaction of other avatars such as spectators, such as laughter, cheers, applause, hiji, etc., and the user's virtual Personal experience may be impaired.

  An object of the present invention is to reduce traffic volume of voice data or prevent damage to a user's virtual experience when sharing virtual experiences among a plurality of users.

  A data transmission apparatus according to an aspect of the present invention includes an acquisition unit, a determination unit, and a transmission control unit. An acquisition part acquires the 1st position data which show the 1st position in virtual space of the 1st avatar, and the voice data of the 1st avatar. The determination unit determines whether a predetermined condition regarding the first position is satisfied. The transmission control unit determines to transmit voice data to the destination terminal when it is determined that the predetermined condition is satisfied, and directs the voice data to the destination terminal when it is determined that the predetermined condition is not satisfied. Decide not to send.

  A terminal according to another aspect of the present invention includes an acquisition unit, a determination unit, and a transmission control unit. An acquisition part acquires the 1st position data which show the 1st position in virtual space of the 1st avatar, and the voice data of the 1st avatar. The determination unit determines whether a predetermined condition regarding the first position is satisfied. The transmission control unit determines to transmit voice data to the destination terminal via the peer-to-peer network when it is determined that the predetermined condition is satisfied, and it is determined that the predetermined condition is not satisfied. In this case, it is decided to transmit voice data to the destination terminal via the client / server network.

  According to the present invention, when sharing a virtual experience among a plurality of users, it is possible to suppress the traffic volume of voice data or prevent the damage of the virtual experience of the user.

The block diagram which illustrates the terminal concerning an embodiment. FIG. 2 is a block diagram illustrating a virtual experience sharing system including the terminal of FIG. 1; Explanatory drawing of default condition (1). Explanatory drawing of default condition (2). Explanatory drawing of default condition (3). Explanatory drawing of default condition (4). 5 is a flowchart illustrating the operation of the terminal of FIG. The block diagram which illustrates the virtual experience sharing system containing the terminal concerning modification 1. 10 is a flowchart illustrating an operation of a terminal according to a modification 1; FIG. 8 is a block diagram illustrating a server according to a second modification;

  Hereinafter, the description of the embodiment will be described with reference to the drawings. Hereinafter, elements which are the same as or similar to the already described elements are denoted by the same or similar reference numerals, and redundant descriptions will be basically omitted. For example, when there are a plurality of identical or similar elements, a common code may be used to explain each element without distinction, and the common code may be used to distinguish and describe each element. In addition, branch numbers may be used.

(Embodiment)
The terminal according to the embodiment can configure a virtual experience sharing system. Such a system is illustrated in FIG. In this system, the terminals 100 are connected to each other via a network such as the Internet, for example, and can transmit and receive data.

  In the system of FIG. 2, instead of a C / S (Client / Server) type network in which the terminals 100 transmit and receive data via a server, a P2P (Peer to Peer) type in which the terminals 100 transmit and receive data directly Networks have been adopted. However, even if the C / S type network is employed instead of or in addition to the P2P type network, the virtual experience sharing system is implemented using the terminal 100 and the server according to the present embodiment. It is possible to build.

  Here, the P2P type network has various merits as compared with the C / S type network. Specifically, the merit is that the delay associated with data transmission is reduced, so it is suitable for sharing a virtual experience with high real-time capability, and since there is no server, it is concentrated on servers in C / S type networks. And the ability to distribute traffic and load, the absence of a single point of failure, and so on.

  On the other hand, when a P2P type network is adopted, the number of links becomes n × (n−1) / 2 with respect to the number of users n, and increases non-linearly. Therefore, in a P2P network, the traffic volume of voice data increases explosively as the number of users usually increases, so the traffic volume problem of voice data is more serious than when a C / S network is adopted. It can be

  The terminal 100 is a VR / AR / MR image displayed on the HMD 10 (first device) attachable to the head of the user, and a speaker (which may be built in the HMD 10 or separately from the HMD 10 ) Is a computer configured to control the VR / AR / MR sound output by the body), and is connected to the HMD 10 and the controller 20 (second device) which can be held by the user. It is done. The controller 20 may be prepared two (for both hands) or one (for one hand) for each user. In addition, the controller 20 may be attached to a part other than the hand. Furthermore, although not shown in FIG. 2, a base station, which is one of the elements included in the position sensor system for detecting the position of the HMD 10 and / or the controller 20, may be further connected to the terminal 100.

  The terminal 100 may be connected to the HMD 10, the controller 20, and / or the base station by a wired communication means such as a Universal Serial Bus (USB) cable, a High-Definition Multimedia Interface (HDMI) (registered trademark) cable, etc. For example, it may be connected by wireless communication means such as Bluetooth (registered trademark), WirelessHD, WHDI (Wireless Home Digital Interface), or the like.

  The terminal 100 transmits voice data generated by the microphone to another terminal 100 (destination terminal) with predetermined conditions, as described later. The other terminal 100 reproduces and outputs this voice data. As a result, it is possible to realize realistic events such as voice chat between users or live in the VR / AR / MR space while suppressing the traffic volume of voice data.

  Also, in order to dynamically control the attitude of the avatar, the terminal 100 acquires control data having values based on the position of the HMD 10 and the position of the controller 20 as described later, and the control data is transmitted to the other terminal 100. You may transmit toward (destination terminal). Control data determine the attitude of the avatar.

  The other terminal 100 receiving the control data generates an avatar image of the user based on the received control data. That is, the user can make the avatar which is his / her second body take a free posture by moving the head wearing the HMD 10 or the hand holding the controller 20.

  Specifically, the other terminal 100 determines the position of the head of the avatar image according to the position of the HMD 10 indicated by the received control data, and the hand of the avatar image according to the position of the controller 20 indicated by the control data. Determine the position of For controlling the posture of the avatar, for example, IK (Inverse Kinematics) technology can be used.

  The HMD 10 may include various sensors, speakers, microphones, and the like in addition to a display device for displaying a VR / AR / MR image including an avatar image. The various sensors may include motion sensors, mounting sensors, or portions of elements included in the position sensor system (markers or cameras described below). The HMD 10 receives image data and voice data from the terminal 100 and outputs them, or sends sensor data of various sensors to the terminal 100 for transmission.

  The display device may be a transmissive display or non-transmissive display. For example, the size and arrangement of the display device are determined to cover at least a part of the field of view of the user wearing the HMD 10. The display device may be configured of a display device for the left eye and a display device for the right eye, or both may be integrated.

  The motion sensor may be, for example, an acceleration sensor, a gyroscope, a magnetic sensor, and the like. Sensor data detected by the motion sensor can be used to estimate the posture (e.g., tilt) of the user's head. Specifically, based on this sensor data, a three-dimensional rotational angle of the user's head, a Yaw angle, a Roll angle, and a Pitch angle, are estimated, and a virtual camera that determines the field of view of the user accordingly. The position of the visual axis of and / or the head of the avatar may be controlled.

  The mounting sensor generates sensor data (event data) indicating that the user has mounted / removed the HMD 10. Although the mechanism of the mounting sensor is arbitrary, for example, the change of the elastic force of the spring provided on the HMD 10, the change of the current flowing between the pads contacting a part of the body such as the user's nose when the HMD 10 is mounted, etc. It is possible to generate event data accordingly.

  The speaker receives audio data from the terminal 100 and outputs an audio based thereon. The speaker is typically a headphone type, but may be configured as another speaker system. The microphone collects voice (which is primarily user speech but may include ambient sounds). The microphone generates voice data based on the collected voice and sends it to the terminal 100.

  The controller 20 may include, for example, a motion sensor similar to the HMD 10, a part of elements included in the position sensor system, and the like, in addition to a button for receiving user input. Furthermore, the controller 20 may include a gripping sensor that generates sensor data (event data) indicating that the user grips / hands off the controller 20. Although the mechanism of the grip sensor is arbitrary, for example, according to the change of the pressure applied to the controller 20 by the user's grip force, the change of the capacitance between the sensor electrode provided on the surface of the controller 20 and the human body, etc. Event data can be generated. The controller 20 transmits user input data, sensor data of various sensors, and the like to the terminal 100.

  The position sensor system is realized by, for example, a combination of a camera (position tracking camera) and a marker (also called a tracker). The marker may be an emitter of infrared light or visible light (for example, an LED (Light Emitting Diode)), or a reflector for reflecting infrared light or visible light emitted from the emitter at the time of shooting by a camera. May be The camera may be an infrared sensor if the marker illuminates or reflects infrared light, and may be a visible light camera if the marker illuminates or reflects visible light.

  Typically, a plurality of markers are attached to the HMD 10 / controller 20, and a camera is attached to a device (base station) installed at a position distant from the HMD 10 / controller 20. The position of the HMD 10 / controller 20 can be estimated based on the captured image of the camera. Specifically, the base station can detect the position, inclination, light emission intensity and the like of the detection point based on the photographed image. The base station may calculate position (coordinates) data of the HMD 10 / controller 20 based on the data of such detection points, or such calculation may be delegated to the terminal 100.

  As a modification, it is also possible to provide a camera on the HMD 10 / controller 20 side and a marker on the base station side. Also, in addition to the markers attached to the HMD 10 and the controller 20, additional markers may be attached to the joints, erasures, etc. of the user. This makes it possible to estimate the posture of the user more accurately.

  The hardware configuration of the terminal 100 will be described below. The terminal 100 may be various electronic devices that can be the control device of the HMD 10, such as a PC (Personal Computer), a mobile terminal (eg, a tablet, a Fablet, a smartphone, a laptop, a feature phone, a wearable device, a portable game machine, etc.) , Stationary game console, etc., but is not limited thereto.

  The terminal 100 may not necessarily be separate from the HMD 10 or another device (such as the controller 20 or the base station). For example, the terminal 100 may be built in the HMD 10 or the base station, or the terminal 100 may be treated as the controller 20.

  The terminal 100 includes, for example, a processor that performs input / output control, communication control (in particular, transmission control of audio data), image / audio processing, predetermined condition determination, and the like. Here, the processor is typically a central processing unit (CPU) and / or a graphics processing unit (GPU), but a microcomputer, a field programmable gate array (FPGA), a digital signal processor (DSP), or any other general purpose Or it may be a dedicated processor or the like.

  The terminal 100 may also execute programs executed by the processor to realize such processing and data used by the processor, for example, various objects for producing a VR / AR / MR experience such as avatars, backgrounds, etc. It contains a memory that temporarily stores the underlying image data. The memory may include a random access memory (RAM) having a work area in which such programs / data are expanded.

  Terminal 100 may handle all data in the on-memory state, or some data may be saved in the auxiliary storage device. The auxiliary storage device may be, for example, a hard disc drive (HDD), a solid state drive (SSD), a flash memory, etc. built in or externally attached to the terminal 100, or a database server accessible from the terminal 100. May be

  The terminal 100 can further use a communication I / F (interface) for connecting to the network. The communication I / F may be incorporated in the terminal 100 or may be externally attached to the terminal 100. The communication I / F is a module for communicating with another terminal 100 and / or an external device such as the HMD 10, the controller 20, a base station, etc., and includes a signal processing circuit for transmission and reception, an antenna, It may include a LAN (Local Area Network) terminal or the like. The communication I / F may be, for example, a module for wide area communication such as mobile communication, a module for wireless / wired LAN, a module for Bluetooth, or the like.

  The terminal 100 can further utilize an input / output I / F for cable connection to an external device such as the HMD 10, the controller 20, a base station, and the like. The input / output I / F is, for example, a USB terminal, a DVI (Digital Visual Interface) terminal, an HDMI terminal, or a terminal for another data transfer cable.

  Terminal 100 may further include a bus for transferring data between each element, for example, a processor, a memory, an auxiliary storage device, a communication I / F, an input / output I / F, and the like.

  The functional configuration of the terminal 100 is illustrated in FIG. The terminal 100 includes an input unit 101, a reception unit 102, a transmission unit 103, an output unit 104, a data acquisition unit 111, a distance calculation unit 112, a condition determination unit 113, a transmission control unit 114, and image generation. And a voice generation unit 116.

  The input unit 101 can be realized by the aforementioned input / output I / F. The input unit 101 receives various data from an external device such as the HMD 10, the controller 20, a base station, a microphone (which may be built in the HMD 10 or may be separate), and the like. The input unit 101 sends the received data to the data acquisition unit 111.

  Specifically, the input unit 101 is an avatar (an avatar associated with the user of the terminal 100, hereinafter referred to as a first avatar) position data indicating a position in a virtual space, a first avatar generated by a microphone Voice data, control data for controlling the posture of the first avatar (which may be integrated with the position data, may be separate data), and the like.

  Here, the position data may be, for example, data indicating the position of the HMD 10 and / or the controller 20, or may be data generated based on these. The same applies to control data. Note that, depending on the relationship between the external device and the terminal 100, part or all of these data may be received by the receiving unit 102 instead of the input unit 101.

  The receiving unit 102 can be realized by the aforementioned communication I / F. The receiving unit 102 receives position data indicating the position in the virtual space of an avatar (an avatar associated with the user of the other terminal 100, hereinafter referred to as another avatar) from another terminal 100 via the network, or other avatars Control data and / or voice data may be received. The receiving unit 102 sends the received data to the data acquisition unit 111.

  The transmission unit 103 can be realized by the aforementioned communication I / F. The transmission unit 103 is controlled by the transmission control unit 114 and transmits voice data, position data and / or control data of the first avatar toward the destination terminal.

  The output unit 104 can be realized by the aforementioned input / output I / F. The output unit 104 outputs data to an external device, for example, the HMD 10, to the speaker, and more specifically, the output unit 104 outputs VR / AR / MR image data to the HMD 10, and outputs VR / AR / MR audio data to the speaker. Output to Furthermore, the output unit 104 may output vibration data for vibrating a vibrator (not shown) for haptic feedback to the HMD 10 and / or the controller 20.

  The data acquisition unit 111 may be realized by the processor described above. The data acquisition unit 111 acquires various data from the input unit 101 and the reception unit 102. The data acquisition unit 111 sends the acquired data to the distance calculation unit 112, the condition determination unit 113, the image generation unit 115, and / or the sound generation unit 116.

  Specifically, in addition to the voice data, position data, and control data of the first avatar, the data obtaining unit 111 can obtain voice data, position data, control data, and the like of other avatars.

  For example, the data acquisition unit 111 sends the position data of the first avatar to the distance calculation unit 112 and the condition determination unit 113, sends the voice data and control data of the first avatar to the condition determination unit 113, and the positions of other avatars. The data may be sent to the distance calculation unit 112, control data of another avatar may be sent to the image generation unit 115, and voice data of the other avatar may be sent to the voice generation unit 116.

  When the external device, for example, the HMD 10, the controller 20 and / or the base station estimates the position of the HMD 10 / controller 20 to generate the position data / control data of the first avatar, the data acquisition unit 111 inputs Position data / control data can be directly obtained from the unit 101 or the receiving unit 102. On the other hand, (the position / posture estimation unit (not shown) of the terminal 100 estimates the position of the HMD 10 / controller 20 based on output data of the position sensor system (for example, data indicating the position, inclination, light emission intensity, etc. of the detection point) If it is necessary to generate position data / control data of the first avatar, the data acquisition unit 111 sends output data of the position sensor system to the position / posture estimation unit to generate position data / control data. You may request it.

  The distance calculation unit 112 may be realized by the processor described above. The distance calculation unit 112 calculates the distance between two points in the virtual space. Specifically, the distance calculation unit 112 can calculate the distance from the reference point in the virtual space to the point where the first avatar or another avatar is present. The reference point may be set at a point where the first avatar or another avatar is present, or may be set at a point where no avatar exists. The distance calculation unit 112 sends the distance data to the condition determination unit 113. In addition, when the predetermined condition mentioned later is a thing irrelevant to the distance between two points in virtual space, the distance calculation part 112 may become unnecessary.

  The condition determination unit 113 can be realized by the processor described above. The condition determination unit 113 can receive the voice data of the first avatar and the control data and / or position data of the first avatar from the data acquisition unit 111, and can further receive distance data from the distance calculation unit 112. The condition determination unit 113 determines whether a predetermined condition regarding the position of the first avatar in the virtual space is satisfied based on the position data and / or distance data of the first avatar. The condition determination unit 113 sends, to the transmission control unit 114, voice data, position data and / or control data of the first avatar together with data indicating the determination result. The predetermined conditions may be defined for the purpose of thinning out audio data to be shared, while preventing damage to the user's virtual experience.

  Specifically, the predetermined condition is (1) that the position of the first avatar in the virtual space (hereinafter referred to as the first position) is in the predetermined zone, (2) the virtual space exists When the positions in the virtual space of a plurality of avatars (which may or may not be all avatars existing in the virtual space) including one avatar are sorted in order of proximity from a predetermined point in the virtual space, (1) The distance between the position in the virtual space of the other avatar associated with the other terminal 100 as the destination terminal and the first position is less than the threshold Or (4) a plurality of avatars excluding the first avatar existing in the virtual space (may be all avatars excluding the first avatar existing in the virtual space) or may not be virtual In space When the distance between the position and the first position is sorted in ascending order, the distance between the position in the virtual space of the other avatar associated with the other terminal 100 as the destination terminal and the first position is predetermined. It may include becoming more than a rank. However, predetermined conditions are not limited to these (1)-(4).

  The predetermined condition (1) will be described with reference to FIG. The default zone may be defined based on the distance from a predetermined point in virtual space. However, the predetermined zone is not limited to this, and may be arbitrarily defined, for example, by specifying the coordinates of the boundary inside and outside the zone. FIGS. 3 and 4 to 6 are bird's eye views looking down on the virtual space, and the circled number marks indicate the position of the avatar (of the head), and for convenience the number is also used as the avatar's code To be. Further, in FIG. 3, the predetermined zone is set to be within a radius of メ ー ト ル meters from the predetermined point represented by the cross mark. In the example of FIG. 3, the position of avatar 1 is not in the predetermined zone, and the position of avatar 2 is in the predetermined zone. Therefore, in the terminal 100 of the user associated with the avatar 1, the condition determination unit 113 determines that the predetermined condition is not satisfied. On the other hand, in the terminal 100 of the user associated with the avatar 2, the condition determination unit 113 determines that the predetermined condition is satisfied. In the predetermined condition (1), there is no distinction between destination terminals, and the determination result is applied in relation to all destination terminals. That is, the user's terminal 100 associated with avatar 1 does not transmit voice data to any other terminal 100, and the user's terminal 100 associated with avatar 2 sends voice data to all other terminals 100.

  The example of FIG. 3 is suitable for a scene in which each user focuses on the behavior of a specific avatar, such as a live event held in a virtual space. According to the example of FIG. 3, the predetermined zone can be regarded as a virtual stage. Since voice data of avatars not placed on the virtual stage is not transmitted to the other terminal 100, voices of avatars placed on the virtual stage are less likely to be buried in voices of other avatars. That is, each user can concentrate on the target avatar's talk and singing voice, and at the same time, can suppress the traffic volume of voice data in the entire system. In addition, in VR / MR / AR image data, an object representing a virtual stage may be arranged in a predetermined zone in order to visually represent the virtual stage. Furthermore, an object representing a virtual audience seat may be placed at a place outside the predetermined zone. The predetermined zone does not have to coincide with the virtual stage, and may include, for example, the front row of virtual audience seats. Thereby, in addition to the virtual stage, the voice data of the reaction by a part of the avatar in the virtual spectator seat is also shared, so it is possible to produce a sense of reality.

  Alternatively, from a different point of view, the predetermined point can be regarded as a virtual microphone placement point. Since voice data of an avatar located far from the virtual microphone is not transmitted to the other terminal 100, the voice of the avatar located near the virtual microphone is less likely to be buried in the voice of the other avatar. That is, each user can concentrate on the target avatar's talk and singing voice, and at the same time, can suppress the traffic volume of voice data in the entire system. An object representing the virtual microphone may be arranged at a predetermined point which is a reference position of a predetermined zone in the VR / MR / AR image data in order to visually represent also the virtual microphone.

The default condition (2) will be described with reference to FIG. In FIG. 4, the crosses represent predetermined points, the distance between the point and avatar 1 is d ₁₀ , the distance between the point and avatar 2 is d ₂₀ , and the distance between the point and avatar 3 is d _30. Is represented by It is assumed that d ₂₀ <d ₁₀ <d ₃₀ and the default ranking is second. Under the predetermined conditions, in the terminal 100 of the user associated with avatar 1 and avatar 2, the condition determination unit 113 determines that the predetermined conditions are satisfied. On the other hand, in the terminal 100 of the user associated with the avatar 3, the condition determination unit 113 determines that the predetermined condition is not satisfied. Also in the predetermined condition (2), there is no distinction between the destination terminals, and the determination result is applied in relation to all the destination terminals. That is, the terminal 100 of the user associated with avatar 3 does not transmit voice data to any other terminal 100, and the terminal 100 of the user associated with avatar 1 and avatar 2 transmits voice data to all other terminals 100. Send.

  Also in the example of FIG. 4, as in the example of FIG. 3, the predetermined point can be regarded as a virtual microphone installation point. However, in the example of FIG. 4, the number of shared avatars of audio data is limited. Therefore, it is possible to prevent the situation that the number of avatars shared with audio data increases with the increase of the scale of the number of avatars existing in the virtual space, and thus the traffic volume of audio data increases. For example, whether the number of avatars present in the virtual space is five or 500, the number of avatars shared with audio data can be limited to two. Note that the predetermined condition (2) may be combined with the predetermined condition (1). For example, the plurality of avatars including the first avatar existing in the virtual space may be limited to those who are in the predetermined zone.

  In addition, it is necessary to acquire position data of a plurality of avatars in addition to the position data of the first avatar in order to determine the predetermined condition (2). Therefore, an increase in the amount of traffic due to mutual transmission of location data can not be avoided. However, the size of the position data is generally smaller than the voice data, and the position data need not always be transmitted (eg, it may be transmitted only when a change is detected). Therefore, the increment of the traffic amount due to the terminals 100 transmitting position data to each other is limited.

The default condition (3) will be described with reference to FIG. In FIG. 5, the distance between avatar 1 and avatar 2 is represented by d ₂₁ , and the distance between avatar 1 and avatar 3 is represented by d ₃₁ . It is assumed that d ₂₁ <threshold <d ₃₁ . Under the predetermined conditions, in the terminal 100 of the user associated with the avatar 1, the condition determination unit 113 determines that the predetermined condition is satisfied in the relationship with the terminal 100 of the user associated with the avatar 2, and It is determined that the predetermined condition is not satisfied in relation to the terminal 100 of the associated user. As described above, under the predetermined conditions (3) and (4), the condition determination unit 113 determines, for each destination terminal, whether the predetermined condition is satisfied. In other words, the terminal 100 may transmit voice data to one terminal 100 but may not transmit voice data to another terminal 100.

  The example of FIG. 5 is suitable for a scene where avatars have a conversation (voice chat) in a virtual space. That is, since the terminal 100 does not receive voice data of other avatars far from the first avatar, the voice of the avatar near the first avatar is less likely to be buried in the voice of the distant avatars. That is, each user can concentrate on the speech of the nearby avatar, and at the same time can suppress the traffic volume of voice data in the entire system. In order to visually represent avatars capable of sharing audio data, in VR / MR / AR image data, different visual representations of other avatars are determined depending on whether or not audio data can be shared. May be For example, avatars that are not in a position where voice data can be shared may be drawn darker than normal, and avatars that are in a position where voice data can be shared have an icon indicating that voice chat is possible. May be

The predetermined condition (4) will be described with reference to FIG. In FIG. 6, the distance between avatar 1 and avatar 2 is d ₂₁ , the distance between avatar 1 and avatar 3 is d ₃₁ , and the distance between avatar 1 and avatar 4 is d ₄₁ . It is assumed that d ₂₁ <d ₃₁ <d ₄₁ and the default ranking is 2nd. Under the predetermined conditions, in the terminal 100 of the user associated with the avatar 1, the condition determination unit 113 determines that the predetermined conditions are satisfied in relation to the terminal 100 of the user associated with the avatar 2 and the avatar 4. It is determined that the predetermined condition is not satisfied in relation to the terminal 100 of the user associated with the avatar 3.

  Similarly to the example of FIG. 5, the example of FIG. 6 is also suitable for a scene where avatars talk (voice chat) with each other in a virtual space. However, in the example of FIG. 6, the number of shared avatars of audio data is limited. Therefore, it is possible to prevent an increase in the number of avatars existing in the virtual space, an increase in the number of shared avatars of audio data, and thus an increase in traffic volume of audio data. For example, whether the number of avatars present in the virtual space is five or 500, the number of avatars shared with audio data can be limited to two. The predetermined condition (4) may be combined with the predetermined condition (3). For example, even if the distance between a certain avatar and the first avatar is equal to or greater than a predetermined rank, if the threshold is exceeded, the predetermined condition is not satisfied in relation to the terminal 100 of the user associated with the avatar. It may be determined.

  By the way, an object for avoiding the determination by the condition determination unit 113, for example, a virtual headset, a virtual pin microphone, or the like may be defined. These objects are visually represented on VR / MR / AR image data, and the user can operate, for example, the controller 20 to attach the objects to the avatar. Note that the visual representation of such an object may or may not resemble a headset or pin microphone. Also, such an object may be attachable to any avatar, or may be attached only to an avatar satisfying certain predetermined conditions such as responding to charging. In any case, while the avatar wears the object, the voice data of the avatar can be shared regardless of the position of the avatar.

  Specifically, the condition determination unit 113 first determines whether the first avatar is wearing a predetermined object in the virtual space (this corresponds to the virtual headset or the virtual pin microphone). Here, the attachment / removal event of the predetermined object by the first avatar may be detected based on user input data from the controller 20, for example. When the condition determination unit 113 determines that the first avatar is wearing the predetermined object, the condition determination unit 113 omits the determination of whether the predetermined condition is satisfied.

  Then, when it is determined by the condition determination unit 113 that the first avatar is wearing the predetermined object, the transmission control unit 114 described later satisfies the predetermined condition in relation to all the destination terminals. It operates in the same manner as in the case where it is determined. That is, the transmission control unit 114 determines to transmit the voice data of the first avatar toward all destination terminals.

  For example, if an avatar of a performer of an event held in a virtual space wears such an object, each user can keep the talk and singing of the avatar uninterrupted even if the avatar moves around the virtual space extensively. I can hear it. In addition, when combined with the above-mentioned predetermined condition (3) or (4), for example, a realistic virtual voice can be heard without interruption of the voice of the target avatar and voice can be exchanged with the user of the nearby avatar. User experience can be provided.

  Similarly, an avatar that can escape from the determination by the condition determination unit 113 may be defined. That is, the voice data of this avatar may be shared regardless of the position of the avatar. For example, voice data of the avatar of the user hosting the virtual space may be shared regardless of the position of the avatar.

  The transmission control unit 114 may be realized by the processor described above. The transmission control unit 114 receives the voice data of the first avatar and the control data and / or position data of the first avatar from the condition determination unit 113 together with data indicating the determination result. The transmission control unit 114 determines whether to transmit at least voice data of the first avatar toward the destination terminal according to the determination result. The transmission control unit 114 transmits, to the transmission unit 103, data (for example, address data) indicating the destination terminal and data (for example, voice data, position data and / or control data) to be sent to the destination terminal.

  Specifically, when it is determined that the predetermined condition is satisfied in relation to the destination terminal, the transmission control unit 114 determines to transmit the voice data of the first avatar to the destination terminal. . Here, the transmission control unit 114 may decide to transmit voice data to a destination terminal via a peer-to-peer type network, for the various merits described above. On the other hand, when it is determined that the predetermined condition is not satisfied in relation to the destination terminal, the transmission control unit 114 determines not to transmit the voice data of the first avatar to the destination terminal.

  The sizes of control data and position data are generally smaller than voice data. Further, assuming that neither control data nor position data is transmitted toward the destination terminal, the first avatar displayed on the HMD 10 connected to the destination terminal remains stationary, and the virtual of the user of the destination terminal It is possible to impair the personal experience. Thus, control data and / or position data may be transmitted to all destination terminals regardless of the determination result.

  That is, when it is determined that the predetermined condition is satisfied in relation to the destination terminal, the transmission control unit 114 determines the voice data of the first avatar and the control data and / or position data of the first avatar. It decides to transmit toward the destination terminal. On the other hand, when it is determined that the predetermined condition is not satisfied in relation to the destination terminal, the transmission control unit 114 does not transmit the voice data of the first avatar to the destination terminal and does not transmit the voice data of the first avatar. It decides to transmit control data and / or position data towards the destination terminal.

  The image generation unit 115 can be realized by the processor described above. The image generation unit 115 performs VR / AR based on, for example, image data of an object (including an avatar) stored in the memory of the terminal 100 or the auxiliary storage device, and control data of another avatar from the data acquisition unit 111. / Generate (update) MR image data. The generation of VR / AR / MR image data may include control of postures of other avatars corresponding to control data. The image generation unit 115 sends the generated image data to the HMD 10 via the output unit 104.

  As described above, the image generation unit 115 determines the positions of the heads / hands of other avatars according to the position of the HMD 10 / controller 20 indicated by the control data, and uses, for example, IK (Inverse Kinetics) technology. The attitude may be controlled.

  The voice generation unit 116 may be realized by the processor described above. The sound generation unit 116 generates VR / AR / MR sound data based on sound data of another avatar from the data acquisition unit 111, for example. For example, the voice generation unit 116 may generate voice data by synthesizing voice data of a plurality of other avatars. The sound generation unit 116 sends the generated sound data to the speaker via the output unit 104.

  The sound generation unit 116 may further acquire position data of another avatar from the data acquisition unit 111. Then, using the sound image localization technology, the sound generation unit 116 may process the sound data of the other avatar so that the user perceives that the sound is generated from the position of the other avatar. This makes it possible to match the position of the avatar perceived visually and the position of the avatar perceived auditorily to produce a sense of presence.

  Hereinafter, the operation of the terminal 100 will be described using FIG. 7. The operation of FIG. 7 may be performed periodically, or may be performed at the timing when it is detected that the position data of the first avatar or another avatar has changed.

  First, the data acquisition unit 111 acquires voice data and position data of the first avatar (step S201). Depending on the predetermined conditions determined in the next step S202, the data acquisition unit 111 may further acquire position data of other avatars. Furthermore, before proceeding to step S202, the distance calculation unit 112 may calculate the distance between two points in the virtual space based on the position data acquired in step S201.

  Next, the condition determination unit 113 determines whether a predetermined condition regarding the position of the first avatar is satisfied (step S202). If it is determined that the predetermined condition is satisfied in relation to at least one destination terminal, the process proceeds to step S203. On the other hand, if it is determined that the predetermined condition is not satisfied in relation to all the destination terminals, the operation of FIG. 7 ends. Note that the transmission control unit 114 may determine to transmit the control data and / or position data of the first avatar to each destination terminal regardless of the determination result of step S202.

  In step S203, the transmission control unit 114 determines to transmit the voice data acquired in step S201 toward the destination terminal determined to satisfy the predetermined condition, and the operation of FIG. 7 ends.

  As described above, the terminal according to the embodiment determines whether the position in the virtual space of the first avatar associated with the user of the terminal satisfies the predetermined condition, and the predetermined condition is not satisfied. If it is determined that the voice data of the first avatar is not sent to the destination terminal. Therefore, according to this terminal, when it is determined that the predetermined condition is not satisfied, no traffic of voice data from the terminal to the destination terminal occurs. On the other hand, since this terminal receives only limited voice data of the avatar, for example, voices of avatars near the virtual microphone or the first avatar are less likely to be buried in voices of other avatars. Therefore, according to this terminal, the user can concentrate on the target avatar's talk or voice, or the voice of the nearby avatar, and at the same time, it is possible to suppress the traffic volume of voice data in the entire system.

(Modification 1)
As mentioned above, although the P2P type network is adopted in the system of FIG. 2, it is possible to construct a virtual experience sharing system even if a C / S type network is adopted additionally to this. is there. In the first modification, each terminal 100 selects either the P2P type network or the C / S type network, and transmits data to the destination terminal.

  According to the first variation, as illustrated in FIG. 8, it is possible to add the server 300 to the virtual experience sharing system of FIG. 2. The server 300 receives data from the terminal 100 as a transmission source, and transmits this to the terminal 100 as a destination.

  In this modification 1, when it is determined that the predetermined condition is satisfied in relation to the destination terminal, the transmission control unit 114 in the terminal 100 determines the voice data of the first avatar via the P2P type network as the destination. Decide to send towards the terminal. On the other hand, when it is determined that the predetermined condition is not satisfied in relation to the destination terminal, the transmission control unit 114 directs the voice data of the first avatar to the destination terminal via the C / S type network. You may decide to send. The control data and / or position data of the first avatar may be transmitted via the same network as the voice data of the first avatar.

  As described above, if the transmission control unit 114 selects the P2P type network or the C / S type network and transmits the voice data of the first avatar, various merits of the P2P type network with respect to the voice data (for example, delay) Although it is not possible to enjoy the reduction), it becomes possible for all users to share audio data of all avatars. That is, since the user can hear more avatars simultaneously, it is easy to feel bustling, excitement, sense of unity, and the like.

  Hereinafter, the operation of the terminal 100 according to the first modification will be described with reference to FIG. The operation of FIG. 9 may be performed periodically, or may be performed at a timing when it is detected that the position data of the first avatar or another avatar has changed.

  First, the data acquisition unit 111 acquires voice data and position data of the first avatar (step S401). Depending on the predetermined conditions determined in the next step S402, the data acquisition unit 111 may further acquire position data of other avatars. Furthermore, before proceeding to step S402, the distance calculation unit 112 may calculate the distance between two points in the virtual space based on the position data acquired in step S401.

  Next, the condition determination unit 113 determines whether a predetermined condition regarding the position of the first avatar is satisfied (step S402). If it is determined that the predetermined condition is satisfied, the process proceeds to step S403. On the other hand, when it is determined that the predetermined condition is not satisfied, the process proceeds to step S404. If the determination result may be different between the destination terminals as in the above-described predetermined condition (3) or (4), step S402 and step S403 or step S404 may be repeated for each destination terminal.

  In step S403, the transmission control unit 114 determines to transmit the voice data acquired in step S401 via the P2P type network toward the destination terminal determined to satisfy the predetermined condition, as shown in FIG. The operation of is finished.

  In step S404, the transmission control unit 114 determines to transmit the voice data acquired in step S401 via the C / S network toward the destination terminal determined that the predetermined condition is not satisfied. The operation of FIG. 9 ends.

(Modification 2)
As mentioned above, although the P2P type network is adopted in the system of FIG. 2, it is possible to construct a virtual experience sharing system even if a C / S type network is adopted instead. . Furthermore, the server 500 according to the modification 2 is not limited to relaying data from the transmission source terminal to the destination terminal but also plays a part of the functions of the terminal 100 described in the first embodiment.

  Specifically, in the second modification, the terminal does not have to perform the above-described predetermined condition determination and transmission control. That is, the terminal transmits voice data, control data and position data of the avatar to the server 500. As described later, avatar position data / control data may be generated in server 500 based on data received from a terminal.

  The hardware configuration of the server 500 will be described below. The server 500 is a computer, and includes, for example, a processor that performs communication control (in particular, transmission control of audio data), predetermined condition determination, and the like.

  Also, the server 500 includes a program that is executed by a processor to implement such processing and a memory that temporarily stores data used by the processor.

  The server 500 may handle all the data in the on-memory state, or some data may be saved in the auxiliary storage device. The auxiliary storage device may be, for example, an HDD, an SSD, a flash memory or the like built in or externally attached to the server 500, or may be a database server accessible from the server 500.

  The server 500 can further use a communication I / F for connecting to the network. The communication I / F may be incorporated in the server 500 or may be externally attached to the server 500. The communication I / F is a module for communicating with a terminal or the like, and may include a signal processing circuit for transmission and reception, an antenna, a LAN terminal, and the like.

  Server 500 may further include a bus for transferring data between each element, such as a processor, memory, auxiliary storage, communication I / F, and the like.

  A functional configuration of the server 500 is illustrated in FIG. The server 500 includes a reception unit 501, a transmission unit 502, a data acquisition unit 511, a distance calculation unit 512, a condition determination unit 513, and a transmission control unit 514. However, the distance calculation unit 512, the condition determination unit 513, and the transmission control unit 514 may be the same as or similar to the elements of the same name in FIG. 1, and may be realized by the above-described processor.

  The receiving unit 501 can be realized by the above-described communication I / F. The reception unit 501 receives voice data and position data of an avatar (this corresponds to the first avatar described in the first embodiment) associated with the user of the transmission source terminal from the transmission source terminal via the network. And may receive control data. The receiving unit 501 sends the received data to the data acquisition unit 511.

  The transmission unit 502 can be realized by the aforementioned communication I / F. The transmission unit 502 is controlled by the transmission control unit 514, and transmits voice data, position data and / or control data of the avatar toward the destination terminal.

  The data acquisition unit 511 may be realized by the processor described above. The data acquisition unit 511 acquires various data from the reception unit 501. The data acquisition unit 511 sends the acquired data to the distance calculation unit 512 and the condition determination unit 513.

  Specifically, the data acquisition unit 511 acquires voice data, position data, and control data of the avatar, sends the position data to the distance calculation unit 512 and the condition determination unit 513, and the voice data and control data are determined by the condition determination unit 513. Can be sent to.

  When the terminal transmits position data / control data, the data acquisition unit 511 can directly acquire position data / control data from the reception unit 501. On the other hand, if it is necessary for the server 500 (of the position / posture estimation unit (not shown) to estimate the position of the HMD 10 / controller 20 based on the output data of the position sensor system to generate position data / control data, data The acquisition unit 511 may send the output data of the position sensor system to the position / posture estimation unit to request generation of position data / control data.

  As described above, in the server according to the second modification, as in the terminal according to the first embodiment, the condition in the virtual space of the first avatar associated with the user of the terminal transmitting the voice data is a predetermined condition. The voice data of the first avatar is not sent to the destination terminal when it is determined that the predetermined condition is not satisfied. Therefore, according to this server, as in the terminal according to the first embodiment, the user can concentrate on the talk and voice of the target avatar or the speaking voice of the nearby avatar, and at the same time, the voice of the entire system The amount of data traffic can be reduced. Note that the server according to the present modification and the terminals according to the first embodiment and the first modification can also be generically referred to as a “data transmission device”.

  The embodiments described above are merely illustrative examples to assist in understanding the inventive concept, and are not intended to limit the scope of the present invention. Embodiments can add, delete, or convert various components without departing from the scope of the present invention.

  Although the above embodiments have described several functional units, these are merely examples of implementation of each functional unit. For example, a plurality of functional units described to be implemented in one device may be implemented over a plurality of separate devices, and conversely, described as being implemented over a plurality of separate devices. It is also possible that the implemented function units are implemented in one device.

  The various functional units described in the above embodiments may be realized by using a circuit. The circuit may be a dedicated circuit that implements a specific function or may be a general-purpose circuit such as a processor.

  At least a part of the processing of each of the above-described embodiments can also be realized by using, for example, a processor mounted on a general-purpose computer as basic hardware. The program for realizing the above process may be provided by being stored in a computer readable recording medium. The program is stored in the recording medium as an installable file or an executable file. The recording medium may be a magnetic disk, an optical disk (CD-ROM, CD-R, DVD or the like), a magneto-optical disk (MO or the like), a semiconductor memory or the like. The recording medium may store the program and may be any computer readable one. Further, the program for realizing the above processing may be stored on a computer (server) connected to a network such as the Internet, and may be downloaded to the computer (client) via the network.

10 ... HMD
20: Controller 100: Terminal 101: Input unit 102, 501: Reception unit 103, 502: Transmission unit 104: Output unit 111, 511: Data acquisition unit 112, 512 ... Distance calculation unit 113, 513 ... Condition determination unit 114, 514 ... Transmission control unit 115 ... Image generation unit 116 ... Audio generation unit 300, 500 ... Server

Claims (10)

An acquisition unit for acquiring first position data indicating a first position of the first avatar in the virtual space, and audio data of the first avatar;
A determination unit that determines whether a predetermined condition regarding the first position is satisfied;
When it is determined that the predetermined condition is satisfied, it is determined to transmit the voice data toward the destination terminal, and when it is determined that the predetermined condition is not satisfied, the voice data is directed to the destination terminal And a transmission control unit that determines not to transmit
Wherein the predetermined condition state, and are said first position is within the predetermined zone in the virtual space,
The determination unit determines whether the first avatar wears a predetermined object in the virtual space, and determines that the first avatar wears the predetermined object. Omit determination of whether or not the predetermined condition is satisfied,
The transmission control unit determines to transmit the audio data to the destination terminal when it is determined that the first avatar is wearing the predetermined object.
Data transmission device.   The transmission control unit according to claim 1, wherein the transmission control unit determines to transmit the voice data to the destination terminal via a peer-to-peer type network when it is determined that the predetermined condition is satisfied. Data transmission device.   The data transmission device according to claim 1, wherein the predetermined zone is determined based on a distance from a predetermined point in the virtual space. The acquisition unit further acquires control data that determines the posture of the first avatar,
The transmission control unit determines to transmit the voice data and at least one of the control data and the first position data to a destination terminal when it is determined that the predetermined condition is satisfied, And transmitting at least one of the control data and the first position data to the destination terminal without transmitting the voice data to the destination terminal when it is determined that the predetermined condition is not satisfied. decide,
The data transmission apparatus according to any one of claims 1 to 3 . Computer,
A first position data indicating a first position of the first avatar in the virtual space, and means for acquiring voice data of the first avatar;
Means for determining whether predetermined conditions for the first position are satisfied;
When it is determined that the predetermined condition is satisfied, it is determined to transmit the voice data toward the destination terminal, and when it is determined that the predetermined condition is not satisfied, the voice data is directed to the destination terminal A program that functions as a means to decide not to send
Wherein the predetermined condition state, and are said first position is within the predetermined zone in the virtual space,
The determining unit determines whether the first avatar wears a predetermined object in the virtual space, and determines that the first avatar wears the predetermined object. Omits the determination of whether the predetermined condition is satisfied,
The determining means determines to transmit the audio data to the destination terminal when it is determined that the first avatar is wearing the predetermined object.
program. An acquisition unit for acquiring first position data indicating a first position of the first avatar in the virtual space, and audio data of the first avatar;
A determination unit that determines whether a predetermined condition regarding the first position is satisfied;
When it is determined that the predetermined condition is satisfied, it is determined to transmit the voice data toward the destination terminal, and when it is determined that the predetermined condition is not satisfied, the voice data is directed to the destination terminal And a transmission control unit that determines not to transmit
The acquisition unit acquires position data indicating a position in the virtual space of a plurality of avatars including the first avatar existing in the virtual space,
The predetermined condition is that, when the positions of the plurality of avatars in the virtual space are sorted in order from a predetermined point in the virtual space, the first position is equal to or higher than a predetermined rank.
Data transmission device. The transmission control unit determines to transmit toward the voice data over the peer-to-peer type network when said predetermined condition is determined to be satisfied to the destination terminal, according to claim 6 Data transmission device. The determination unit determines whether the first avatar wears a predetermined object in the virtual space, and determines that the first avatar wears the predetermined object. Omit determination of whether or not the predetermined condition is satisfied,
The transmission control unit determines to transmit the audio data to the destination terminal when it is determined that the first avatar is wearing the predetermined object.
The data transmission apparatus according to claim 6 or 7 . The acquisition unit further acquires control data that determines the posture of the first avatar,
The transmission control unit determines to transmit the voice data and at least one of the control data and the first position data to a destination terminal when it is determined that the predetermined condition is satisfied, And transmitting at least one of the control data and the first position data to the destination terminal without transmitting the voice data to the destination terminal when it is determined that the predetermined condition is not satisfied. decide,
The data transmission apparatus according to any one of claims 6 to 8 . Computer,
A first position data indicating a first position of the first avatar in the virtual space, and means for acquiring voice data of the first avatar;
Means for determining whether predetermined conditions for the first position are satisfied;
When it is determined that the predetermined condition is satisfied, it is determined to transmit the voice data toward the destination terminal, and when it is determined that the predetermined condition is not satisfied, the voice data is directed to the destination terminal A program that functions as a means to decide not to send
The acquiring means acquires position data indicating positions in the virtual space of a plurality of avatars including the first avatar existing in the virtual space,
The predetermined condition is that, when the positions of the plurality of avatars in the virtual space are sorted in order from a predetermined point in the virtual space, the first position is equal to or higher than a predetermined rank.
program.