JP6289703B1 - Information processing method, information processing program, information processing system, and information processing apparatus - Google Patents

Abstract

An object of the present invention is to improve entertainment in a virtual space. An information processing method includes: generating virtual space data; generating visual field image data indicating a visual field image displayed on the HMD based on the movement of the HMD of the user terminal and the virtual space data; The step of moving the hand of the avatar 4A according to the movement of the hand of the user wearing the HMD, the step of acquiring voice data based on the voice input to the microphone, and the first condition related to the hand state of the avatar 4A When the avatar 4B is located in the high transmission area in the virtual space, the step of setting the volume parameter L to the volume parameter L1 and the avatar 4B is located in the high transmission area in the virtual space. If not, the step of setting the volume parameter L to the volume parameter L2, and the audio data and volume parameter Based on, it includes the steps of outputting the voice of the user to the headphone of the user terminal. [Selection] Figure 12

Description

  The present disclosure relates to an information processing method, an information processing program, an information processing system, and an information processing apparatus.

  Patent Document 1 calculates a relative positional relationship between a moving body and a sound source when the moving body serving as a viewpoint in the game space moves or the sound source moves during execution of the game program. Disclosed is a process (sound image localization process) for processing a sound output from a sound source using a localization parameter based on the relative positional relationship.

JP 2007-050267 A

  However, in Patent Document 1, in order to further improve the entertainment property of the virtual space, different parameters (for example, attenuation coefficient) related to the sound output from the object (sound source object) defined as the sound source are set in the virtual space. There is no disclosure of a technique for setting different areas. In particular, there is no disclosure of a technique for imparting directivity to sound output from a sound source object in a virtual space. Here, the virtual space includes a VR (Virtual Reality) space, an AR (Augmented Reality) space, and an MR (Mixed Reality) space.

  The present disclosure aims to improve the entertainment property of a virtual space.

According to one aspect of the present disclosure, an information processing method includes:
a) generating virtual space data defining a virtual space including an operation object and a target object;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device associated with the first user terminal and the virtual space data;
c) moving the operation object in response to movement of a part of the body of the first user wearing the head mounted device;
d) obtaining voice data based on the voice input to the voice input unit;
e) When a first condition related to the state of the operation object is satisfied,
e1) when the target object is located in a first region in the virtual space, setting a parameter related to the audio data as a first parameter;
e2) When the target object is located in a second area different from the first area in the virtual space, a parameter related to the audio data is set to a second parameter different from the first parameter. When,
f) affecting the target object or a second user terminal associated with the target object based on the audio data and the parameter;
including.

  According to the present disclosure, the entertainment property of the virtual space can be improved.

It is the schematic which shows the virtual space delivery system which concerns on embodiment (henceforth this embodiment only) of this invention. It is the schematic which shows a user terminal. It is a figure which shows the head of the user with which HMD was mounted | worn. It is a figure which shows the hardware constitutions of a control apparatus. It is a flowchart which shows the process which displays a visual field image on HMD. It is xyz space figure which shows an example of virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows the hardware constitutions of the server shown in FIG. The state (a) is a diagram showing a virtual space provided to the user A. The state (b) is a diagram showing a virtual space provided to the user B. It is a sequence figure for explaining an example of processing which synchronizes movement of each avatar between user terminals. It is a flowchart for demonstrating an example of the volume parameter setting process which concerns on this embodiment. A state (a) is a top view which shows the virtual space provided to the user A in which the high transmission area | region was set. The state (b) is a front view of the avatar 4A. It is a top view which shows the virtual space provided to the user A to which the high transmission area | region was set. It is a top view which shows the virtual space provided to the user A to which the high transmission area | region was set. A state (a) is a top view which shows the virtual space provided to the user A in which the high transmission area | region was set. The state (b) is a front view of the avatar 4A. A state (a) is a top view which shows the virtual space provided to the user A in which the high transmission area | region was set. The state (b) is a front view of the avatar 4A. A state (a) is a top view which shows the virtual space provided to the user A in which the high transmission area | region was set. The state (b) is a front view of the avatar 4A.

[Description of Embodiments Presented by the Present Disclosure]
An overview of an embodiment indicated by the present disclosure will be described.
(1) a) generating virtual space data defining a virtual space including an operation object and a target object;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device associated with the first user terminal and the virtual space data;
c) moving the operation object in response to movement of a part of the body of the first user wearing the head mounted device;
d) obtaining voice data based on the voice input to the voice input unit;
e) When a first condition related to the state of the operation object is satisfied,
e1) when the target object is located in a first region in the virtual space, setting a parameter related to the audio data as a first parameter;
e2) When the target object is located in a second area different from the first area in the virtual space, a parameter related to the audio data is set to a second parameter different from the first parameter. When,
f) affecting the target object or a second user terminal associated with the target object based on the audio data and the parameter;
An information processing method including:

  According to the above method, when the first condition related to the state of the operation object is satisfied, the parameter related to the audio data is the first parameter depending on whether the target object is located in the first area or the second area. Set to parameter or second parameter. Thereafter, based on the audio data and the parameters (first parameter or second parameter), it is possible to influence the target object or the second user terminal associated with the target object. Therefore, the entertainment property of the virtual space can be improved.

(2) g) when the first condition is not satisfied, further includes a step of setting a parameter related to the audio data as a third parameter;
The information processing method according to item (1).

  According to the above method, when the second condition related to the state of the operation object is not satisfied, the parameter related to the audio data is set as the third parameter. Thereafter, based on the audio data and the third parameter, the target object or the second user terminal associated with the target object can be affected. Therefore, the entertainment property of the virtual space can be improved.

(3) The first to third parameters are set based on a relative positional relationship between the viewpoint position of the visual field image and the target object.
The information processing method according to item (2).

  According to the above method, since the first to third parameters are set based on the relative positional relationship between the viewpoint position of the visual field image and the target object, the entertainment property of the virtual space can be improved.

(4) The operation object includes a first operation object and a second operation object,
The first condition according to any one of items (1) to (3), including a second condition related to a relative positional relationship between the first operation object and the second operation object. Information processing method.

  According to the above method, when the first condition including the second condition related to the relative positional relationship between the first operation object and the second operation object is satisfied, the target object is placed in the first area or the second area. It is possible to set different parameters (first parameter and second parameter) depending on whether they are located. Therefore, the entertainment property of the virtual space can be improved.

  (5) The information processing method according to any one of items (1) to (4), wherein the first condition includes a third condition related to a shape of the operation object.

  According to the above method, when the first condition including the third condition related to the shape of the operation object is satisfied, different parameters (depending on whether the target object is located in the first area or the second area) ( It is possible to set the first parameter and the second parameter. Therefore, the entertainment property of the virtual space can be improved.

  (6) The first condition includes an item (1) related to a relative positional relationship between a virtual camera that defines the field-of-view image based on the movement of the head mounted device and the operation object. The information processing method according to any one of (5) to (5).

  According to the above method, whether the target object is located in the first region or the second region when the first condition including the fourth condition related to the relative positional relationship between the operation object and the virtual camera is satisfied. Accordingly, different parameters (first parameter and second parameter) can be set. Therefore, the entertainment property of the virtual space can be improved.

  (7) From the item (1), at least one of the first area and the second area is partitioned based on a visual field of a virtual camera that defines the visual field image based on a movement of the head mounted device. The information processing method according to any one of (6).

  According to the above method, at least one of the first region and the second region is partitioned based on the visual field of the virtual camera. In this way, different parameters (first parameter and second parameter) can be set in the virtual space based on the visual field of the virtual camera. Therefore, the entertainment property of the virtual space can be improved.

  (8) The information processing method according to any one of items (1) to (6), wherein the first area and the second area are partitioned based on a position of the operation object.

  According to the above method, the first area and the second area are partitioned based on the position of the operation object. In this way, different parameters (first parameter and second parameter) can be set in the virtual space based on the position of the operation object. Therefore, the entertainment property of the virtual space can be improved.

(9) The operation object includes a first operation object and a second operation object,
The first area and the second area are partitioned based on the position of the first operation object and the position of the second operation object.
The information processing method according to any one of items (1) to (6).

  According to the above method, the first area and the second area are partitioned based on the position of the first operation object and the position of the second operation object. In this way, different predetermined parameters (first parameter and second parameter) can be set in the virtual space based on the positions of the first operation object and the second operation object. Therefore, the entertainment property of the virtual space can be improved.

  (10) The first area and the second area are divided based on a position of a virtual camera that defines the visual field image based on a movement of the head mounted device and a position of the operation object. The information processing method according to any one of (1) to (6).

  According to the above method, the first area and the second area are partitioned based on the position of the virtual camera and the position of the operation object. As described above, it is possible to set different predetermined parameters (first parameter and second parameter) in the virtual space based on the position of the virtual camera and the position of the operation object. Therefore, the entertainment property of the virtual space can be improved.

(11) a) generating virtual space data defining a target object displayed superimposed on the real space;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device associated with the first user terminal and the virtual space data;
c) identifying the movement of the body part of the user wearing the head mounted device;
d) obtaining voice data based on the voice input to the voice input unit;
e) when a first condition relating to the state of the body part is satisfied,
e1) when the target object is located in a first area in the real space, setting a parameter related to the audio data as a first parameter;
e2) When the target object is located in a second area different from the first area in the real space, a step of setting a parameter related to the audio data to a second parameter different from the first parameter When,
f) affecting the target object or a second user terminal associated with the target object based on the audio data and the parameter;
Including an information processing method.

  According to the above method, the entertainment property of the virtual space can be improved.

  (12) An information processing program for causing a computer to execute the information processing method according to any one of items (1) to (11).

  According to the information processing program, the entertainment property of the virtual space can be improved.

  (13) A first user terminal including a voice input unit and a head-mounted device, and a server connected to be communicable with the first user terminal, and any one of items (1) to (11) An information processing system configured to execute the information processing method according to claim.

  According to the information processing system, the entertainment property of the virtual space can be improved.

(14) a processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
When the computer-readable instruction is executed by the processor, the information processing apparatus executes the information processing method according to any one of items (1) to (11).

  According to the information processing apparatus, the entertainment property of the virtual space can be improved. It should be noted that the information processing apparatus includes both a user terminal and a server.

[Details of Embodiments Presented by the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, about the member which has the same reference number as the member already demonstrated in description of this embodiment, the description is not repeated for convenience of explanation.

  First, an outline of the configuration of the virtual space distribution system 100 will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter simply referred to as distribution system 100). As shown in FIG. 1, the distribution system 100 includes a user terminal 1A (first user terminal) operated by a user A (first user) and a user terminal 1B (first user) operated by a user B (second user). 2 user terminals) and the server 2. User terminals 1A and 1B are communicably connected to a server 2 via a communication network 3 such as the Internet. In the present embodiment, the virtual space includes a VR (Virtual Reality) space, an AR (Arranged Reality) space, and an MR (Mixed Reality) space. Further, hereinafter, for convenience of explanation, the user terminals 1A and 1B may be simply referred to as the user terminal 1 in some cases. Further, the users A and B may be simply referred to as a user U. In the present embodiment, the user terminals 1A and 1B are assumed to have the same configuration.

  Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic diagram showing the user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mounted device (HMD) 110, a headphone 116, a microphone 118, a position sensor 130, an external controller 320, and a control device mounted on the user U's head. 120.

  The HMD 110 includes a display unit 112, an HMD sensor 114, and a gaze sensor 140. The display unit 112 includes a non-transmissive display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. Thereby, the user U can immerse in the virtual space by viewing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the user U's left eye and a right-eye display unit configured to provide an image to the user U's right eye. Also good. Further, the HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance.

  The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (such as an angular velocity sensor and a gyro sensor), and detects various movements (tilt and the like) of the HMD 110 mounted on the user U's head. be able to.

  The gaze sensor 140 has an eye tracking function that detects the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right eye gaze sensor irradiates, for example, infrared light to the right eye of the user U, and detects reflected light reflected from the right eye (particularly the cornea and iris), thereby acquiring information related to the rotation angle of the right eye's eyeball. May be. On the other hand, the left eye gaze sensor irradiates the left eye of the user U with, for example, infrared light, and detects reflected light reflected from the left eye (particularly the cornea and iris), thereby providing information on the rotation angle of the left eye's eyeball. May be obtained.

  Headphones 116 (audio output units) are attached to the left and right ears of the user U, respectively. The headphones 116 are configured to receive audio data (electrical signals) from the control device 120 and output audio based on the received audio data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Furthermore, the microphone 118 is configured to transmit audio data to the control device 120.

  The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 by wireless or wired communication, and is configured to detect information on the position, inclination, or light emission intensity of a plurality of detection points (not shown) provided in the HMD 110. . Further, the position sensor 130 is configured to detect information on the position, inclination and / or light emission intensity of a plurality of detection points (not shown) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared light or visible light. The position sensor 130 may include an infrared sensor and a plurality of optical cameras.

  The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, which is the user U's hand in the present embodiment), and thereby the avatar hand displayed in the virtual space. Used to control the operation. The external controller 320 is an external controller 320R for right hand operated by the right hand of the user U (hereinafter simply referred to as controller 320R) and an external controller 320L for left hand operated by the left hand of the user U (hereinafter simply referred to as controller 320L). And). The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the finger of the right hand. Further, the right hand of the avatar that exists in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the finger of the left hand. Further, the left hand of the avatar that exists in the virtual space moves according to the movement of the controller 320L.

  The control device 120 is a computer configured to control the HMD 110. The control device 120 identifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and installs the position of the virtual camera in the virtual space and the HMD 110 in the real space based on the identified position information. Thus, the position of the user U can be accurately associated. Further, the control device 120 identifies the operation of the external controller 320 based on information acquired from the position sensor 130 and / or a sensor built in the external controller 320, and based on the identified operation of the external controller 320. Thus, the motion of the avatar's hand displayed in the virtual space can be accurately associated with the motion of the external controller 320 in the real space. In particular, the control device 120 identifies the operation of the controller 320L based on the information acquired from the position sensor 130 and / or the sensor built in the controller 320L, and determines the virtual operation based on the identified operation of the controller 320L. The motion of the left hand of the avatar displayed in the space and the motion of the controller 320L in the real space (the motion of the left hand of the user U) can be accurately associated. Similarly, the control device 120 specifies the operation of the controller 320R based on the information acquired from the position sensor and / or the sensor built in the controller 320R, and based on the operation of the specified controller 320R, the virtual space The motion of the right hand of the avatar displayed inside the motion of the controller 320R in the real space (the motion of the right hand of the user U) can be accurately associated.

  Further, the control device 120 specifies the gaze of the right eye and the gaze of the left eye of the user U based on the information transmitted from the gaze sensor 140 (the left eye gaze sensor and the right eye gaze sensor), and the right eye gaze. It is possible to specify the point of gaze that is the intersection of the left eye gaze. Furthermore, the control device 120 can specify the line of sight of both eyes of the user U (the line of sight of the user U) based on the specified gazing point. Here, the line of sight of the user U is the line of sight of both eyes of the user U, and coincides with the direction of a straight line passing through the middle point of the line connecting the right eye and the left eye of the user U and the gazing point.

  Next, with reference to FIG. 3, a method for acquiring information related to the position and inclination of the HMD 110 will be described. FIG. 3 is a diagram illustrating the head of the user U wearing the HMD 110. Information on the position and inclination of the HMD 110 that is linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands up is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the direction orthogonal to the v-axis and the w-axis is defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 is an angle around each uvw axis (that is, a yaw angle indicating rotation around the v axis, a pitch angle indicating rotation around the u axis, and a center around the w axis). The inclination determined by the roll angle indicating rotation) is detected. The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

  Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram illustrating a hardware configuration of the control device 120. As illustrated in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are connected to each other via a bus 126 so as to communicate with each other.

  The control device 120 may be configured as a personal computer, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. In addition, some functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

  The control unit 121 includes a memory and a processor. The memory includes, for example, a ROM (Read Only Memory) in which various programs are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and / or a GPU (Graphics Processing Unit), and a program specified from various programs embedded in the ROM is expanded on the RAM. It is comprised so that various processes may be performed in cooperation with.

  In particular, the control unit 121 may control various operations of the control device 120 by a processor developing a control program on a RAM and executing the control program in cooperation with the RAM. The control unit 121 displays a field image on the display unit 112 of the HMD 110 based on the field image data. Thereby, the user U can be immersed in the virtual space.

  The storage unit (storage) 123 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. The storage unit 123 may store data related to the user U authentication program, various images, and objects (for example, an avatar). Furthermore, a database including tables for managing various data may be constructed in the storage unit 123.

  The I / O interface 124 is configured to connect the position sensor 130, the HMD 110, the external controller 320, the headphones 116, and the microphone 118 to the control device 120 so that they can communicate with each other, for example, a USB (Universal) The terminal includes a serial bus (D) terminal, a digital visual interface (DVI) terminal, a high-definition multimedia interface (HDMI) (registered trademark) terminal, and the like. The control device 120 may be wirelessly connected to each of the position sensor 130, the HMD 110, the external controller 320, the headphones 116, and the microphone 118.

  The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as the server 2 via the communication network 3 and various processing circuits for wireless connection, and communicates via the communication network 3. It is configured to conform to the communication standard.

  Next, processing for displaying a field-of-view image on the HMD 110 will be described with reference to FIGS. FIG. 5 is a flowchart showing a process for displaying the visual field image on the HMD 110. FIG. 6 is an xyz space diagram showing an example of the virtual space 200. The state (a) in FIG. 7 is a yx plan view of the virtual space 200 shown in FIG. The state (b) in FIG. 7 is a zx plan view of the virtual space 200 shown in FIG. FIG. 8 is a diagram illustrating an example of the visual field image V displayed on the HMD 110.

  As illustrated in FIG. 5, in step S <b> 1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 6, the virtual space 200 is defined as an omnidirectional sphere centered on the center position 210 (in FIG. 6, only the upper half celestial sphere is shown). In the virtual space 200, an xyz coordinate system with the center position 210 as the origin is set. The virtual camera 300 defines a visual axis L for specifying the visual field image V (see FIG. 8) displayed on the HMD 110. The uvw coordinate system that defines the visual field of the virtual camera 300 is determined so as to be linked to the uvw coordinate system that is defined around the head of the user U in the real space. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

  Next, in step S <b> 2, the control unit 121 specifies the field of view CV (see FIG. 7) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 identifies the position and orientation of the virtual camera 300 in the virtual space 200 based on information regarding the position and tilt of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300 and specifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a part of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, a part of the virtual space 200 displayed on the HMD 110). Equivalent to Further, the visual field CV includes a first region CVa set as an angular range of the polar angle α around the visual axis L in the xy plane shown in the state (a) of FIG. 7, and the state (b) of FIG. 7. The xz plane shown has a second region CVb set as an angle range of the azimuth angle β with the visual axis L as the center. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the information regarding the identified line of sight of the user U and the position and inclination of the HMD 110. The orientation of the virtual camera 300 (the visual axis L of the virtual camera 300) may be determined. As will be described later, the control unit 121 may determine the orientation of the face of the user U's avatar based on information on the position and inclination of the HMD 110.

  As described above, the control unit 121 can specify the visual field CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 updates the visual field CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. be able to. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the visual field CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change in the line of sight of the user U.

  Next, in step S <b> 3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

  Next, in step S4, the control unit 121 displays the field image V on the display unit 112 of the HMD 110 based on the field image data (see FIG. 7). As described above, the visual field CV of the virtual camera 300 is changed according to the movement of the user U wearing the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 is changed. I can immerse myself in 200.

  Note that the virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye view image data indicating the left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates right-eye view image data indicating a right-eye view image based on the virtual space data and the view of the right-eye virtual camera. Thereafter, the control unit 121 displays the left-eye field image on the left-eye display unit based on the left-eye field image data, and also displays the right-eye field image on the right-eye display unit based on the right-eye field image data. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally due to the parallax between the left-eye visual field image and the right-eye visual field image. As will be described later, the virtual camera may be arranged at the position of the avatar eye operated by the user. For example, the left-eye virtual camera may be placed in the left eye of the avatar, while the right-eye virtual camera may be placed in the right eye of the avatar.

  Further, the processes in steps S1 to S4 shown in FIG. 5 may be executed for each frame (a still image constituting a moving image). For example, when the frame rate of the moving image is 90 fps, the processes in steps S1 to S4 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). As described above, since the processes in steps S1 to S4 are repeatedly performed at predetermined intervals, the visual field of the virtual camera 300 is updated according to the operation of the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110. Is updated.

  Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 9 is a diagram illustrating a hardware configuration of the server 2. As shown in FIG. 9, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are communicably connected to each other via a bus 24. The control unit 23 includes a memory and a processor. The memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

  The storage unit (storage) 22 is, for example, a large-capacity HDD. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Moreover, the memory | storage part 22 may store the data regarding the user management information for managing each user, various images, and objects (for example, avatar etc.). The communication interface 21 is configured to connect the server 2 to the communication network 3.

  Next, an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to FIGS. The state (a) in FIG. 10 is a diagram showing the virtual space 200A provided to the user A. The state (b) in FIG. 10 is a diagram showing the virtual space 200B provided to the user B. FIG. 11 is a sequence diagram for explaining an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as shown in FIG. 10, as a precondition, an avatar 4A (first avatar) associated with the user terminal 1A (user A) and an avatar 4B associated with the user terminal 1B (user B) (first 2 avatars) share the same virtual space. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

  As shown in the state (a) of FIG. 10, the virtual space 200A of the user A includes an avatar 4A and an avatar 4B. The avatar 4A is operated by the user A and interlocked with the operation of the user A. The avatar 4A uses the left hand 43A (first operation object) linked to the operation of the controller 320L of the user terminal 1A (the operation of the left hand of the user A) and the operation of the controller 320R of the user terminal 1A (the operation of the right hand of the user A). It has a interlocking right hand 42A (second operation object).

  The avatar 4A is defined as a sound collector that collects a virtual camera 300A (see FIG. 14), a sound source object defined as the sound source of the user A's voice input to the microphone 118, and a sound propagating on the virtual space 200A. And a collected sound object. Note that the sound source object and the sound collection object may be separated from the avatar 4A. Virtual camera 300A may be placed in the eyes of avatar 4A. In particular, when the virtual camera 300A includes a left-eye virtual camera and a right-eye virtual camera, the left-eye virtual camera is disposed in the left eye of the avatar 4A, and the right-eye virtual camera is disposed in the right eye of the avatar 4A. May be. In the following description, it is assumed that the visual field CV of the avatar 4A matches the visual field CV (see FIG. 15) of the virtual camera 300A arranged in the eyes of the avatar 4A.

  In the virtual space 200A, a case where the body of the avatar 4A is not visualized is also assumed. Even in this case, the avatar 4A arranged in the virtual space 200A includes a virtual camera 300A, a sound source object, a sound collection object, a left hand 43A, and a right hand 42A.

  The avatar 4B is operated by the user B and interlocked with the operation of the user B. The avatar 4B uses the left hand 43B (first operation object) linked to the operation of the controller 320L of the user terminal 1B (the operation of the left hand of the user B) and the operation of the controller 320R of the user terminal 1B (the operation of the right hand of the user B). It has a linked right hand 42B (second operation object).

  The avatar 4B is defined as a sound collector that collects a virtual camera 300B (not shown), a sound source object defined as the sound source of the user B's voice input to the microphone 118, and a sound propagating on the virtual space 200B. And a collected sound object. Here, the avatar 4B or the sound collection object is defined as a target object. Note that the sound source object and the sound collection object may be separated from the avatar 4B. Virtual camera 300B may be placed in the eyes of avatar 4B. In particular, when the virtual camera 300B includes a left-eye virtual camera and a right-eye virtual camera, the left-eye virtual camera is disposed on the left eye of the avatar 4B, and the right-eye virtual camera is disposed on the right eye of the avatar 4B. May be. In the following description, it is assumed that the visual field CV of the avatar 4B matches the visual field CV of the virtual camera 300B arranged in the eyes of the avatar 4B.

  In the virtual space 200B, it is assumed that the body of the avatar 4B is not visualized. Even in this case, the avatar 4B arranged in the virtual space 200B includes the virtual camera 300B, the sound source object, the sound collection object, the left hand 43B, and the right hand 42B.

  Further, the positions of the avatars 4A and 4B may be specified according to the position of the HMD 110 of the user terminals 1A and 1B. Similarly, the face orientations of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Furthermore, the operation of the hand of the avatar 4A, 4B (an example of an operation object) may be specified according to the operation of the external controller of the user terminals 1A, 1B. In particular, the action of the left hand of the avatars 4A and 4B is specified according to the action of the controller 320L of the user terminals 1A and 1B, and the action of the right hand of the avatars 4A and 4B is determined according to the action of the controller 320R of the user terminals 1A and 1B. May be specified. Further, the relative positions of the black eyes with respect to the white eyes of the avatars 4A and 4B may be specified according to data indicating the lines of sight of the users A and B detected by the gaze sensor 140. In particular, the relative position of the left black eye with respect to the left white eye of the avatar may be specified according to data indicating the line of sight of the user's left eye detected by the left eye gaze sensor. Furthermore, the relative position of the right black eye with respect to the right white eye of the avatar may be specified in accordance with data indicating the line of sight of the user's right eye detected by the right eye gaze sensor.

  As shown in the state (b) of FIG. 10, the virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

  Next, referring to FIG. 11, in step S10, the user terminal 1A generates voice data of the user A. For example, when the user A inputs voice into the microphone 118 (voice input unit) of the user terminal 1A, the voice of the user A is converted into an electrical signal by the microphone 118. Thereafter, the electric signal is amplified by an amplifier circuit and then converted into audio data by A / D conversion. Thereafter, the audio data is transmitted to the control unit 121 via the I / O interface 124. In this way, the control unit 121 acquires audio data corresponding to the user A's audio.

  Next, in step S11, the control unit 121 of the user terminal 1A generates control information of the avatar 4A, and then generates audio data indicating the control information of the generated avatar 4A and the voice of the user A (voice data of the user A) ) To the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12). Here, the control information of the avatar 4A is information necessary for controlling the operation of the avatar 4A. For example, the control information of the avatar 4A includes information about the position of the avatar 4A (position information), information about the face direction of the avatar 4A (face direction information), information about the face state of the avatar 4A (face information), Information (hand information) about the hand state of the avatar 4A may be included. The hand information of the avatar 4A includes information regarding the state of the left hand 43A of the avatar 4A (left hand information) and information regarding the state of the right hand 42A of the avatar 4A (right hand information).

  Next, in step S <b> 13, the control unit 121 of the user terminal 1 </ b> B generates control information for the avatar 4 </ b> B and transmits the generated control information for the avatar 4 </ b> B to the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4B from the user terminal 1B (step S14). Here, the control information of the avatar 4B is information necessary for controlling the operation of the avatar 4B. For example, the control information of the avatar 4B includes information about the position of the avatar 4B (position information), information about the face direction of the avatar 4B (face orientation information), information about the face state of the avatar 4B (face information), Information (hand information) regarding the hand state of the avatar 4B may be included. The avatar 4B hand information includes information (left hand information) regarding the state of the left hand 43B of the avatar 4B and information (right hand information) regarding the state of the right hand 42B of the avatar 4B.

  Next, the control unit 23 of the server 2 sets a volume parameter L related to the voice data of the user A received from the user terminal 1A (Step S15). Here, the volume parameter L is a parameter that defines the volume (sound pressure level) of the voice of the user A output to the headphones 116 (audio output unit) of the user terminal 1B. A method for setting the volume parameter L will be described later.

  Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S16), while transmitting the control information of the avatar 4A, the voice data of the user A, and the volume parameter L to the user terminal 1B ( Step S20). Then, after receiving the control information of the avatar 4B in step S17, the control unit 121 of the user terminal 1A indicates the virtual space 200A (see state (a) in FIG. 10) based on the control information of the avatars 4A and 4B. The virtual space data is updated (step S18). For example, when the avatars 4A and 4B move, virtual space data indicating the virtual space 200A including the moved avatars 4A and 4B is generated. Thereafter, the control unit 121 of the user terminal 1A updates the visual field CV of the avatar 4A (virtual camera 300) according to the position and inclination of the HMD 110, and then updates the virtual space data and the updated visual field of the avatar 4A. Based on the CV, the visual field image displayed on the HMD 110 is updated (step S19).

  On the other hand, the control unit 121 of the user terminal 1B receives the control information of the avatar 4A, the voice data of the user A, and the volume parameter L in step S21, and then, based on the control information of the avatars 4A and 4B, the virtual space 200B. The virtual space data indicating (see state (b) in FIG. 10) is updated (step S22). Thereafter, the control unit 121 of the user terminal 1B updates the visual field CV of the avatar 4B (virtual camera 300) according to the position and inclination of the HMD 110, and then updates the virtual space data and the updated visual field of the avatar 4B. Based on the CV, the visual field image displayed on the HMD 110 is updated (step S23).

  Thereafter, the control unit 121 of the user terminal 1B outputs the voice of the user A to the headphones 116 based on the voice data of the user A and the volume parameter L (step S24). Here, the control unit 121 can determine the volume of the voice of the user A output to the headphones 116 based on the volume parameter L. Further, the control unit 121 processes the user A's voice data based on the user A's voice data and the volume parameter L, and then outputs the user A's voice to the headphones 116 based on the processed voice data. May be. Thus, voice chat (VR chat) between users (between avatars) can be realized in the virtual space.

  In the present embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, while the avatar 4B. Is transmitted to the user terminal 1A. In this way, it is possible to synchronize the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B.

  Next, an example of the information processing method according to the present embodiment (that is, an example of the volume parameter L setting process defined in step S15 in FIG. 11) will be described with reference to FIGS. FIG. 12 is a flowchart for explaining an example of the volume parameter L setting process (the process defined in step S15 in FIG. 11) according to the present embodiment. The state (a) in FIG. 13 is a plan view showing the virtual space 200A in which the high transmission area S1 is set. The state (b) of FIG. 13 is a front view of the avatar 4A.

  In the present embodiment, as a precondition, it is assumed that the avatar 4A, the avatar 4B, and the avatar 4C share the same virtual space as shown in the state (a) of FIG. The avatar 4C is an avatar operated by the user C and is associated with the user terminal 1C. That is, it is assumed that users A, B, and C share one virtual space via the communication network 3. The virtual space 200A provided to the user A includes an avatar 4A, an avatar 4B, and an avatar 4C. As illustrated in FIG. 11, the control unit 121 of the user terminal 1A generates virtual space data indicating the virtual space 200A. As a precondition, the sound source object and the sound collection object are included in the avatars 4A, 4B, and 4C.

  As shown in FIG. 12, in step S30, the control unit 23 of the server 2 (hereinafter simply referred to as the control unit 23) determines the distance D1 between the avatar 4A (sound source object) and the avatar 4B (sound collection object). Is identified. Specifically, the control unit 23 specifies the position of the avatar 4A based on the position information of the avatar 4A included in the control information of the avatar 4A, and uses the position information of the avatar 4B included in the control information of the avatar 4B. Based on this, the position of the avatar 4B is specified. Thereafter, the control unit 23 specifies the distance D1 between the avatar 4A and the avatar 4B based on the specified position of the avatar 4A and the specified position of the avatar 4B. If the sound source object and the sound collection object are separated from the avatar, the control unit 23 specifies the distance between the sound source object associated with the avatar 4A and the sound collection object associated with the avatar 4B. Also good. Further, since the virtual camera 300A is located in the eyes of the avatar 4A, the viewpoint position of the visual field image displayed on the HMD 110 is the position of the avatar 4A.

  Next, in step S31, the control unit 23 specifies a distance D2 between the left hand 43A and the right hand 42A based on the position of the left hand 43A of the avatar 4A and the position of the right hand 42A of the avatar 4A. In particular, the control unit 23 specifies the position of the left hand 43A of the avatar 4A based on the left hand information of the avatar 4A included in the control information of the avatar 4A, and positions of the right hand 42A of the avatar 4A based on the right hand information of the avatar 4A. Is identified.

  Next, the control unit 23 determines whether or not the distance D2 between the left hand 43A and the right hand 42A of the avatar 4A is equal to or less than a predetermined distance Dth (step S32). If the determination result of step S32 is NO, the process proceeds to step S40. On the other hand, when it is determined that the distance D2 is equal to or less than the predetermined distance Dth (YES in step S32), the control unit 23 determines whether the palm of the left hand 43A of the avatar 4A and the palm of the right hand 42A are facing each other. (Step S33). Specifically, the control unit 23 specifies the state of the left hand 43A based on the left hand information of the avatar 4A and specifies the state of the right hand 42A based on the right hand information of the avatar 4A. Thereafter, the control unit 23 determines whether the palm of the left hand 43A and the palm of the right hand 42A are opposed to each other based on the state of the left hand 43A and the state of the right hand 42A.

  If the determination result of step S33 is NO, the process proceeds to step S40. On the other hand, when it is determined that the palm of the left hand 43A and the palm of the right hand 42A are opposed to each other (YES in step S33), the control unit 23 sets the high transmission area S1 in the virtual space 200A (step S34). Here, in the high transmission area S1, the transmission degree of sound propagating in the virtual space is higher than the transmission degree of sound in the area other than the high transmission area S1. That is, in the high transmission region S1, the attenuation coefficient α1 (dB / distance) of the sound propagating in the virtual space is smaller than the attenuation coefficient α2 (dB / distance) of the region other than the high transmission region S1 (α1 <α2). Here, the attenuation coefficient α is a parameter that defines the amount of attenuation per unit distance of the sound propagating in the virtual space 200.

  The condition defined in step S32 and the condition defined in step S33 are the first conditions related to the state of the avatar 4A's hand (left hand 43A and right hand 42A). In step S34, the control unit 23 sets the high transmission area S1 in the virtual space 200A based on the position of the left hand 43A of the avatar 4A and the position of the right hand 42A of the avatar 4A. Specifically, the control unit 23 specifies the reference position P1 (state (a) in FIG. 13) based on the position of the left hand 43A and the position of the right hand 42A, and then performs high transmission starting from the reference position P1. Region S1 is set. Here, the reference position P1 may be determined as a midpoint of a line segment connecting the reference position of the left hand 43A and the reference position of the right hand 42A. The high transmission region S1 is defined as an angular region defined as a polar angle range starting from the reference position P1 on the xy plane and an azimuth angle range defined from the reference position P1 on the xz plane. And an angle region.

  Next, in step S35, the control unit 23 determines whether or not the avatar 4B is located in the high transmission area S1 (first area) based on the position of the avatar 4B and the high transmission area S1. When the control unit 23 determines that the avatar 4B is located in the high transmission area S1 (YES in step S35), the control unit 23 sets an attenuation coefficient α that defines the attenuation amount per unit distance of the sound propagating in the virtual space 200A. The attenuation coefficient α1 is set (step S36). Thereafter, the control unit 23 sets the volume parameter L (parameter related to audio data) to the volume parameter L1 (first parameter) based on the attenuation coefficient α1 and the distance D1 between the avatar 4A and the avatar 4B. (Step S37).

  On the other hand, the control unit 23 determines that the avatar 4B is not located in the high transmission area S1 (in other words, determines that the avatar 4B is located in the outer area (second area) of the high transmission area S1. (If NO in step S35), the attenuation coefficient α is set to the attenuation coefficient α2 (step S38). Here, the attenuation coefficient α2 is larger than the attenuation coefficient α1 (α1 <α2). Thereafter, the control unit 23 sets the volume parameter L to the volume parameter L2 (second parameter) based on the attenuation coefficient α2 and the distance D1 between the avatar 4A and the avatar 4B (step S39).

  In addition, when the first condition related to the hand state of the avatar 4A is not satisfied (that is, when the hand state of the avatar 4A does not satisfy at least one of the condition of step S32 and the condition of step S33), In step S40, the control unit 23 sets the attenuation coefficient α to the attenuation coefficient α0, and then sets the volume parameter L to the volume parameter L0 (third parameter) based on the attenuation coefficient α0 and the distance D1 (step S40). S41). When the first condition related to the hand state of the avatar 4A is not satisfied, the high transmission region S1 is not set in the virtual space 200A, and thus the attenuation coefficient α of the entire region of the virtual space 200A becomes the attenuation coefficient α0. . Here, the attenuation coefficient α0 = α1 <α2 may be satisfied, or α1 <α2 = α0 may be satisfied. Further, the attenuation coefficients α0, α1, and α2 may be different values (for example, α1 <α0 <α2, α0 <α1 <α2 or α1 <α2 <α0).

In the present embodiment, the value of the volume parameter L decreases as the value of the distance D1 between the avatar 4A (sound source object) and the avatar 4B (sound collection object) increases. On the contrary, the value of the volume parameter L increases as the distance D1 between the avatar 4A and the avatar 4B decreases. Furthermore, the value of the volume parameter L decreases as the value of the attenuation coefficient α (dB / distance) increases. On the contrary, the smaller the value of the attenuation coefficient α, the larger the value of the volume parameter L. In addition, as the volume parameter L increases, the volume (sound pressure level) of the voice of the user A output to the headphones 116 of the user terminal 1B increases. On the contrary, the smaller the volume parameter L, the smaller the volume of the voice of the user A output to the headphones 116 of the user terminal 1B. For example, the control unit 23 may set the volume parameter L based on the distance D1, the attenuation coefficient α, and the following relational expression (1). The relational expression (1) is merely an example, and a plurality of relational expressions among the distance D1, the attenuation coefficient α, and the volume parameter L are assumed.

L = L0−α × D1 (1)

L: Volume parameter (dB)
L0: Volume parameter (dB) at the reference distance D0
α: Damping coefficient (dB / distance)
D1: Distance between avatar 4A and avatar 4B

For example, when the attenuation coefficient α is α1, the volume parameter L1 can be expressed as follows from the relational expression (1).

L1 = L0−α1 × D1

  According to the present embodiment, when the control unit 23 determines that the hand state of the avatar 4A satisfies the conditions defined in steps S32 and S33, the control unit 23 determines whether the avatar 4B is located in the high transmission area S1. Thus, the attenuation coefficient α is set to the attenuation coefficient α1 or the attenuation coefficient α2 (α1 <α2). Next, the volume parameter L associated with the voice data of the user A is set to the volume parameter L1 or the volume parameter L2. Thereafter, based on the voice data of the user A and the volume parameter L (volume parameters L1, L2), the voice of the user A is output to the headphones 116 of the user terminal 1B associated with the avatar 4B. In particular, the volume of the voice of the user A output to the headphones 116 of the user terminal 1B is determined based on the transmitted volume parameter L.

  In this way, the high transmission area S1 is set in the virtual space 200A according to the hand state of the avatar 4A (user A). Furthermore, the volume of the voice of the user A output to the headphones 116 of the user terminal 1B changes depending on whether or not the avatar 4B is located in the high transmission area S1. For example, it is assumed that the distance between the avatar 4A and the avatar 4B is the same as the distance between the avatar 4A and the avatar 4C. As shown in the state (a) of FIG. 13, when the avatar 4B is located in the high transmission area S1 and the avatar 4C is located outside the high transmission area S1, the attenuation coefficient α1 is smaller than the attenuation coefficient α2. Therefore, the volume of the sound output from the headphones 116 of the user terminal 1B is larger than the volume of the sound output from the headphones 116 of the user terminal 1C. Thus, it becomes possible to give directivity to the voice propagating through the virtual space 200A according to the state of the hand of the avatar 4A (user A).

  For example, it is assumed that the user B who operates the avatar 4B can hear the voice of the user A, while the user C who operates the avatar 4C cannot hear the voice of the user A. In addition, it is assumed that the user B who operates the avatar 4B can hear the voice of the user A at a high volume, while the user C who operates the avatar 4C can hear the voice of the user A at a normal volume. For this reason, the user A can give a voice instruction to the user B who operates the avatar 4B without being noticed by the user C who operates the avatar 4C. Therefore, the entertainment property of the virtual space can be improved.

  On the other hand, according to the present embodiment, when it is determined that the hand state of the avatar 4A does not satisfy the conditions defined in steps S32 and S33, the attenuation coefficient α in the entire area of the virtual space 200A is changed to the attenuation coefficient α0. Is set. Thereafter, the volume parameter L is set to the volume parameter L0 based on the attenuation coefficient α0 and the distance D1. Then, based on the user A's voice data and the volume parameter L0, the user A's voice is output from the headphones 116 of the user terminal 1B. In particular, the volume of the voice of the user A output to the headphones 116 of the user terminal 1B is determined based on the transmitted volume parameter L0. As described above, when it is determined that the hand state of the avatar 4A (user A) does not satisfy the conditions defined in steps S32 and S33, the voice propagating through the virtual space 200A has no directivity.

  In this embodiment, it is assumed that the avatar 4B is operated by the user B. However, when the avatar 4B is controlled by the control unit 23 (for example, a game program), the control unit 23 sets the set volume. Based on the parameter L, a parameter associated with the avatar 4B may be determined. For example, the control unit 23 may determine the amount of damage given to the avatar 4B based on the volume parameter L. Specifically, the control unit 23 may determine the amount of damage given to the avatar 4B based on the volume parameter L and the LUT (lookup table) indicating the relationship between the volume parameter L and the amount of damage. Good. In this regard, the greater the volume parameter L, the greater the amount of damage given to the avatar 4B. As shown in the state (a) of FIG. 13, the avatar 4B controlled by the control unit 23 is located in the high transmission region S1, while the avatar 4C controlled by the control unit 23 is located in the outer region of the high transmission region S1. In this case, since the attenuation coefficient α1 is smaller than the attenuation coefficient α2, the amount of damage given to the avatar 4B may be larger than the amount of damage given to the avatar 4C. Thus, since the amount of damage given to an avatar can be changed according to the arrangement position of an avatar, the entertainment property of virtual space can be improved. Moreover, the control part 23 may update the table which shows the amount of damage of an avatar after determining the amount of damage given to an avatar.

  In the description of this embodiment, when the avatar 4B is located in the high transmission area S1, the attenuation coefficient α is set to α1, while when the avatar 4B is located outside the high transmission area, the attenuation coefficient α. Is set to α2 (α1 <α2). Further, the volume parameter L is set based on the attenuation coefficient α and the distance D1 between the avatar 4A and the avatar 4B. However, the setting process of the volume parameter L in the present embodiment is not limited to this. For example, the volume parameter L may be set directly without considering the attenuation coefficient α and the distance D1. Specifically, when the avatar 4B is located in the high transmission area S1, the volume parameter L is set to L1, while when the avatar 4B is located outside the high transmission area, the volume parameter L is set to L2 (L2 <L1) may be set. Similarly, the volume parameter L may be set to L0 when the hand state of the avatar 4A does not satisfy both the condition defined in step S32 and the condition defined in step S33.

  In the present embodiment, as shown in FIG. 11, the control unit 23 of the server 2 sets the volume parameter L, but the control unit 121 of the user terminal 1A may set the volume parameter L. The control unit 121 of the user terminal 1B may set the volume parameter L. When the control unit 121 of the user terminal 1A sets the volume parameter L, the volume parameter L setting process may be executed between the process of step S17 and the process of step S18. After setting the volume parameter L, the control unit 121 of the user terminal 1A transmits the volume parameter L to the user terminal 1B via the server 2. When the control unit 121 of the user terminal 1B sets the volume parameter L, the volume parameter L setting process may be executed after the process of step S22.

  In the present embodiment, the condition related to the hand state of the avatar 4A shown in FIG. 12 (hereinafter referred to as the avatar hand condition) includes two conditions, the condition of step S32 and the condition of step S33. The present embodiment is not limited to this. For example, the avatar hand condition may be configured by either one of the condition in step S32 and the condition in step S33.

  Next, another example of the avatar hand condition will be described. That is, the avatar hand condition is related to the relative positional relationship between the hand of the avatar 4A and the virtual camera 300A in addition to the condition defined in step S32 and the condition defined in step S33 shown in FIG. A condition (fourth condition) may be included. As an example of the fourth condition, as illustrated in FIG. 14, the control unit 23 of the server 2 determines whether the left hand 43A and the right hand 42A of the avatar 4A are included in a predetermined area based on the position of the virtual camera 300A. May be determined. In this case, when the left hand 43A and the right hand 42A are included in the predetermined area, the control unit 23 determines that the fourth condition is satisfied. As another example of the fourth condition, the control unit 23 determines that the distance between the left hand 43A and the virtual camera 300A is equal to or less than a predetermined distance Dth2, and the distance between the right hand 42A and the virtual camera 300A is a predetermined distance. It may be determined whether the distance is equal to or less than Dth2. In this case, when the distance between the left hand 43A and the virtual camera 300A is equal to or smaller than the predetermined distance Dth2, and the distance between the right hand 42A and the virtual camera 300A is equal to or smaller than the predetermined distance Dth2, the control unit 23 It is determined that the fourth condition is satisfied. The avatar hand condition may be configured only from the fourth condition.

  In addition, the avatar hand condition is related to the relative positional relationship between the hand of the avatar 4A and the mouth of the avatar 4A in addition to the condition defined in step S32 and the condition defined in step S33 shown in FIG. The condition (fifth condition) may be included. As an example of the fifth condition, the control unit 23 may determine whether the left hand 43A and the right hand 42A of the avatar 4A are included in a predetermined area based on the mouth position of the avatar 4A. In this case, when the left hand 43A and the right hand 42A are included in the predetermined area, the control unit 23 determines that the fifth condition is satisfied. As another example of the fifth condition, the control unit 23 determines that the distance between the left hand 43A and the mouth of the avatar 4A is not more than a predetermined distance Dth3 and the distance between the right hand 42A and the mouth of the avatar 4A is It may be determined whether the distance is equal to or less than a predetermined distance Dth3. In this case, when the distance between the left hand 43A and the mouth of the avatar 4A is a predetermined distance Dth3 or less and the distance between the right hand 42A and the mouth of the avatar 4A is a predetermined distance Dth3 or less, the control unit 23 determines that the fifth condition is satisfied. The avatar hand condition may be configured only from the fifth condition.

  In the present embodiment, the high transmission area S1 is set based on the position of the left hand 43A of the avatar 4A and the position of the right hand 42A of the avatar 4A, but the setting process of the high transmission area S1 is not limited to this. . For example, as shown in FIG. 15, the high transmission area S2 may be set based on the field of view CV of the virtual camera 300A. That is, the area within the visual field CV of the virtual camera 300A is set as the high transmission area S2.

  In addition, a predetermined angle region in the field of view CV of the virtual camera 300A may be set as the high transmission region S2. For example, the high transmission area S2 may be set based on the visual axis La of the virtual camera 300A. In this case, the high transmission region S2 is defined as an angular region defined as a polar angle range centered on the visual axis La on the xy plane, and an azimuth angle range centered on the visual axis La on the xz plane. Angle range.

(First modification)
Next, an information processing method according to the first modification of the present embodiment with reference to FIGS. 12 and 16 (that is, a first modification of the volume parameter L setting process defined in step S15 in FIG. 11). Will be described. The state (a) in FIG. 16 is a plan view showing the virtual space 200A in which the high transmission area S3 is set. The state (b) of FIG. 16 is a front view of the avatar 4A. Note that the information processing method according to the first modification differs from the information processing method according to the present embodiment described above in the processes defined in steps S31 to S34. Accordingly, only the difference between the information processing method according to the present embodiment and the information processing method according to the first modification will be described below.

  First, in step S31 shown in FIG. 12, the distance D3 between the right hand 42A of the avatar 4A (an example of one hand of the avatar 4A) and the virtual camera 300A is specified. Next, in step S32, the control unit 23 of the server 2 determines whether the distance D3 between the right hand 42A and the virtual camera 300A is equal to or less than a predetermined distance Dth3. When the control unit 23 determines that the distance D3 between the right hand 42A and the virtual camera 300A is equal to or less than the predetermined distance Dth3 (YES in step S32), the control unit 23 determines whether the right hand 42A is open (step S32). S33). When it is determined that the right hand 42A is opened (YES in step S33), the control unit 23 sets the high transmission area S3 in the virtual space 200A (step S34).

  For example, as shown in the state (a) of FIG. 16, the control unit 23 sets the high transmission region S3 based on the position of the virtual camera 300A, the position of the right hand 42A, and the field of view CV of the virtual camera 300A. Also good. Specifically, of the two boundaries of the visual field CV of the virtual camera 300A, the boundary on the left hand 43A side of the avatar 4A is defined as a boundary B1, while the boundary on the right hand 42A side of the avatar 4A is defined as a boundary B2. In this case, one boundary L1 of the high transmission region S3 coincides with the boundary B1 of the visual field CV of the virtual camera 300A. Further, the other boundary L2 of the high transmission area S3 is set so as to pass through predetermined positions of the virtual camera 300A and the right hand 42A. In this way, the high transmission area S3 defined by the boundaries L1 and L2 may be set.

  On the other hand, instead of the high transmission area S3, as shown in the state (a) of FIG. 17, the control unit 23 sets the high transmission area S4 based on the position of the virtual camera 300A and the position of the right hand 42A. May be. Specifically, one boundary L2 of the high transmission area S4 is set so as to pass through predetermined positions of the virtual camera 300A and the right hand 42A. In addition, the other boundary L3 of the high transmission region S4 is set to be parallel to the imaging surface of the virtual camera 300A. Thus, the high transmission area S4 defined by the boundaries L2 and L3 may be set.

  In this example, the right hand 42A is cited as an example of one hand of the avatar 4A, but the avatar hand conditions (conditions of steps S32 and S33) may be determined based on the position and state of the left hand 43A of the avatar 4A. . Further, the high transmission area may be set based on the position of the left hand 43A and the position of the virtual camera 300A.

  In step S31, the distance between the right hand 42A and the reference position of the mouth of the avatar 4A may be specified. In this case, in step S32, the control unit 23 may determine whether or not the distance between the right hand 42A and the reference position of the mouth of the avatar 4A is equal to or less than a predetermined distance.

(Second modification)
Next, an information processing method according to the second modification of the present embodiment with reference to FIGS. 12 and 18 (that is, a second modification of the volume parameter L setting process defined in step S15 in FIG. 11). Will be described. The state (a) in FIG. 18 is a plan view showing the virtual space 200A in which the high transmission area S5 is set. The state (b) in FIG. 18 is a front view of the avatar 4A. Note that the information processing method according to the second modification differs from the information processing method according to the present embodiment described above in the processing defined in steps S31 to S34. Accordingly, only the difference between the information processing method according to the present embodiment and the information processing method according to the second modification will be described below.

  In this example, since the process of steps S31 and S32 shown in FIG. 12 is not executed, after the process of step S30 is executed, the control unit 23 of the server 2 indicates that the shape of the right hand 42A of the avatar 4A is the finger in step S33. It is determined whether or not the shape is a feed. When it is determined that the shape of the right hand 42A is the pointing shape (YES in step S33), the control unit 23 sets the high transmission area S5 in the virtual space 200A (step S34).

  For example, as shown in the state (a) of FIG. 18, the control unit 23 may specify the position of the tip 420A of the index finger of the right hand 42A and set the high transmission region S5 starting from the tip 420A. . The high transmission region S5 includes an angle region defined as a polar angle range starting from the tip 420A on the xy plane, and an angle region defined as an azimuth angle range starting from the tip 420A on the xz plane. You may have. In this way, the high transmission area S5 may be set.

  In this example, the avatar hand condition includes only a condition related to the shape of the right hand 42A. Further, the high transmission area S5 is determined based only on the position of the right hand 42A (specifically, the position of the tip 420A). In this example, the right hand 42A is cited as an example of one hand of the avatar 4A, but the avatar hand condition may be determined based on the shape of the left hand 43A of the avatar 4A. Further, the high transmission area may be set based on the position of the left hand 43A.

  In the present embodiment, the movement of the avatar's hand is controlled according to the movement of the external controller 320 indicating the movement of the user U's hand, but in the virtual space according to the movement amount of the user U's hand itself. The movement of the avatar's hand may be controlled. For example, instead of using an external controller, by using a glove-type device or a ring-type device worn on the user's finger, the position sensor 130 can detect the position and movement amount of the user U's hand, The movement and state of the user's U finger can be detected. Further, the position sensor 130 may be a camera configured to image the user U's hand (including fingers). In this case, the position and amount of movement of the user U's hand are captured based on the image on which the user's hand is displayed without capturing any device directly on the user's finger by imaging the user's hand using the camera. Can be detected, and the movement and state of the finger of the user U can be detected.

  In the present embodiment, the hand of the avatar 4A (an example of an operation object) is moved according to the movement of the hand that is a part of the user A's body, and the condition related to the hand state of the avatar 4A (avatar hand condition) ) Is satisfied, a high transmission area is set in the virtual space 200A. However, the present embodiment is not limited to this. For example, when the foot of the avatar 4A is moved in accordance with the movement of the foot of the user A and a condition (avatar foot condition) related to the state of the foot of the avatar 4A is satisfied, the high transmission area in the virtual space 200A May be set. Thus, the operation object is not limited to the avatar's hands (left hand and right hand).

  Further, in order to implement various processes executed by the control unit 121 of the user terminal 1 by software, a control program for causing the computer (processor) to execute various processes may be incorporated in the storage unit 123 or the memory in advance. Good. Alternatively, the control program can be a magnetic disk (HDD, floppy disk), optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), magneto-optical disk (MO, etc.), flash memory (SD card, USB memory, SSD). Etc.) may be stored in a computer-readable storage medium. In this case, the storage medium is communicably connected to the control device 120, whereby the control program stored in the storage medium is incorporated into the storage unit 123. The control unit 121 executes various processes by loading the control program incorporated in the storage unit 123 onto the RAM and executing the loaded program.

  Furthermore, in order to implement various processes executed by the control unit 23 of the server 2 by software, a control program for causing a computer (processor) to execute various processes may be incorporated in the storage unit 22 or the memory in advance. . Alternatively, the control program can be a magnetic disk (HDD, floppy disk), optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), magneto-optical disk (MO, etc.), flash memory (SD card, USB memory, SSD). Etc.) may be stored in a computer-readable storage medium. In this case, the control program stored in the storage medium is incorporated into the storage unit 22 by connecting the storage medium to the server 2 so as to be communicable. The control unit 23 executes various processes by loading the control program incorporated in the storage unit 22 onto the RAM and executing the program loaded by the processor.

  The control program may be downloaded from a computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded control program is incorporated into the storage unit 123 of the control device 120 or the storage unit 22 of the server 2.

  As mentioned above, although embodiment of this indication was described, the technical scope of this invention should not be limitedly interpreted by description of this embodiment. This embodiment is an example, and it is understood by those skilled in the art that various modifications can be made within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalents thereof.

  In this embodiment, the virtual space is used to provide a virtual experience such as VR (Virtual Reality), AR (Arranged Reality), and MR (Mixed Reality) to the user. When the virtual space provides VR, background data stored in the memory is used as the background of the virtual space. When the virtual space provides AR or MR, the real space is used as the background. In this case, the HMD 110 includes a transmissive display device (optical see-through or video see-through display device), so that the real space can be used as a background. If virtual space is applied to MR, the object may be influenced by real space. As described above, when the virtual space includes at least a part of a virtual scene such as a background or a virtual object, the user can be provided with a virtual experience capable of interacting with the virtual scene.

When the virtual space is applied to AR or MR, the user's hand may be used instead of the avatar hand. In particular, a condition (user hand condition) related to the state of user A's hand (a part of the body of user A) may be applied instead of the condition related to the hand state of avatar 4A (avatar hand condition). . In this case, when a condition (user hand condition) similar to the above-described condition (avatar hand condition) in the virtual space is satisfied in the real space, a high transmission area may be set in the real space. Thereafter, the avatar 4B (target object) or the user terminal B (second user terminal) associated with the avatar 4B may be influenced based on the audio data and the volume parameter.

1, 1A, 1B, 1C: User terminal 2: Server 3: Communication network 4A, 4B, 4C: Avatar 21: Communication interface 22: Storage unit 23: Control unit 24: Bus 42A, 42B: Right hand 43A, 43B: Left hand 100 : Virtual space distribution system (distribution system)
110: HMD
112: Display unit 114: HMD sensor 116: Headphone 118: Microphone 120: Control device 121: Control unit 123: Storage unit 124: I / O interface 125: Communication interface 126: Bus 130: Position sensor 140: Gaze sensor 200, 200A , 200B: Virtual space 210: Center position 300, 300A: Virtual camera 320: External controller 320L: Left hand external controller (controller)
320R: External controller for right hand (controller)
420A: tip

Claims (8)

a) generating virtual space data defining a virtual space including a hand object and a target object;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device associated with the first user terminal and the virtual space data;
c) moving the hand object in response to the movement of the hand of the first user wearing the head mounted device;
d) obtaining voice data based on the voice input to the voice input unit;
e) In a case where the first condition related to the state of the hand object is satisfied,
when e1) the target object is located in the first region in the virtual space, setting a volume parameter associated with the voice data in the first volume parameter,
if e2) the target object is located in a second area different from the first region in the virtual space includes the steps of setting a different second volume parameter and the volume parameter of the first volume parameter,
f) based on the audio data and the volume parameter, the amount of damage applied to said target object, or viewing including the steps, the determining the volume of sound output from the second user terminal associated with the target object ,
The first condition includes at least one of the following conditions i) to iii):
i) a condition relating to a relative positional relationship between a left hand object and a right hand object among the hand objects;
ii) conditions relating to the shape of the hand object;
iii) a condition relating to a relative positional relationship between the virtual object defining the visual field image based on the movement of the head mounted device and the hand object;
The first area and the second area are partitioned based on one of the visual field of the virtual camera, the position of the hand object, the position of the virtual camera, and the position of the hand object.
Information processing method. g) when said first condition is not satisfied, further comprising information processing method according to claim 1 the step of setting the volume parameter in the third volume parameter. The information processing method according to claim 2, wherein the first to third volume parameters are set based on a relative positional relationship between a viewpoint position of the visual field image and the target object.   The information processing method according to any one of claims 1 to 3, wherein the first area and the second area are partitioned based on a position of the left hand object and a position of the right hand object. a) generating virtual space data defining a target object displayed superimposed on the real space;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device associated with the first user terminal and the virtual space data;
c) identifying the movement of the hand of the user wearing the head mounted device;
d) obtaining voice data based on the voice input to the voice input unit;
e) when a first condition related to the hand condition is satisfied,
when e1) the target object is located in the first area in the real space, setting a volume parameter associated with the voice data in the first volume parameter,
if e2) the target object is located in a second area different from the first region in the real space, and setting the different second volume parameter and the volume parameter of the first volume parameter,
f) based on the audio data and the volume parameter, the amount of damage applied to said target object, or viewing including the steps, the determining the volume of sound output from the second user terminal associated with the target object ,
The first condition includes at least one of the following conditions i) to iii):
i) a condition relating to a relative positional relationship between the left hand and the right hand of the hands;
ii) conditions related to the hand shape;
iii) conditions related to the relative positional relationship between the virtual camera defining the field-of-view image based on the movement of the head mounted device and the hand;
The first area and the second area are partitioned based on any one of the visual field of the virtual camera, the position of the hand, the position of the virtual camera, and the position of the hand.
Information processing method. An information processing program for causing a computer to execute the information processing method according to any one of claims 1 to 5 . A first user terminal having a voice input portion and the head-mounted device, the first user terminal and a communicatively connected server, information processing as claimed in any one of claims 1 6 An information processing system configured to perform the method. A processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
Wherein the computer readable instructions are executed by the processor, wherein the information processing apparatus to execute an information processing method as claimed in any one of claims 1 7, the information processing apparatus.

Images