JP2018163637A - Information processing method, information processing program, information processing system, and information processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a communication load in an information processing system.SOLUTION: An information processing method includes the steps of: generating virtual space data for defining a virtual space containing an avatar 4A associated with a user A and provided to a user B; updating a visual field image displayed in an HMD at a frame rate C3 on the basis of the movement of the HMD and the virtual space data; acquiring first location information of a black eye of the user A at an acquisition rate C1 on the basis of the detection data detected by a sensor; generating second location information of a black eye of the avatar 4A corresponding to the black eye of the user A on the basis of the first location information; transmitting the second location information toward a user terminal 1B at a transmission rate C2; and updating the location of the black eye of the avatar 4A in the virtual space at the frame rate C3 on the basis of the second location information.SELECTED DRAWING: Figure 12

Description

  The present disclosure relates to an information processing method, an information processing program, an information processing system, and an information processing apparatus.

  Patent Document 1 discloses a technique for controlling an avatar's operation (such as an avatar's posture) according to a user's operation (for example, the user's posture) after sharing a single VR space among a plurality of users. ing.

Japanese Patent No. 6017008

  By the way, when avatars enjoy communication in a virtual space, it is preferable to change the expression of the avatar associated with the user according to the expression of the user. In particular, it is possible to synchronize the facial expression of the user and the facial expression of the avatar by acquiring information indicating the facial expression of the user using a sensor such as a gaze sensor or a face camera. On the other hand, when the transmission rate for transmitting the position information or the like is lower than the acquisition rate (number of times per unit time) for acquiring the position information or the like of a part of the user's body (for example, black eyes), a lower transmission rate In consideration, it is necessary to synchronize and control the facial expression of the user (movement of a part of the body) and the facial expression of the avatar (movement of a part of the body of the avatar).

  An object of the present disclosure is to provide an information processing method and an information processing apparatus capable of reducing a communication load in an information processing system. Furthermore, it aims at providing the information processing program for making a computer perform the said information processing method. It is another object of the present invention to provide an information processing system with a reduced communication load.

According to one aspect of the present disclosure, a first user terminal associated with the first user having a sensor configured to detect movement of a part of the body of the first user, and a head mounted device There is provided an information processing method executed by a computer in an information processing system having a second user terminal associated with the second user.
The information processing method includes:
(A) generating virtual space data including a first avatar associated with the first user and defining a virtual space provided to the second user;
(B) updating a visual field image displayed on the head mounted device at a first rate based on the movement of the head mounted device and the virtual space data;
(C) obtaining first position information of a part of the body of the first user at a second rate based on detection data detected by the sensor;
(D) generating second position information of a part of the body of the first avatar corresponding to the part of the body of the first user based on the first position information;
(E) transmitting the first location information or the second location information to the second user terminal at a third rate lower than the first rate and the second rate;
(F) updating the position of the part of the body of the first avatar in the virtual space at the first rate based on the second position information.
In the first position information, the part of the body of the first user among the plurality of partial areas constituting the first body area including the part of the body of the first user is currently located. This is information indicating the first partial area.
In the second position information, the part of the body of the first avatar among the plurality of partial areas constituting the second body area including the part of the body of the first avatar is currently located. This is information indicating the second partial area.
In step (e),
The first position information or the second position information generated immediately before the time point at which the first position information or the second position information is transmitted is transmitted toward the second user terminal.

  According to the present disclosure, it is possible to reduce the communication load in the information processing system.

It is the schematic which shows the virtual space delivery system which concerns on embodiment (henceforth this embodiment only) of this invention. It is the schematic which shows a user terminal. It is a figure which shows the head of the user with which HMD was mounted | worn. It is a figure which shows the hardware constitutions of a control apparatus. It is a flowchart which shows the process which displays a visual field image on HMD. It is xyz space figure which shows an example of virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows the hardware constitutions of the server shown in FIG. The state (a) is a diagram showing a virtual space provided to the user A. The state (b) is a diagram showing a virtual space provided to the user B. It is a sequence figure for explaining an example of processing which synchronizes movement of each avatar between user terminals. It is a sequence diagram for demonstrating an example of the information processing method which concerns on 1st Embodiment. The state (a) is a diagram showing the black eyes and white eyes of the user A. The state (b) is a diagram illustrating a body region including the black eyes and white eyes of the user A. It is a figure which shows an example of the movement locus | trajectory of the center position of the black eye of the user A. State (a) is a diagram showing the black eyes and white eyes of the avatar 4A. The state (b) is a diagram illustrating a body region including the black eyes and white eyes of the avatar 4A. A state (a) is a figure for demonstrating the process which updates the position of the eye of the avatar 4A with a frame rate higher than a transmission rate based on the 2nd position information transmitted to the user terminal 1B. The state (b) is a diagram showing the second position information transmitted to the user terminal 1B at each time. It is a figure for demonstrating a mode that the position of the eye of the avatar 4A gradually moves toward the center position O2 of partial area | region R2 from center position O1 of partial area | region R1. The state (a) is a diagram for explaining processing for updating the position of the black eye of the avatar 4A at a frame rate higher than the transmission rate based on the second position information and the second direction information transmitted to the user terminal 1B. It is. The state (b) is a diagram showing the second position information and the second direction information transmitted to the user terminal 1B at each time. It is a figure for demonstrating a mode that the position of the black eye of Avatar 4A moves gradually toward the intersection X1 from center position O1 of partial area | region R1. It is a figure for demonstrating the process which acquires 2nd direction information based on the synthetic | combination vector of the movement vector which shows the movement of the user's A black eye.

[Description of Embodiments Presented by the Present Disclosure]
An overview of an embodiment indicated by the present disclosure will be described.
(1) a sensor configured to detect a movement of a part of the body of the first user, a first user terminal associated with the first user, a head mounted device, and a second user An information processing method executed by a computer in an information processing system comprising a second user terminal associated with
The information processing method includes:
(A) generating virtual space data including a first avatar associated with the first user and defining a virtual space provided to the second user;
(B) updating a visual field image displayed on the head mounted device at a first rate based on the movement of the head mounted device and the virtual space data;
(C) obtaining first position information of a part of the body of the first user at a second rate based on detection data detected by the sensor;
(D) generating second position information of a part of the body of the first avatar corresponding to the part of the body of the first user based on the first position information;
(E) transmitting the first location information or the second location information to the second user terminal at a third rate lower than the first rate and the second rate;
(F) updating the position of the part of the body of the first avatar in the virtual space at the first rate based on the second position information;
In the first position information, the part of the body of the first user among the plurality of partial areas constituting the first body area including the part of the body of the first user is currently located. Information indicating the first partial region,
In the second position information, the part of the body of the first avatar among the plurality of partial areas constituting the second body area including the part of the body of the first avatar is currently located. Information indicating the second partial region,
In step (e),
An information processing method for transmitting the first position information or the second position information generated immediately before transmitting the first position information or the second position information to the second user terminal.

  According to the above method, when the first location information or the second location information is transmitted toward the second user terminal at the third rate lower than the first rate and the second rate, the first location information or the second rate information is transmitted. The first position information or the second position information generated immediately before the time point at which the two position information is transmitted is transmitted toward the second user terminal. Thus, the generated first position information or second position information is transmitted to the second user terminal in a thinned state. Further, the first position information (or second position information) transmitted to the second user terminal is the first partial area (or the first position) where a part of the body of the first user (or the first avatar) is currently located. 2 part area | region), Comprising: It is not information which shows the positional coordinate of the one user's (or 1st avatar) body part. Therefore, since the amount of information to be transmitted to the second user terminal can be reduced, the communication load between the first user terminal and the second user terminal can be reduced. Thus, an information processing method capable of reducing the communication load in the information processing system can be provided.

(2) The part of the body of the first user is the black eyes of the first user;
The part of the body of the first avatar is the black eye of the first avatar;
The first body region includes a white eye of the first user;
The information processing method according to item (1), wherein the second body region includes a white eye of the first avatar.

  According to the above method, the first position information of the first user's black eye is acquired based on the detection data detected by the sensor. In addition, based on the first position information, the second position information of the black eye of the first avatar corresponding to the first user's black eye is generated. Furthermore, the position of the black eye of the first avatar in the virtual space is updated based on the second position information. Thus, since the position of the first user's black eye and the position of the first avatar's black eye can be synchronized, the user's facial expression can be reflected in the avatar's facial expression realistically.

(3) In step (f),
When the part of the body of the first user moves from the first partial area to a third partial area that is a partial area to which the part of the body of the first user is moved, The one of the body of the first avatar is gradually moved so that the part of the body gradually moves toward a center position of a fourth partial area that is a partial area to which the part of the body of the first avatar is moved. The information processing method according to item (1) or (2), wherein the position of the part is updated.

  According to the above method, when a part of the body of the first user moves from the first partial area to the third partial area, the part of the body of the first avatar gradually moves toward the center position of the fourth partial area. Move to. As described above, when the part of the body of the first user moves from the first partial area to the third partial area, the part of the body of the first avatar is smoothly directed from the second partial area to the fourth partial area. It is possible to move to. In other words, when the body part of the first user moves from the first partial area to the third partial area, the body part of the first avatar instantly changes from the second partial area to the fourth partial area. Do not move (warp). Therefore, it becomes possible to make the movement of the body part of the avatar smoothly follow the movement of the body part of the user.

(4) (g) obtaining the first direction information of the part of the body of the first user;
(H) acquiring the second direction information of the part of the body of the first avatar based on the one direction information; and
The first direction information is information indicating a third partial area that is a partial area of the part of the body of the first user.
The second direction information is information indicating a fourth partial area that is a partial area of the part of the body of the first avatar.
The step (e) includes transmitting the one-direction information or the second direction information at the third rate,
In the step (f), from the items (1) to (3), the position of the part of the body of the first avatar in the virtual space is updated based on the second position information and the second direction information. ). The information processing method according to any one of.

  According to the above method, the position of a part of the body of the first avatar in the virtual space is updated based on the second position information and the second direction information. Thus, the position of the part of the body of the first avatar can be updated in consideration of the destination of the part of the body of the first avatar.

(5) In step (f),
Updating the position of the part of the body of the first avatar so that the part of the body of the first avatar gradually moves in the second partial area toward the fourth partial area; The information processing method according to 4).

  According to the above method, a part of the body of the first avatar gradually moves in the second partial area (movement source partial area) toward the fourth partial area (movement destination partial area). Thus, when a part of the user's body moves from the first partial area to the third partial area, the part of the body of the first avatar moves smoothly from the second partial area to the fourth partial area. It becomes possible to make it. Therefore, it becomes possible to make the movement of the body part of the avatar smoothly follow the movement of the body part of the user.

  (6) The item (4) or (5), wherein the third partial area is specified based on at least one movement direction of the part of the body of the first user that moves within the first partial area. Information processing method described in 1.

  According to the above method, based on at least one movement direction of the body part of the first user that moves within the first partial area, the third partial area of the movement destination of the part of the first user body is determined. It becomes possible to specify.

  (7) An information processing program for causing a computer to execute the information processing method according to any one of items (1) to (6).

  Based on the above, it is possible to provide an information processing program capable of reducing the communication load in the information processing system.

(8) a first user terminal having a sensor configured to detect movement of a part of the body of the first user and associated with the first user;
A second user terminal associated with the second user, and configured to execute the information processing method according to any one of items (1) to (6) Information processing system.

  According to the above, an information processing system with a reduced communication load can be provided.

(9) a processor;
An information processing apparatus comprising: a memory that stores a computer readable instruction; and when the computer readable instruction is executed by the processor, the information processing apparatus is any one of items (1) to (6). An information processing apparatus that executes the information processing method according to item.

  According to the above, it is possible to provide an information processing apparatus capable of reducing the communication load in the information processing system. It should be noted that the information processing apparatus is either a user terminal or a server.

[Details of Embodiments Presented by the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, about the element which has the same reference number as the element already demonstrated in description of this embodiment, the description is not repeated for convenience of explanation.

  First, an outline of the configuration of the virtual space distribution system 100 (information processing system) will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter simply referred to as distribution system 100). As shown in FIG. 1, the distribution system 100 includes a user terminal 1A (first user terminal) operated by a user A (first user) wearing a head mounted device (HMD) 110 (first HMD), and an HMD 110 ( A user terminal 1B (second user terminal) operated by a user B (second user) wearing the second HMD) and a server 2; User terminals 1A and 1B are communicably connected to a server 2 via a communication network 3 such as the Internet. Further, hereinafter, for convenience of explanation, the user terminals 1A and 1B may be simply referred to as the user terminal 1 in some cases. Further, the users A and B may be simply referred to as a user U. In the present embodiment, the user terminals 1A and 1B are assumed to have the same configuration.

  Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic diagram showing the user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mounted device (HMD) 110, a headphone 116, a face camera 117, a microphone 118, a position sensor 130, and an external controller that are mounted on the user U's head. 320 and a control device 120.

  The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, and a face camera 113. The display unit 112 includes a non-transmissive display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. Thereby, the user U can immerse in the virtual space by viewing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the user U's left eye and a right-eye display unit configured to provide an image to the user U's right eye. Also good. Further, the HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance.

  The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (such as an angular velocity sensor and a gyro sensor), and detects various movements (tilt and the like) of the HMD 110 mounted on the user U's head. be able to.

  The gaze sensor 140 has an eye tracking function that detects the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right eye gaze sensor irradiates, for example, infrared light to the right eye of the user U, and detects reflected light reflected from the right eye (particularly the cornea and iris), thereby acquiring information related to the rotation angle of the right eye's eyeball. May be. On the other hand, the left eye gaze sensor irradiates the left eye of the user U with, for example, infrared light, and detects reflected light reflected from the left eye (particularly the cornea and iris), thereby providing information on the rotation angle of the left eye's eyeball. May be obtained. The acquisition rate of the detection data acquired by the gaze sensor 140 is larger than the transmission rate of avatar control information (described later) and the frame rate of the visual field image displayed on the HMD 110 of the user terminal 1B. For example, the acquisition rate of detection data is 200 Hz.

  The face camera 113 is configured to acquire an image (particularly, a moving image) in which the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user U are displayed with the HMD 110 mounted on the user U. Has been. The face camera 113 is arranged at a predetermined position inside the HMD 110 so that the user's U eyes and eyebrows can be imaged. The frame rate of the moving image acquired by the face camera 113 is larger than the transmission rate of the avatar control information and the frame rate of the visual field image displayed on the HMD 110 of the user terminal 1B. For example, the frame rate of the acquired moving image is 200 fps.

  Headphones 116 (audio output units) are attached to the left and right ears of the user U, respectively. The headphones 116 are configured to receive audio data (electrical signals) from the control device 120 and output audio based on the received audio data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Furthermore, the microphone 118 is configured to transmit audio data to the control device 120.

  The face camera 117 is configured to acquire an image (particularly, a moving image) in which the mouth of the user U and its periphery are displayed. The face camera 117 may be arranged at a position facing the user U's mouth so that the user U's mouth and its periphery can be imaged. Further, the face camera 117 may be connected to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be larger than the frame rate of the visual field image displayed on the HMD 110.

  The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 by wireless or wired communication, and is configured to detect information on the position, inclination, or light emission intensity of a plurality of detection points (not shown) provided in the HMD 110. . Further, the position sensor 130 is configured to detect information on the position, inclination and / or light emission intensity of a plurality of detection points (not shown) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared light or visible light. The position sensor 130 may include an infrared sensor and a plurality of optical cameras.

  The external controller 320 is used to control the movement of the avatar's hand displayed in the virtual space by detecting the movement of the user's U hand. The external controller 320 is an external controller 320R for right hand operated by the right hand of the user U (hereinafter simply referred to as controller 320R) and an external controller 320L for left hand operated by the left hand of the user U (hereinafter simply referred to as controller 320L). And). The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the finger of the right hand. Further, the right hand of the avatar that exists in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the finger of the left hand. Further, the left hand of the avatar that exists in the virtual space moves according to the movement of the controller 320L.

  The control device 120 is a computer configured to control the HMD 110. The control device 120 specifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and based on the specified position information, the position of the virtual camera (avatar position) in the virtual space, and the reality. The position of the user U wearing the HMD 110 in the space can be accurately associated. Further, the control device 120 identifies the operation of the external controller 320 based on information acquired from the position sensor 130 and / or a sensor built in the external controller 320, and based on the identified operation of the external controller 320. Thus, the motion of the avatar's hand displayed in the virtual space can be accurately associated with the motion of the external controller 320 in the real space. In particular, the control device 120 identifies the operation of the controller 320L based on the information acquired from the position sensor 130 and / or the sensor built in the controller 320L, and determines the virtual operation based on the identified operation of the controller 320L. The motion of the left hand of the avatar displayed in the space and the motion of the controller 320L in the real space (the motion of the left hand of the user U) can be accurately associated. Similarly, the control device 120 specifies the operation of the controller 320R based on the information acquired from the position sensor and / or the sensor built in the controller 320R, and based on the operation of the specified controller 320R, the virtual space The motion of the right hand of the avatar displayed inside the motion of the controller 320R in the real space (the motion of the right hand of the user U) can be accurately associated.

  Further, the control device 120 can associate the facial expression (state) of the user U with the facial expression (state) of the avatar of the user U based on the moving images acquired by the face cameras 113 and 117. Details of the process of associating the facial expression of the user U with the facial expression of the avatar will be described later.

  Further, the control device 120 can identify the right eye line of sight and the left eye line of sight of the user U, respectively, and can identify the gaze point that is the intersection of the right eye line of sight and the left eye line of sight. Furthermore, the control device 120 can specify the line of sight of both eyes of the user U (the line of sight of the user U) based on the specified gazing point. Here, the line of sight of the user U is the line of sight of both eyes of the user U, and coincides with the direction of a straight line passing through the middle point of the line connecting the right eye and the left eye of the user U and the gazing point. The control device 120 also determines the center position of the black eye of the user U and the left eye of the user U based on the detection data (eye tracking data) transmitted from the gaze sensor 140 (the left eye gaze sensor and the right eye gaze sensor). The center position of each black eye can be specified.

  Next, a method for acquiring information related to the position and inclination of the HMD 110 will be described below with reference to FIG. FIG. 3 is a diagram illustrating the head of the user U wearing the HMD 110. Information on the position and inclination of the HMD 110 that is linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands up is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the direction orthogonal to the v-axis and the w-axis is defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 is an angle around each uvw axis (that is, a yaw angle indicating rotation around the v axis, a pitch angle indicating rotation around the u axis, and a center around the w axis). The inclination determined by the roll angle indicating rotation) is detected. The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

  Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram illustrating a hardware configuration of the control device 120. As illustrated in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are connected to each other via a bus 126 so as to communicate with each other.

  The control device 120 may be configured as a personal computer, a smartphone, a fablet, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. In addition, some functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

  The control unit 121 includes a memory and a processor. The memory includes, for example, a ROM (Read Only Memory) in which various programs are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and / or a GPU (Graphics Processing Unit), and a program specified from various programs embedded in the ROM is expanded on the RAM. It is comprised so that various processes may be performed in cooperation with.

  In particular, the control unit 121 may control various operations of the control device 120 by a processor developing a control program on a RAM and executing the control program in cooperation with the RAM. The control unit 121 displays a field image on the display unit 112 of the HMD 110 based on the field image data. Thereby, the user U can be immersed in the virtual space.

  The storage unit (storage) 123 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. The storage unit 123 may store data related to the user U authentication program, various images, and objects (for example, an avatar). Furthermore, a database including tables for managing various data may be constructed in the storage unit 123.

  The I / O interface 124 is configured to communicably connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118 to the control device 120, respectively. For example, it is configured by a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control device 120 may be wirelessly connected to each of the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118.

  The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as the server 2 via the communication network 3 and various processing circuits for wireless connection, and communicates via the communication network 3. It is configured to conform to the communication standard.

  Next, processing for displaying a field-of-view image on the HMD 110 will be described with reference to FIGS. FIG. 5 is a flowchart showing a process for displaying the visual field image on the HMD 110. FIG. 6 is an xyz space diagram showing an example of the virtual space 200. The state (a) in FIG. 7 is a yx plan view of the virtual space 200 shown in FIG. The state (b) in FIG. 7 is a zx plan view of the virtual space 200 shown in FIG. FIG. 8 is a diagram illustrating an example of the visual field image V displayed on the HMD 110.

  As illustrated in FIG. 5, in step S <b> 1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 6, the virtual space 200 is defined as an omnidirectional sphere centered on the center position 210 (in FIG. 6, only the upper half celestial sphere is shown). In the virtual space 200, an xyz coordinate system with the center position 210 as the origin is set. The virtual camera 300 defines a visual axis L for specifying the visual field image V (see FIG. 8) displayed on the HMD 110. The uvw coordinate system that defines the visual field of the virtual camera 300 is determined so as to be linked to the uvw coordinate system that is defined around the head of the user U in the real space. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

  Next, in step S <b> 2, the control unit 121 specifies the field of view CV (see FIG. 7) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 identifies the position and orientation of the virtual camera 300 in the virtual space 200 based on information regarding the position and tilt of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300 and specifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a part of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, a part of the virtual space 200 displayed on the HMD 110). Equivalent to Further, the visual field CV includes a first region CVa set as an angular range of the polar angle α around the visual axis L in the xy plane shown in the state (a) of FIG. 7, and the state (b) of FIG. 7. The xz plane shown has a second region CVb set as an angle range of the azimuth angle β with the visual axis L as the center. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the information regarding the identified line of sight of the user U and the position and inclination of the HMD 110. The orientation of the virtual camera 300 (the visual axis L of the virtual camera) may be determined. As will be described later, the control unit 121 may determine the orientation of the face of the user U's avatar based on information on the position and inclination of the HMD 110.

  As described above, the control unit 121 can specify the visual field CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 updates the visual field CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. be able to. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the visual field CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change in the line of sight of the user U.

  Next, in step S <b> 3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

  Next, in step S4, the control unit 121 displays the field image V on the display unit 112 of the HMD 110 based on the field image data (see FIG. 7). As described above, the visual field CV of the virtual camera 300 is changed according to the movement of the user U wearing the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 is changed. I can immerse myself in 200.

  Note that the virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye view image data indicating the left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates right-eye view image data indicating a right-eye view image based on the virtual space data and the view of the right-eye virtual camera. Thereafter, the control unit 121 displays the left-eye field image on the left-eye display unit based on the left-eye field image data, and also displays the right-eye field image on the right-eye display unit based on the right-eye field image data. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally due to the parallax between the left-eye visual field image and the right-eye visual field image. As will be described later, the virtual camera may be arranged at the position of the avatar eye operated by the user. For example, the left-eye virtual camera may be placed in the left eye of the avatar, while the right-eye virtual camera may be placed in the right eye of the avatar.

  Moreover, the process of step S1-S4 shown in FIG. 5 may be performed for every frame. For example, when the frame rate of the moving image is 90 fps, the processes in steps S1 to S4 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). As described above, since the processes in steps S1 to S4 are repeatedly performed at predetermined intervals, the visual field of the virtual camera 300 is updated according to the operation of the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110. Is updated.

  Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 9 is a diagram illustrating a hardware configuration of the server 2. As shown in FIG. 9, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are communicably connected to each other via a bus 24. The control unit 23 includes a memory and a processor. The memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

  The storage unit (storage) 22 is, for example, a large-capacity HDD. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Moreover, the memory | storage part 22 may store the data regarding the user management information for managing each user, various images, and objects (for example, avatar etc.). The communication interface 21 is configured to connect the server 2 to the communication network 3.

[Explanation of motion control between avatars]
Next, an example of processing for synchronously controlling the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to FIGS. The state (a) in FIG. 10 is a diagram showing the virtual space 200A provided to the user A. The state (b) in FIG. 10 is a diagram showing the virtual space 200B provided to the user B. FIG. 11 is a sequence diagram for explaining an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as shown in FIG. 10, as a precondition, an avatar 4A (first avatar) associated with the user terminal 1A (user A) and an avatar 4B associated with the user terminal 1B (user B) (first 2 avatars) share the same virtual space. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

  As shown in the state (a) of FIG. 10, the virtual space 200A of the user A includes an avatar 4A and an avatar 4B. The avatar 4A is operated by the user A and interlocked with the operation of the user A. The avatar 4A has a left hand linked to the operation of the controller 320L of the user terminal 1A (the movement of the left hand of the user A), a right hand linked to the operation of the controller 320R of the user terminal 1A, and an expression linked to the facial expression of the user A. Have a face to do. The face of the avatar 4A has a plurality of face parts (for example, eyes, eyebrows, mouths, etc.). The avatar 4B is operated by the user B and interlocked with the operation of the user B. The avatar 4B has a left hand linked to the operation of the controller 320L of the user terminal 1B showing the operation of the user B's left hand, a right hand linked to the operation of the controller 320R of the user terminal 1B showing the operation of the user B's right hand, It has a face that is linked to the facial expression of user B. The face of the avatar 4B has a plurality of face parts (for example, eyes, eyebrows, mouths, etc.).

  Further, the positions of the avatars 4A and 4B may be specified according to the position of the HMD 110 of the user terminals 1A and 1B. Similarly, the face orientations of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Furthermore, the movement of the hands of the avatars 4A and 4B may be specified according to the movement of the external controller 320 of the user terminals 1A and 1B. The facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B. In particular, the position of the black eye for the white eyes of the avatars 4A and 4B may be specified according to the position of the black eye for the white eyes of the users A and B.

  As shown in the state (b) of FIG. 10, the virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

  Next, referring to FIG. 11, in step S10, the control unit 121 of the user terminal 1A generates voice data of the user A. For example, when the user A inputs voice to the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. Thereafter, the microphone 118 transmits the generated audio data to the control unit 121 via the I / O interface 124.

  Next, in step S11, the control unit 121 of the user terminal 1A generates control information of the avatar 4A, and then generates audio data indicating the control information of the generated avatar 4A and the voice of the user A (voice data of the user A) ) To the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12).

  Here, the control information of the avatar 4A is information necessary for controlling the operation of the avatar 4A. The control information of the avatar 4A includes information about the position of the avatar 4A (position information), information about the face orientation of the avatar 4A (face orientation information), and information about the state of the avatar 4A's hand (left hand and right hand) (hand information). ) And information (face information) related to the facial expression of the avatar 4A.

  The face information of the avatar 4A has information indicating the states of a plurality of face parts. The information indicating the state of the plurality of face parts includes information (eye information) indicating the eye state of the avatar 4A (the shape of the white eye and the position of the black eye with respect to the white eye), and information on the eyebrow of the avatar 4A (the position and shape of the eyebrows) Etc.) and information (mouth information) indicating the mouth state (mouth position and shape, etc.) of the avatar 4A.

  Specifically, the eye information of the avatar 4A includes information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyebrow information of the avatar 4A includes information indicating the state of the left eyebrow of the avatar 4A and information indicating the state of the right eyebrow of the avatar 4A.

  The control unit 121 of the user terminal 1A acquires an image showing the eyes (left eye and right eye) and eyebrow (left eyebrow and right eyebrow) of the user A from the face camera 113 mounted on the HMD 110, and then acquires the acquired image. Based on a predetermined image processing algorithm, the state of user A's eyes (left eye and right eye) and eyebrows (left eyebrows and right eyebrows) is specified. Next, based on the eye and eyebrow state of the specified user A, the control unit 121 includes information indicating the eye state of the avatar 4A (left eye and right eye) and the eyebrow of the avatar 4A (left eyebrow and right eyebrow). Information indicating the state is generated. Of the information indicating the eye state of the user A, information indicating the position (center position) of the black eye of the user A may be acquired by the gaze sensor 140 instead of the face camera 113.

  Similarly, the control unit 121 of the user terminal 1A acquires an image indicating the mouth of the user A and the periphery thereof from the face camera 117, and then based on the acquired image and a predetermined image processing algorithm, the control unit 121 of the user A The state of is identified. Next, the control unit 121 generates information indicating the mouth state of the avatar 4A based on the identified mouth state of the user A.

  As described above, the control unit 121 of the user terminal 1A includes the eye information of the avatar 4A corresponding to the eyes of the user A, the eyebrow information of the avatar 4A corresponding to the user A's eyebrows, and the avatar 4A corresponding to the mouth of the user A. It is possible to generate the mouth information.

  In addition, the control unit 121 uses a plurality of types of faces stored in the storage unit 123 or the memory based on an image captured by the face camera 113, an image captured by the face camera 117, and a predetermined image processing algorithm. The facial expression (for example, smile) of the user A may be specified from among the facial expressions (for example, smile, sad expression, no expression, angry expression, surprised expression, troubled expression, etc.). Thereafter, the control unit 121 can generate face information indicating a facial expression (for example, a smile) of the avatar 4A based on the identified facial expression of the user A. In this case, a storage unit 123 stores a facial expression database including a plurality of types of facial expressions of the avatar 4A and facial information of the plurality of avatars 4A each associated with one of the plurality of types of facial expressions. May be.

  For example, when the facial expression of the avatar 4A is specified as a smile, the control unit 121 acquires facial information indicating the smile of the avatar 4A based on the facial expression (smile) of the avatar 4A and the facial expression database. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information, and mouth information of the avatar 4A when the facial expression of the avatar 4A is smiling.

Next, in step S <b> 13, the control unit 121 of the user terminal 1 </ b> B generates control information for the avatar 4 </ b> B and transmits the generated control information for the avatar 4 </ b> B to the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4B from the user terminal 1B (step S14).

  Here, the control information of the avatar 4B is information necessary for controlling the operation of the avatar 4B. The control information of the avatar 4B includes information about the position of the avatar 4B (position information), information about the face direction of the avatar 4B (face orientation information), and information about the state of the avatar 4B's hands (left hand and right hand) (hand information). ) And information (face information) related to the facial expression of the avatar 4B.

  The face information of the avatar 4B has information indicating the state of a plurality of face parts. The information indicating the state of the plurality of face parts includes information (eye information) indicating the eye state of the avatar 4B (white eye shape and position of the black eye with respect to the white eye), and eyebrow information of the avatar 4B (eyebrows position and shape). Etc.) and information (mouth information) indicating the mouth state (mouth position and shape, etc.) of the avatar 4B. The method for acquiring the face information of the avatar 4B is the same as the method for acquiring the face information of the avatar 4A.

  Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S15), and transmits the control information of the avatar 4A and the voice data of the user A to the user terminal 1B (step S19). Then, after receiving the control information of the avatar 4B in step S16, the control unit 121 of the user terminal 1A updates the state of the avatars 4A and 4B based on the control information of the avatars 4A and 4B, and then the virtual space 200A. The virtual space data indicating (see the state (a) in FIG. 10) is updated (step S17).

  Specifically, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the face information of the avatars 4A and 4B. Thus, the virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

  Thereafter, the control unit 121 of the user terminal 1A specifies the visual field CV of the avatar 4A (virtual camera 300) according to the position and inclination of the HMD 110, and then updates the virtual space data and the visual field CV of the avatar 4A. Based on this, the visual field image displayed on the HMD 110 is updated (step S18).

  On the other hand, the control unit 121 of the user terminal 1B updates the state of the avatars 4A and 4B based on the control information of the avatars 4A and 4B after receiving the control information of the avatar 4A and the voice data of the user A in step S20. The virtual space data indicating the virtual space 200B (see the state (b) in FIG. 10) is updated (step S21).

  Specifically, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the face information of the avatars 4A and 4B. Thus, the virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

  Thereafter, the control unit 121 of the user terminal 1B specifies the visual field CV of the avatar 4B (virtual camera 300) according to the position and inclination of the HMD 110, and then updates the virtual space data and the visual field CV of the avatar 4B. Based on this, the visual field image displayed on the HMD 110 is updated (step S22).

  Next, the control unit 121 of the user terminal 1B includes the received voice data of the user A, information about the position of the avatar 4A included in the control information of the avatar 4A, information about the position of the avatar 4B, and a predetermined voice processing algorithm. Based on the above, the voice data of the user A is processed. Thereafter, the control unit 121 transmits the processed audio data to the headphones 116 (audio output unit), and then the headphones 116 outputs the user A's audio based on the processed audio data (step S23). Thus, voice chat (VR chat) between users (between avatars) can be realized in the virtual space.

  In the present embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, while the avatar 4B. Is transmitted to the user terminal 1A. In this way, it is possible to synchronize the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B.

[Description of Information Processing Method According to First Embodiment]
Next, the information processing method according to the first embodiment will be described with reference to FIGS. FIG. 12 is a sequence diagram for explaining an example of the information processing method according to the first embodiment. The state (a) in FIG. 13 is a diagram showing the black eye 53 and the white eye 52 of the user A. The state (b) of FIG. 13 is a diagram showing a body region F (first body region) composed of a plurality of partial regions S including the black eye 53 and the white eye 52 of the user A. FIG. 14 is a diagram illustrating an example of a movement locus of the center position Q of the black eye 53 of the user A. The state (a) of FIG. 15 is a diagram showing the black eye 63 and the white eye 62 of the avatar 4A. The state (b) of FIG. 15 is a diagram showing a body region E (second body region) including the black eyes 63 and the white eyes 62 of the avatar 4A.

In the present embodiment, the synchronous control of the position of the black eye 53 (part of the body) of the user A and the position of the black eye 63 (part of the body) of the avatar 4A associated with the user A will be described. That is, the process of updating the position of the black eye 63 of the avatar 4A based on the position information (second position information) of the black eye 63 of the avatar 4A among the control information of the avatar 4A already described will be described. More specifically, in the present embodiment, the user terminal 1B is in the virtual space 200B provided to the user B based on the position information of the black eye 63 of the avatar 4A transmitted from the user terminal 1A via the server 2. The position of the black eye 63 of the avatar 4A is updated. In this embodiment, the acquisition rate C1 (second rate) for acquiring the position information of the black eye 63 of the avatar 4A, the transmission rate C2 (third rate) of transmitting the position information of the black eye 63 of the avatar 4A, and the user The frame rate C3 (first rate) for updating the position of the black eye 63 of the avatar 4A on the terminal 1B side has the following relationship.

Acquisition rate C1> Frame rate C3> Transmission rate C2

For example, the acquisition rate C1 is 200 Hz, the transmission rate C2 is 60 Hz, and the frame rate C3 is 90 fps. Thus, in this embodiment, since the transmission rate C2 is lower than the acquisition rate C1, it is necessary to synchronize the facial expression of the user A and the facial expression of the avatar 4A in consideration of the low transmission rate C2. More specifically, it is necessary to simultaneously control the position of the black eye 53 of the user A and the position of the black eye 63 of the avatar 4A in consideration of the low transmission rate C2. An information processing method capable of reducing the communication load in the distribution system 100 shown in FIG. 1 when the transmission rate is low will be described below. In the present embodiment, it is assumed that the eyes of the user A include the left eye and the right eye of the user A. Further, the eyes of the avatar 4A include the left eye and the right eye of the avatar 4A.

  As shown in FIG. 12, in step S30, the control unit 121 of the user terminal 1A acquires detection data (eye tracking data) from the gaze sensor 140 at an acquisition rate C1 (for example, 200 Hz). Next, the control unit 121 specifies the center position Q of the black eye 53 of the user A based on the detection data acquired at the acquisition rate C1 (step S31). Thereafter, in step S32, the control unit 121 acquires first position information indicating the partial area S (first partial area) where the center position Q of the black eye 53 of the user A is currently located at the acquisition rate C1.

  As shown in the state (b) of FIG. 13, the body region F is set so as to include the black eyes 53 and the white eyes 52 of the user A, and then the body region F is divided into a plurality of partial regions S. . As shown in the figure, when the center position Q of the black eye 53 of the user A is currently located in the partial area S1, the control unit 121 determines the correspondence between the partial areas S and the position coordinates (for example, The partial area S where the center position Q of the black eye 53 of the user A is currently located is specified as the partial area S1 based on a relational expression or a table. Thereafter, the control unit 121 acquires first position information indicating the partial region S1. As illustrated in FIG. 14, when the center position Q is located in the partial region S1, the control unit 121 acquires first position information indicating the partial region S1. When the center position Q is located in the partial area S2, the control unit 121 acquires first position information indicating the partial area S2.

  Next, based on the first position information, the control unit 121 obtains second position information indicating the partial area R (second partial area) where the center position P of the black eye 63 of the avatar 4A is currently located at the acquisition rate C1. (Step S33). As shown in the state (a) of FIG. 14, the body region E is set so as to include the black eyes 63 and the white eyes 62 of the avatar 4 </ b> A, and then the body region E is divided into a plurality of partial regions R. . Each of the plurality of partial regions R may be associated with one of the plurality of partial regions S. For example, when the first position information indicates the partial region S1, the control unit 121 determines the partial region R1 based on the correspondence (relations, tables, etc.) between the plurality of partial regions R and the plurality of partial regions S. The second position information indicating is acquired.

  Next, in step S <b> 34, the control unit 121 transmits the second position information to the server 2 at the transmission rate C <b> 2 (for example, 60 Hz) via the communication network 3. In particular, the second position information is information for controlling the center position P of the black eye 63 of the avatar 4A, and is one of the control information of the avatar 4A. For this reason, the control part 121 transmits the control information (refer FIG. 11) of the avatar 4A containing 2nd positional information to the server 2 via the communication network 3. FIG.

  Further, since the transmission rate C2 (for example, 60 Hz) is smaller than the acquisition rate C1 (for example, 200 Hz), the control unit 121 transmits the second position information acquired at the acquisition rate C1 to the server 2 in a thinned state. . For example, when three pieces of second position information are acquired during a time interval ΔT12 between a time Ta1 at which one second position information is transmitted and a time Ta2 at which the next second position information is transmitted. The control unit 121 transmits the second position information generated immediately before the time (time T2) to transmit the second position information to the server 2.

  Next, after receiving the control information of the avatar 4A including the second position information, the server 2 transmits the control information of the avatar 4A to the user terminal 1B via the communication network 3 (step S35). Thereafter, the control unit 121 of the user terminal 1B receives the control information of the avatar 4A including the second position information via the communication network 3 and the communication interface 125 (step S36). Next, the control unit 121 of the user terminal 1B, based on the received second position information, the avatar 4A in the virtual space 200B (see FIG. 10) provided to the user B at the frame rate C3 (for example, 90 fps). The center position P of the black eye 63 (see FIG. 15) is updated (step S37). For example, when the received second position information indicates the partial area R1, the control unit 121 determines that the central position P is such that the central position P of the black eye 63 of the avatar 4A is located in the partial area R1 based on the second position information. Update. Here, it is assumed that the body region E including the black eyes 63 and the white eyes 62 of the avatar 4A arranged in the virtual space 200B is set, and the body region E is divided into a plurality of partial regions R. . For this reason, the control unit 121 can update the center position P of the black eye 63 of the avatar 4A based on the transmitted second position information.

  Further, when updating the central position P of the black eye 63 of the avatar 4A, the control unit 121 also updates the operations of the avatars 4A and 4B in the virtual space 200B (for example, the motions of hands and faces) at the frame rate C3. . In this manner, after updating the virtual space data indicating the virtual space 200B, the control unit 121 updates the head of the user B at the frame rate C3 based on the movement of the HMD 110 (the visual field of the virtual camera) and the virtual space data. The visual field image displayed on the mounted HMD 110 is updated.

  According to the present embodiment, when the second position information is transmitted toward the user terminal 1B via the server 2 at the transmission rate C2 lower than the acquisition rate C1 and the frame rate C3, the second position information is transmitted. The second position information generated immediately before the point of time is transmitted to the user terminal 1B via the server 2. Thus, the generated second position information is transmitted toward the user terminal 1B in a thinned state. Further, the second position information transmitted to the user terminal 1B is information indicating the partial region R where the black eye 63 of the avatar 4A is currently located, and is not information indicating the position coordinates of the black eye 63 of the avatar 4A. Therefore, since the amount of information to be transmitted to the user terminal 1B can be reduced, the communication load between the user terminal 1A and the user terminal 1B can be reduced. As described above, an information processing method capable of reducing the communication load in the distribution system 100 can be provided.

  Further, based on the second position information of the black eye 63 of the avatar 4A corresponding to the black eye 53 of the user A, the center position P of the black eye 63 of the avatar 4A in the virtual space 200B is updated on the user terminal 1B side. As described above, since the center position Q of the black eye 53 of the user A and the center position P of the black eye 63 of the avatar 4A arranged in the virtual space 200B can be controlled synchronously, the facial expression of the user A can be changed to the avatar 4A. It is possible to reflect it realistically in the facial expression.

  In the present embodiment, the control unit 121 of the user terminal 1A transmits the second position information to the user terminal 1B via the server 2 at the transmission rate C2, but instead of the second position information, the first position information is transmitted to the user terminal 1B. The position information may be transmitted to the user terminal 1B via the server 2 at the transmission rate C1. In this case, since the transmission rate C2 is smaller than the acquisition rate C1, the control unit 121 transmits the first position information generated immediately before the first position information is transmitted to the server 2. That is, the first position information acquired by the control unit 121 is transmitted to the server 2 in a thinned state. Thereafter, the control unit 121 of the user terminal 1B generates the second position information based on the first position information after receiving the first position information from the server 2. At this time, the second position information may be generated at the same rate as the transmission rate C2. Thereafter, the control unit 121 of the user terminal 1B updates the position of the black eye 63 of the avatar 4A arranged in the virtual space 200B at the frame rate C3 based on the generated second position information.

  In the present embodiment, the process defined in steps S32 to S34 may not be executed on the user terminal 1A side. In this regard, the processing defined in steps S32 to S34 may be executed on the server 2 side.

[Avatar 4A black eye update process]
Next, the process defined in step S37 will be described in detail with reference to FIGS. More specifically, based on the second position information transmitted at the transmission rate C2 (for example, 60 Hz), the center position P of the black eye 63 of the avatar 4A is determined at a frame rate C3 (for example, 90 fps) higher than the transmission rate C2. The update process will be described. State (a) in FIG. 16 illustrates processing for updating the center position P of the black eye 63 of the avatar 4A at a frame rate C3 higher than the transmission rate C2 based on the second position information transmitted to the user terminal 1B. FIG. The state (b) in FIG. 16 is a diagram illustrating the second position information transmitted to the user terminal 1B at each time T (T1, T2, T3...). FIG. 17 is a diagram for explaining how the center position P of the black eye 63 of the avatar 4A gradually moves from the center position O1 of the partial area R1 toward the center position O2 of the partial area R2.

  As shown in the state (a) of FIG. 16, since the frame rate C3 is higher than the transmission rate C2, the number of updates of the black eye position of the avatar 4A per unit time (for example, 1 second) is the unit time (for example, More than the number of second position information transmitted per second). For example, as shown in FIGS. 16 and 17, when the second position information indicating the partial region R1 is transmitted to the user terminal 1B at time T3, the control unit 121 of the user terminal 1B causes the times t5 and t6 (> T3 ), The center position P of the black eye 63 of the avatar 4A is arranged at the center position O1 of the partial region R1. Here, as shown in FIG. 17, the center position P at time t5 is shown as center position P5, and the center position P at time t6 is shown as center position P6.

  Next, when the center position Q of the black eye 53 of the user A moves from the partial area S1 (first partial area) to the partial area S2 (third partial area) (see FIG. 13), the control unit 121 of the user terminal 1B. Is such that the central position P of the black eye 63 of the avatar 4A gradually moves from the partial area R1 (second partial area) toward the central position O2 of the partial area R2 (fourth partial area) that is the destination partial area. The center position P of the black eye 63 of the avatar 4A is updated. Specifically, first, the control unit 121 specifies a line segment L1 in which the center position P connects the center position O1 of the partial region R1 and the center position O2 of the partial region R2. Next, after specifying the length D1 of the line segment L1, the control unit 121 determines the moving speed (the moving amount for each frame) of the center position P. For example, the control unit 121 may determine the moving speed of the center position P as (1/4) D1. Thereafter, the control unit 121 updates the center position P so that the center position P gradually moves on the line segment L1 at the determined moving speed.

  Here, the center position P at times t7, t8, and t9 is defined as center positions P7, P8, and P9, respectively, and the moving speed of the center position P is set to (1/4) D1. Note that time T4 <t7 <t8 <t9. In this case, as shown in FIG. 17, the control unit 121 sets the center position P7 to the line segment L1 so that the distance between the center position P7 and the center position O1 becomes (1/4) D1 at time t7. Place on top. Next, the control unit 121 arranges the center position P8 on the line segment L1 so that the distance between the center position P8 and the center position O1 is (1/2) D1 at time t8. Further, at time t9, control unit 121 arranges center position P9 on line segment L1 so that the distance between center position P9 and center position O1 is (3/4) D1.

  Thus, when the center position Q moves from the partial area S1 to the partial area S2, the center position P gradually moves toward the central position O2 of the partial area R2. As described above, when the black eye 53 of the user A moves from the partial area S1 to the partial area S2, the center position P of the black eye 63 of the avatar 4A is instantaneously moved (warped) from the central position O1 to the central position O2. In comparison, the black eye 63 of the avatar 4A can be smoothly moved from the partial region R1 toward the partial region R2. Accordingly, the movement of the black eye 63 of the avatar 4A can smoothly follow the movement of the black eye 53 of the user A, and the facial expression of the user A can be realistically reflected on the facial expression of the avatar 4A.

  In the description of the present embodiment, (1/4) D1 is given as an example of the moving speed of the center position P, but the moving speed of the center position P is not limited to this. Further, when receiving the second position information indicating a predetermined number (for example, three) of partial regions R2, the control unit 121 gradually moves the center position P toward the center position O2 at a predetermined moving speed. You may let them.

[Description of Information Processing Method According to Second Embodiment]
Next, an information processing method according to the second embodiment will be described with reference to FIGS. 12 and 18-20. The information processing method according to the present embodiment is different from the information processing method according to the first embodiment in that the center position P of the black eye 63 of the avatar 4A is updated based on the second position information and the second direction information. Here, the second direction information is information indicating a partial area to which the center position P of the black eye 63 of the avatar 4A is moved. In the state (a) of FIG. 18, the center position P of the black eye 63 of the avatar 4A is updated at the frame rate C3 higher than the transmission rate C2 based on the second position information and the second direction information transmitted to the user terminal 1B. It is a figure for demonstrating the process to perform. The state (b) in FIG. 18 is a diagram illustrating the second position information and the second direction information transmitted to the user terminal 1B at each time T (T1, T2, T3...). FIG. 19 is a diagram for explaining how the center position P of the black eye 63 of the avatar 4A gradually moves from the center position O1 of the partial region R1 toward the intersection X1. FIG. 20 is a diagram for describing a process of acquiring the second direction information based on the combined vector Vc of the movement vectors V1, V2, and V3 indicating the movement of the black eye 53 of the user A.

  In the present embodiment, as illustrated in FIG. 12, after the process of step S <b> 33, the control unit 121 of the user terminal 1 </ b> A displays first direction information indicating the partial area S to which the center position Q of the user A ’s black eye 53 To get. Below, an example of the acquisition method of 1st direction information is demonstrated with reference to FIG.

  As shown in FIG. 20, for example, when the acquisition rate C1 is 200 Hz and the transmission rate C2 is 60 Hz, the time Ta1 when the first second position information is transmitted and the next second position information are transmitted. It is assumed that the control unit 121 of the user terminal 1 </ b> A acquires the three center positions Q of the user's black eye 53 during the time interval ΔT <b> 12 with the time Ta <b> 2 (> Ta <b> 1). Here, of the three central positions Q, the central position Q at time T12 is defined as the central position Q12, the central position Q at time T13 is defined as the central position Q13, and the central position Q at time T14 is the center. The position is defined as Q14. Further, the center position Q at time T11 (<time Ta1) is defined as the center position Q11. Here, it is assumed that time T11 <Ta1 <T12 <T13 <T14. The second position information transmitted at time Ta1 indicates the partial area S1 to which the center position Q11 belongs. On the other hand, the second position information transmitted at time Ta2 indicates the partial region S1 to which the center position Q14 belongs.

  Next, the control unit 121 specifies the movement vectors V1 to V3 based on each center position Q. Specifically, the control unit 121 specifies the movement vector V1 based on the center positions Q11 and Q12, specifies the movement vector V2 based on the center position Q12 and the center position Q13, and determines the center position Q13 and the center position. The movement vector V3 is specified based on Q14. Thereafter, the control unit 121 specifies a combined vector Vc of the three movement vectors V1 to V3. Note that the control unit 121 may specify the combined vector Vc based only on the center positions Q11 and Q14.

  Next, the control unit 121 specifies the partial area S to which the center position Q is moved based on the direction of the combined vector Vc. For example, as illustrated in FIG. 20, when the combined vector Vc faces the partial area S2, the control unit 121 determines the partial area S that is the movement destination as the partial area S2. In this way, the control unit 121 can acquire the first direction information indicating the partial region S2. Further, the control unit 121 may determine that the destination partial area S cannot be specified (NULL) based on the scalar quantity of the combined vector Vc. For example, the control unit 121 may determine that the destination partial region S cannot be specified when it is determined that the scalar quantity of the combined vector Vc is equal to or less than a predetermined threshold. As described above, the control unit 121 may identify the destination partial area S based on both the direction of the combined vector Vc and the scalar quantity.

Next, based on the correspondence between the first direction information, the partial area S, and the partial area R, the control unit 121 changes the partial area R to which the center position P of the black eye 63 of the avatar 4A is moved to the partial area R2. Then, the second direction information indicating the partial region R2 is acquired. Thereafter, in step S34, the control unit 121 transmits the second position information and the second direction information at the transmission rate C2.

  Next, after receiving the second position information and the second direction information, the server 2 transmits the second position information and the second direction information to the user terminal 1B via the communication network 3 (step S35). Thereafter, the control unit 121 of the user terminal 1B receives the second position information and the second direction information via the communication network 3 and the communication interface 125 (step S36). Next, the control unit 121 of the user terminal 1B updates the center position P of the black eye 63 of the avatar 4A in the virtual space 200B at the frame rate C3 based on the received second position information and second direction information ( Step S37).

  Next, the process defined in step S37 will be described in detail with reference to FIGS. More specifically, the center position P is updated at a frame rate C3 (for example, 90 fps) higher than the transmission rate C2, based on the second position information and the second direction information transmitted at the transmission rate C2 (for example, 60 Hz). Processing to be performed will be described.

  In this respect, when the second position information indicates the partial area R1 and the second direction information indicates the partial area R2, the control unit 121 determines that the center position P of the black eye 63 of the avatar 4A is partially directed toward the partial area R2. The center position P is updated so as to move gradually in the region R1. Specifically, first, the control unit 121 specifies a line segment L1 that connects the center position O1 of the partial region R1 and the center position O2 of the partial region R2. Next, the control unit 121 specifies the intersection X1 between the boundary line B1 and the line segment L1 between the partial region R1 and the partial region R2. Thereafter, the control unit 121 specifies a distance D2 between the intersection X1 and the center position O1.

  Here, center positions P at times t1, t2, t3, and t4 are defined as center positions P1, P2, P3, and P4, respectively (note that times t1 <T1 <t2 <t3 <t4). Further, it is assumed that the center position P1 is located at the center position O1. In this case, as shown in FIG. 19, the control unit 121 sets the center position P2 to the line segment L1 so that the distance between the center position P2 and the intersection X1 is (1/2) × D2 at time t2. Place on top. Next, at time t3, the control unit 121 arranges the center position P3 on the line segment L1 so that the distance between the center position P3 and the intersection X1 becomes (1/2) 2 × D2. Furthermore, the control unit 121 arranges the center position P4 on the line segment L1 so that the distance between the center position P4 and the intersection X1 is (1/2) 3 × D2 at time t4. Thus, at time tn, the center position Pn may be arranged on the line segment L1 such that the distance between the center position Pn and the intersection X1 is (1/2) n-1 × D2. Further, at time tn, the center position Pn may be arranged on the line segment L1 so that the distance between the center position Pn and the intersection X1 is an-1 × D2 (where a <1 is there). Further, the control unit 121 determines the moving speed of the center position P as, for example, (1/4) D2, and then the center position P gradually moves on the line segment L1 at the moving speed (1/4) D2. Thus, the center position P may be updated. Note that the value of the moving speed is merely an example.

  Further, when the control unit 121 receives the second position information indicating the partial region R2 at time T4, the center position P gradually moves on the line segment L1 toward the central position O2 of the partial region R2 after time t7. The center position P may be updated so as to move to. In this case, the center position P may be located on the intersection X1 at time t7.

  In this way, by using the second direction information, the center position P of the black eye 63 of the avatar 4A can be gradually moved in the partial region R1 toward the partial region R2. For this reason, when the center position Q of the black eye 53 of the user A moves from the partial region S1 to the partial region S2, the black eye 63 of the avatar 4A can be smoothly moved from the partial region R1 toward the partial region R2. Become. Accordingly, the movement of the black eye 63 of the avatar 4A can smoothly follow the movement of the black eye 53 of the user A, and the facial expression of the user A can be realistically reflected on the facial expression of the avatar 4A.

  In the description of the present embodiment, it is assumed that the virtual space data indicating the virtual space 200B is updated on the user terminal 1B side, but the virtual space data may be updated on the server 2 side. Furthermore, although it is assumed that the visual field image data corresponding to the visual field image is updated on the user terminal 1B side, the visual field image data may be updated on the server 2 side. In this case, the user terminal 1 </ b> B displays the field image on the HMD 110 based on the field image data transmitted from the server 2.

  Further, in order to implement various processes executed by the control unit 121 of the user terminal 1 by software, a control program for causing the computer (processor) to execute various processes may be incorporated in the storage unit 123 or the memory in advance. Good. Alternatively, the control program includes a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), a magneto-optical disk (MO, etc.), flash memory (SD card, USB memory, SSD). Etc.) may be stored in a computer-readable storage medium. In this case, when the storage medium is connected to the control device 120, the control program stored in the storage medium is incorporated into the storage unit 123. The control unit 121 executes various processes by loading the control program incorporated in the storage unit 123 onto the RAM and executing the loaded program.

  The control program may be downloaded from a computer on the communication network 3 via the communication interface 125. Similarly in this case, the downloaded control program is incorporated into the storage unit 123.

  As mentioned above, although embodiment of this indication was described, the technical scope of this invention should not be limitedly interpreted by description of this embodiment. This embodiment is an example, and it is understood by those skilled in the art that various modifications can be made within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalents thereof.

  In the present embodiment, an example in which the user's (avatar) black eye is picked up as an example of a part of the user's (avatar) body and the center position of the user's black eye and the center position of the avatar's black eye are controlled synchronously has been described. However, the body part of the user (avatar) is not limited to black eyes. For example, as an example of a part of the user's (avatar) body, the position of the user's tongue and the position of the avatar's tongue may be synchronously controlled. Thus, a part of the user's (avatar) body is not limited to the user's (avatar) black eyes, and the information processing method according to the present embodiment can be applied to various parts of the user's (avatar) body. Applicable.

  In this embodiment, the virtual space is used to provide a virtual experience such as VR (Virtual Reality), AR (Arranged Reality), and MR (Mixed Reality) to the user. When the virtual space provides VR, background data stored in the memory is used as the background of the virtual space. When the virtual space provides AR or MR, the real space is used as the background. In this case, the HMD 110 includes a transmissive display device (optical see-through or video see-through display device), so that the real space can be used as a background. If virtual space is applied to MR, the object may be influenced by real space. As described above, when the virtual space includes at least a part of a virtual scene such as a background or a virtual object, the user can be provided with a virtual experience capable of interacting with the virtual scene.

1, 1A, 1B: User terminal 2: Server 3: Communication network 4A, 4B: Avatar 21: Communication interface 22: Storage unit 23: Control unit 24: Bus 52: White eye 53: Black eye 62: White eye 63: Black eye 100: Virtual Spatial distribution system 110: HMD
112: Display unit 113: Face camera 114: HMD sensor 116: Headphone 117: Face camera 118: Microphone 120: Control device 121: Control unit 123: Storage unit 124: I / O interface 125: Communication interface 126: Bus 130: Position Sensor 140: Gaze sensor 200, 200A, 200B: Virtual space 210: Center position 300: Virtual camera 320: External controller 320L: Left hand external controller (controller)
320R: External controller for right hand (controller)
A, B: User B1: Boundary line X1: Intersection E, F: Body region R, R1, R2: Partial region S, S1, S2: Partial region V1, V2, V3: Movement vector Vc: Composite vector

Claims (9)

A sensor configured to detect movement of a part of the body of the first user; a first user terminal associated with the first user; a head mounted device; and associated with a second user. An information processing method executed by a computer in an information processing system including the second user terminal,
The information processing method includes:
(A) generating virtual space data including a first avatar associated with the first user and defining a virtual space provided to the second user;
(B) updating a visual field image displayed on the head mounted device at a first rate based on the movement of the head mounted device and the virtual space data;
(C) obtaining first position information of a part of the body of the first user at a second rate based on detection data detected by the sensor;
(D) generating second position information of a part of the body of the first avatar corresponding to the part of the body of the first user based on the first position information;
(E) transmitting the first location information or the second location information to the second user terminal at a third rate lower than the first rate and the second rate;
(F) updating the position of the part of the body of the first avatar in the virtual space at the first rate based on the second position information;
In the first position information, the part of the body of the first user among the plurality of partial areas constituting the first body area including the part of the body of the first user is currently located. Information indicating the first partial region,
In the second position information, the part of the body of the first avatar among the plurality of partial areas constituting the second body area including the part of the body of the first avatar is currently located. Information indicating the second partial region,
In step (e),
An information processing method for transmitting the first position information or the second position information generated immediately before transmitting the first position information or the second position information to the second user terminal. The portion of the first user's body is the first user's black eye;
The part of the body of the first avatar is the black eye of the first avatar;
The first body region includes a white eye of the first user;
The information processing method according to claim 1, wherein the second body region includes a white eye of the first avatar. In step (f),
When the part of the body of the first user moves from the first partial area to a third partial area that is a partial area to which the part of the body of the first user is moved, The one of the body of the first avatar is gradually moved so that the part of the body gradually moves toward a center position of a fourth partial area that is a partial area to which the part of the body of the first avatar is moved. The information processing method according to claim 1, wherein the position of the part is updated. (G) obtaining first direction information of the part of the body of the first user;
(H) acquiring the second direction information of the part of the body of the first avatar based on the one direction information; and
The first direction information is information indicating a third partial area that is a partial area of the part of the body of the first user.
The second direction information is information indicating a fourth partial area that is a partial area of the part of the body of the first avatar.
The step (e) includes transmitting the one-direction information or the second direction information at the third rate,
The step (f) updates the position of the part of the body of the first avatar in the virtual space based on the second position information and the second direction information. The information processing method according to any one of the above. In step (f),
The position of the part of the body of the first avatar is updated so that the part of the body of the first avatar gradually moves in the second partial area toward the fourth partial area. 5. The information processing method according to 4.   6. The information processing method according to claim 4, wherein the third partial area is specified based on at least one movement direction of the part of the body of the first user that moves in the first partial area. .   An information processing program for causing a computer to execute the information processing method according to any one of claims 1 to 6. A first user terminal associated with the first user having a sensor configured to detect movement of a part of the body of the first user;
An information processing system comprising: a second user terminal associated with a second user, the information processing system having a display unit and configured to execute the information processing method according to any one of claims 1 to 6 . A processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
The information processing apparatus that executes the information processing method according to any one of claims 1 to 6 when the computer-readable instruction is executed by the processor.