JP6738308B2 - Information processing method, program, virtual space distribution system and device - Google Patents

Description

The present disclosure relates to an information processing method, a program for causing a computer to execute the information processing method, a virtual space distribution system and a device.

It is known to share a virtual space via a network between a plurality of users who are equipped with a head mounted device (Head Mounted Device: hereinafter simply referred to as HMD). Here, the virtual space includes a VR (Virtual Reality) space, an AR (Augmented Reality) space, and an MR (Mixed Reality) space. For example, Patent Document 1 discloses a technique of sharing one VR space among a plurality of users and then controlling an avatar's movement (avatar's posture, etc.) according to the user's movement (eg, user's posture, etc.). Disclosure.

Japanese Patent No. 6017008

By the way, when the user wears the HMD, the motion of the avatar is controlled according to the motion of the user, while when the user does not wear the HMD, the motion of the avatar is controlled according to the motion of the user. Not done. In this respect, in the technique described in Patent Document 1, when two avatars (first avatar and second avatar) exist in the virtual space, the first user who operates the first avatar removes the HMD. In this case, the second user who operates the second avatar cannot grasp that the first user does not wear the HMD. For example, it is assumed that the first user removes the HMD due to sudden circumstances or the like while the first user and the second user are communicating in the virtual space. In this case, since the second user does not know that the first user does not wear the HMD, the second user may feel uncomfortable with the fact that the first avatar (first user) does not react. Thus, there is room for improving communication between users in a virtual space (or a virtual experience provided to users).

The present disclosure is directed to providing users with a rich virtual experience.

According to one aspect which this indication shows, it has the 1st user terminal which has the 1st head mount device with which the head of the 1st user is equipped, and the 2nd head mount device with which the head of the 2nd user is equipped. An information processing method executed by a computer in a virtual space distribution system including a second user terminal,
The information processing method is
(A) generating virtual space data defining a virtual space including a first avatar associated with the first user and a second avatar associated with the second user;
(B) updating a visual field image displayed on the second head-mounted device based on the movement of the second head-mounted device and the virtual space data;
(D) If the first user receives non-wearing information indicating that the first head mount device is not installed, the first user does not install the first head mount device. Presenting to the second user by a second user terminal.

According to the present disclosure, it is possible to provide a user with a rich virtual experience.

1 is a schematic diagram showing a virtual space distribution system according to an embodiment of the present invention (hereinafter, simply referred to as the present embodiment). It is a schematic diagram showing a user terminal. It is a figure which shows the head of the user who mounted|worn HMD. It is a figure which shows the hardware constitutions of a control apparatus. It is a flowchart which shows the process which displays a visual field image on HMD. It is an xyz space diagram which shows an example of virtual space. State (a) is a yx plan view of the virtual space shown in FIG. 6. State (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows the hardware constitutions of the server shown in FIG. State (a) is a diagram showing a virtual space provided to the user A. The state (b) is a diagram showing the virtual space provided to the user B. It is a sequence diagram for explaining an example of a process of synchronizing the movement of each avatar between user terminals. It is an example of a flowchart for explaining the information processing method according to the present embodiment when the user A removes the HMD. State (a) is an example of a flowchart for explaining a process for determining whether or not the user A has removed the HMD using the HMD sensor. State (b) is an example of a flowchart for explaining the process for determining whether or not the user A has removed the HMD using the mounting sensor. It is a figure which shows an example of the virtual space provided to the user B for demonstrating the information processing method which concerns on this embodiment. It is a figure which shows another example of the virtual space provided to the user B for demonstrating the information processing method which concerns on this embodiment. It is an example of a flowchart for explaining the information processing method according to the present embodiment when the user A wears the HMD again. It is a figure which shows another example of the virtual space provided to the user B for demonstrating the information processing method which concerns on this embodiment.

[Description of Embodiments Presented by the Present Disclosure]
The outline of the embodiments shown in the present disclosure will be described.
(1) A first user terminal having a first head mounted device mounted on the head of a first user and a second user terminal having a second head mounted device mounted on the head of a second user. An information processing method executed by a computer in a provided virtual space distribution system,
The information processing method is
(A) generating virtual space data defining a virtual space including a first avatar associated with the first user and a second avatar associated with the second user;
(B) updating a visual field image displayed on the second head-mounted device based on the movement of the second head-mounted device and the virtual space data;
(D) If the first user receives non-wearing information indicating that the first head mount device is not installed, the first user does not install the first head mount device. Presenting to the second user with a second user terminal.

According to the above method, when the non-wearing information indicating that the first user does not wear the first head mounted device (hereinafter, first HMD) is received, the first user makes the first HMD by the second user terminal. The second user is presented that the user is not wearing. In this way, the second user can easily understand that the first user does not wear the first HMD while communicating with the first user in the virtual space. Therefore, it is possible to provide the user with a rich virtual experience.

In particular, if the second user does not know that the first user does not wear the first HMD, the second user may feel uncomfortable with the first user who does not show the reaction. In this way, by letting the second user understand that the first user does not wear the first HMD, it is possible to avoid a situation in which the second user feels uncomfortable with the first user who does not show a reaction. It will be possible.

(2) The step (d) includes
An item including the step of visualizing that the first user does not wear the first head mounted device on a visual field image displayed on the second head mounted device when the non-wearing information is received. The information processing method described in 1).

According to the above method, when the non-wearing information indicating that the first user does not wear the first head mount device (first HMD) is received, the first user does not wear the first HMD. The information shown is visualized on the visual field image displayed on the second head mounted device (second HMD). As described above, the second user visually recognizes the information indicating that the first user does not wear the first HMD through the second HMD while communicating with the first user in the virtual space. It is possible to easily understand that one user does not wear the first HMD.

(3) (e) further including a step of updating the facial expression of the first avatar based on the facial information when the facial information indicating the facial expression of the first avatar is received,
The step (d) includes
Item (1) or (including the step of visualizing the first avatar whose facial expression is set to the first aspect on the visual field image displayed on the second head mounted device when the non-wearing information is received. The information processing method described in 2).

According to the above method, when the non-wearing information indicating that the first user is not wearing the first head mounted device (first HMD) is received, the first avatar whose facial expression is set to the first mode Are visualized on the visual field image displayed on the second head mounted device (second HMD). In this way, the second user visually recognizes that the facial expression of the first avatar has a predetermined facial expression while communicating with the first user in the virtual space. Can easily grasp that the first HMD is not attached.

In particular, when the first user does not wear the first HMD, the facial expression of the first avatar is not updated at all. Therefore, the second user may feel uncomfortable with the first user if he/she does not know that the first user does not wear the first HMD. In this way, by letting the second user understand that the first user does not wear the first HMD, it is possible to avoid a situation in which the second user feels uncomfortable with the first user who does not show a reaction. It will be possible.

(4) The information processing method according to item (3), wherein the first aspect is a facial expression of the first avatar set as default.

According to the above method, when the non-wearing information indicating that the first user does not wear the first head mounted device (first HMD) is received, the facial expression of the first avatar is set to the default value. It becomes the facial expression of 1 avatar. In this way, the second user can visually recognize that the facial expression of the first avatar is the default facial expression while communicating with the first user in the virtual space. , It is possible to easily understand that the first user does not wear the first HMD.

(5) The information processing method according to item (3), wherein the first aspect is a facial expression selected in advance as an aspect indicating that the first user does not wear the first head mount device.

According to the above method, when the non-wearing information indicating that the first user does not wear the first head mounted device (first HMD) is received, the facial expression of the first avatar is determined by the first user to be the first. The facial expression preselected by the first user (hereinafter referred to as the selected facial expression) is set as a mode indicating that the head mounted device is not mounted. As described above, the second user visually recognizes that the facial expression of the first avatar is the selected facial expression while communicating with the first user in the virtual space. It is possible to easily understand that the first HMD is not attached.

(6) The first user terminal further includes a mounting sensor configured to detect whether the first user is wearing the first head mounted device,
When the wearing sensor detects that the first user does not wear the first head mount device, the first user terminal outputs the non-wearing information, and the second user terminal outputs. The information processing method according to any one of items (1) to (5), in which the non-wearing information that is set is received from the first user terminal.

According to the above method, it is specified that the first user does not wear the first head mount device (first HMD) according to the information transmitted from the wearing sensor of the first user terminal. After that, non-wearing information indicating that the first user does not wear the first HMD is transmitted. In this way, it is possible to automatically specify that the first user does not wear the first HMD, according to the information output from the wearing sensor of the first user terminal.

(7) (f) If the first head mount device receives mounting information indicating that the first head mount device has been detached from the first user and then reattached, the first user attaches the first head mount device. The information processing method according to any one of items (1) to (6), further including the step of presenting to the second user that the second user terminal is performing

According to the above method, when the mounting information indicating that the first head mount device (first HMD) has been detached from the first user and then reattached is received, the second user terminal causes the first user to move the first HMD to the first HMD. The second user is presented that the wearer is wearing. In this way, the second user can easily recognize that the first user has reattached the first HMD. Therefore, it is possible to provide the user with a rich virtual experience.

(8) A program for causing a computer to execute the information processing method according to any one of items (1) to (7).

According to the above program, it is possible to provide the user with a rich virtual experience.

(9) A first user terminal having a first head mounted device mounted on the head of the first user and a second user terminal having a second head mounted device mounted on the head of the second user. A virtual space distribution system comprising: the information processing method according to any one of items (1) to (7).

According to the above virtual space distribution system, a rich virtual experience can be provided to the user.

(10) A processor,
A device having a memory for storing computer readable instructions,
An apparatus for executing the information processing method according to any one of items (1) to (7) when the computer-readable instructions are executed by the processor.

According to the above device, a rich virtual experience can be provided to the user.

[Details of Embodiments Presented in the Present Disclosure]
Hereinafter, embodiments shown in the present disclosure will be described with reference to the drawings. For the sake of convenience of description, the description of the elements having the same reference numbers as the elements already described in the description of the present embodiment will not be repeated.

First, an outline of the configuration of the virtual space distribution system 100 will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter, simply referred to as a distribution system 100). As illustrated in FIG. 1, the distribution system 100 includes a user terminal 1A (first user terminal) operated by a user A (first user) wearing a head mounted device (HMD) 110 (first HMD), and an HMD 110 ( A user terminal 1B (second user terminal) operated by a user B (second user) wearing a second HMD) and a server 2 are provided. The user terminals 1A and 1B are communicatively connected to the server 2 via a communication network 3 such as the Internet. In the present embodiment, the virtual space includes a VR (Virtual Reality) space, an AR (Argented Reality) space, and an MR (Mixed Reality) space. In addition, hereinafter, for convenience of description, each of the user terminals 1A and 1B may be simply referred to as the user terminal 1. Further, each user A and B may be simply referred to as a user U. Moreover, in this embodiment, the user terminals 1A and 1B are assumed to have the same configuration.

Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic diagram showing the user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mounted device (HMD) 110 mounted on the head of the user U, headphones 116, a face camera 117, a microphone 118, a position sensor 130, and an external controller. 320 and the control device 120.

The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, a face camera 113, and a mounting sensor 115. The display unit 112 includes a non-transmissive display device configured to completely cover the visual field (visual field) of the user U wearing the HMD 110. Accordingly, the user U can immerse himself in the virtual space by seeing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the left eye of the user U and a right-eye display unit configured to provide an image to the right eye of the user U. Good. Further, the HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance thereof.

The HMD sensor 114 is mounted near the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and an inclination sensor (such as an angular velocity sensor and a gyro sensor), and detects various movements (inclination) of the HMD 110 mounted on the head of the user U. be able to.

The gaze sensor 140 has an eye tracking function of detecting the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right-eye gaze sensor irradiates the right eye of the user U with infrared light, for example, and detects reflected light reflected from the right eye (particularly, the cornea and the iris) to acquire information about the rotation angle of the eyeball of the right eye. You may. On the other hand, the left-eye gaze sensor irradiates the left eye of the user U with, for example, infrared light, and detects reflected light reflected from the left eye (particularly, the cornea and the iris) to detect information about the rotation angle of the eyeball of the left eye. May be obtained.

The face camera 113 is configured to acquire an image (particularly, a moving image) in which the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user U are displayed while the HMD 110 is attached to the user U. Has been done. The face camera 113 is arranged at a predetermined position inside the HMD 110 so that the eye and eyebrows of the user U can be imaged. The frame rate of the moving image acquired by the face camera 113 may be higher than the frame rate of the visual field image (described later) displayed on the HMD 110.

The mounting sensor 115 is configured to generate event information for indicating whether the user U wears the HMD 110. Specifically, when the user U removes the HMD 110 (when the state of the HMD 110 transits from the mounted state to the non-mounted state), the mounting sensor 115 generates event information and then controls the event information. To the device 120. Further, when the user U wears the HMD 110 (when the state of the HMD 110 transits from the non-wearing state to the wearing state), the wearing sensor 115 generates event information and then sends the event information to the control device 120. Send. Further, the configuration of the mounting sensor 115 is not particularly limited. For example, assume that when the user U wears the HMD 110, the elastic force of the spring provided in the HMD 110 changes. In this case, the mounting sensor 115 may generate event information based on the change in the elastic force of the spring. Further, it is assumed that when the user U wears the HMD 110, the current flowing between two pads that contact a part of the body (nose portion) provided on the HMD 110 changes. In this case, the mounting sensor 115 may generate event information based on the change in the current flowing between the two pads.

The headphones 116 (audio output unit) are attached to the left and right ears of the user U, respectively. The headphones 116 are configured to receive voice data (electrical signal) from the control device 120 and output voice based on the received voice data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Further, the microphone 118 is configured to transmit audio data to the control device 120.

The face camera 117 is configured to acquire an image (particularly, a moving image) in which the mouth of the user U and its periphery are displayed. The face camera 117 may be arranged at a position facing the mouth of the user U so that the mouth of the user U and its periphery can be imaged. The face camera 117 may be connected to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be higher than the frame rate of the visual field image displayed on the HMD 110.

The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicatively connected to the control device 120 wirelessly or by wire, and is configured to detect information on positions, inclinations, or emission intensities of a plurality of detection points (not shown) provided on the HMD 110. .. Further, the position sensor 130 is configured to detect information regarding the position, inclination, and/or emission intensity of a plurality of detection points (not shown) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared rays or visible light. In addition, the position sensor 130 may include an infrared sensor or a plurality of optical cameras.

The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, which is the hand of the user U in this embodiment), and thus the hand of the avatar displayed in the virtual space. Used to control movement. The external controller 320 includes a right-hand external controller 320R operated by the right hand of the user U (hereinafter, simply referred to as controller 320R) and a left-hand external controller 320L operated by the user U's left hand (hereinafter, simply referred to as controller 320L). ), and have. The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the fingers of the right hand. In addition, the right hand of the avatar existing in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the fingers of the left hand. In addition, the left hand of the avatar existing in the virtual space moves according to the movement of the controller 320L.

The controller 120 is a computer configured to control the HMD 110. The control device 120 specifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and based on the specified position information, the position of the virtual camera in the virtual space (the position of the avatar) and the reality. The position of the user U wearing the HMD 110 in the space can be accurately associated. Furthermore, the control device 120 specifies the operation of the external controller 320 based on the information acquired from the position sensor 130 and/or the sensor incorporated in the external controller 320, and based on the specified operation of the external controller 320. Thus, the motion of the avatar's hand displayed in the virtual space and the motion of the external controller 320 in the physical space can be accurately associated. In particular, the control device 120 identifies the operation of the controller 320L based on the information obtained from the position sensor 130 and/or the sensor incorporated in the controller 320L, and based on the identified operation of the controller 320L, the virtual device The left hand motion of the avatar displayed in the space and the motion of the controller 320L in the real space (the left hand motion of the user U) can be accurately associated with each other. Similarly, the control device 120 specifies the operation of the controller 320R based on the information acquired from the position sensor and/or the sensor incorporated in the controller 320R, and based on the specified operation of the controller 320R, the virtual space. The motion of the right hand of the avatar displayed inside and the motion of the controller 320R in the real space (motion of the right hand of the user U) can be accurately associated.

Further, the control unit 121 can associate the facial expression (state) of the user U and the facial expression (state) of the avatar of the user U based on the moving images acquired by the face cameras 113 and 117. Details of the process of associating the facial expression of the user U with the facial expression of the avatar will be described later.

In addition, the control device 120 specifies the line of sight of the right eye and the line of sight of the user U, respectively, based on the information transmitted from the gaze sensor 140 (the gaze sensor for the left eye and the gaze sensor for the right eye), and determines the line of sight of the right eye. The gazing point, which is the intersection of the line of sight of the left eye, can be specified. Furthermore, the control device 120 can specify the line of sight of the eyes of the user U (the line of sight of the user U) based on the specified gazing point. Here, the line-of-sight of the user U is the line-of-sight of the eyes of the user U, and coincides with the direction of a straight line passing through the midpoint of the line segment connecting the right and left eyes of the user U and the gazing point.

Next, with reference to FIG. 3, a method of acquiring information regarding the position and inclination of the HMD 110 will be described. FIG. 3 is a diagram showing the head of the user U wearing the HMD 110. Information regarding the position and inclination of the HMD 110 that is linked to the movement of the head of the user U who wears the HMD 110 can be detected by the position sensor 130 and/or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined around the head of the user U wearing the HMD 110. The vertical direction in which the user U stands upright is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the direction orthogonal to the v-axis and the w-axis is defined as the u-axis. The position sensor 130 and/or the HMD sensor 114 has an angle about each uvw axis (that is, a yaw angle indicating rotation about the v axis, a pitch angle indicating rotation about the u axis, and a w axis). The inclination determined by the roll angle indicating the rotation) is detected. The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected change in angle around each uvw axis.

Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram showing a hardware configuration of the control device 120. As shown in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I/O (input/output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I/O interface 124, and the communication interface 125 are communicably connected to each other via a bus 126.

The control device 120 may be configured as a personal computer, a smartphone, a phablet, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. Further, a part of the functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

The control unit 121 includes a memory and a processor. The memory includes, for example, a ROM (Read Only Memory) storing various programs and the like, and a RAM (Random Access Memory) having a plurality of work areas storing various programs executed by the processor and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and/or a GPU (Graphics Processing Unit), and a program specified by various programs installed in the ROM is expanded on the RAM, and the RAM is stored. It is configured to execute various kinds of processing in cooperation with.

In particular, the control unit 121 may control various operations of the control device 120 by the processor developing the control program on the RAM and executing the control program in cooperation with the RAM. The control unit 121 displays the visual field image on the display unit 112 of the HMD 110 based on the visual field image data. As a result, the user U can immerse himself in the virtual space.

The storage unit (storage) 123 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 123 may store the authentication program of the user U and data regarding various images and objects (for example, avatars). Further, the storage unit 123 may be constructed with a database including tables for managing various data.

The I/O interface 124 is configured to communicatively connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118 to the control device 120, respectively. For example, it is configured by a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control device 120 may be wirelessly connected to each of the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118.

The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as the server 2 via the communication network 3 and various processing circuits for wireless connection, and for communicating via the communication network 3. It is configured to comply with the communication standard of.

Next, the processing for displaying the visual field image on the HMD 110 will be described with reference to FIGS. 5 to 8. FIG. 5 is a flowchart showing a process of displaying a visual field image on the HMD 110. FIG. 6 is an xyz space diagram showing an example of the virtual space 200. The state (a) of FIG. 7 is a yx plan view of the virtual space 200 shown in FIG. The state (b) of FIG. 7 is a zx plan view of the virtual space 200 shown in FIG. FIG. 8 is a diagram showing an example of the visual field image V displayed on the HMD 110.

As shown in FIG. 5, in step S1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 6, the virtual space 200 is defined as a celestial sphere centered on the center position 210 (in FIG. 6, only the upper half celestial sphere is shown). Further, in the virtual space 200, an xyz coordinate system having a center position 210 as an origin is set. The virtual camera 300 defines the visual axis L for specifying the visual field image V (see FIG. 8) displayed on the HMD 110. The uvw coordinate system that defines the field of view of the virtual camera 300 is determined to be linked to the uvw coordinate system that is defined with the head of the user U in the physical space as the center. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 in association with the movement of the user U wearing the HMD 110 in the real space.

Next, in step S2, the control unit 121 specifies the field of view CV (see FIG. 7) of the virtual camera 300. Specifically, the control unit 121 acquires information regarding the position and inclination of the HMD 110 based on the data indicating the state of the HMD 110 transmitted from the position sensor 130 and/or the HMD sensor 114. Next, the control unit 121 identifies the position and orientation of the virtual camera 300 in the virtual space 200 based on the information regarding the position and inclination of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300, and specifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a partial area of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, a partial area of the virtual space 200 displayed on the HMD 110). Equivalent to). Further, the visual field CV has a first area CVa set as an angular range of the polar angle α around the visual axis L on the xy plane shown in the state (a) of FIG. 7 and a state (b) of FIG. On the xz plane shown, there is a second region CVb set as an angular range of the azimuth angle β about the visual axis L. Note that the control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the identified line of sight of the user U and the information regarding the position and inclination of the HMD 110. , The orientation of the virtual camera 300 (field L of the virtual camera) may be determined. Further, as will be described later, the control unit 121 may determine the orientation of the face of the avatar of the user U based on the information regarding the position and inclination of the HMD 110.

In this way, the control unit 121 can specify the visual field CV of the virtual camera 300 based on the data from the position sensor 130 and/or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 updates the visual field CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and/or the HMD sensor 114. be able to. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the visual field CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change of the line of sight of the user U.

Next, in step S3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates view image data based on the virtual space data defining the virtual space 200 and the view CV of the virtual camera 300.

Next, in step S4, the control unit 121 displays the visual field image V on the display unit 112 of the HMD 110 based on the visual field image data (see FIG. 7). As described above, the visual field CV of the virtual camera 300 changes according to the movement of the user U wearing the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 changes. You can immerse yourself in 200.

The virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates the left-eye view image data indicating the left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates the right-eye view image data indicating the right-eye view image based on the virtual space data and the view of the right-eye virtual camera. After that, the control unit 121 displays the visual field image for the left eye on the display unit for the left eye based on the visual field image data for the left eye, and the visual field image for the right eye on the display unit for the right eye based on the visual field image data for the right eye. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally by the parallax between the visual field image for the left eye and the visual field image for the right eye. The virtual camera may be arranged at the position of the eyes of the avatar operated by the user, as described later. For example, the left-eye virtual camera may be placed on the left eye of the avatar, while the right-eye virtual camera may be placed on the right eye of the avatar.

Further, the processing of steps S1 to S4 shown in FIG. 5 may be executed for each frame (still image forming a moving image). For example, when the frame rate of the moving image is 90 fps, the processes of steps S1 to S4 may be repeatedly executed at ΔT=1/90 (second) intervals. In this way, since the processes of steps S1 to S4 are repeatedly executed at predetermined intervals, the visual field of the virtual camera 300 is updated according to the operation of the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 is updated. Will be updated.

Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 9 is a diagram showing a hardware configuration of the server 2. As shown in FIG. 9, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are communicably connected to each other via a bus 24. The control unit 23 includes a memory and a processor. The memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU and/or a GPU.

The storage unit (storage) 22 is, for example, a large-capacity HDD or the like. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 22 may store user management information for managing each user and data regarding various images and objects (for example, avatars). The communication interface 21 is configured to connect the server 2 to the communication network 3.

Next, an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to FIGS. The state (a) of FIG. 10 is a diagram showing the virtual space 200A provided to the user A. The state (b) of FIG. 10 is a diagram showing the virtual space 200B provided to the user B. FIG. 11 is a sequence diagram for explaining an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as a precondition, as shown in FIG. 10, an avatar 4A (first avatar) associated with the user terminal 1A (user A) and an avatar 4B (first user) associated with the user terminal 1B (user B). 2 avatars) share the same virtual space. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

As shown in the state (a) of FIG. 10, the virtual space 200A of the user A includes an avatar 4A and an avatar 4B. The avatar 4A is operated by the user A and is linked to the operation of the user A. The avatar 4A includes the left hand (a part of the virtual body of the avatar 4A) that is linked to the operation of the controller 320L of the user terminal 1A (the left hand operation of the user A) and the operation of the controller 320R of the user terminal 1A (the right hand of the user A). The user has a right hand that is linked to the action) and a face whose facial expression is linked to the facial expression of the user A. The face of the avatar 4A has a plurality of face parts (for example, eyes, eyebrows, mouth, etc.). The avatar 4B is operated by the user B and is linked to the operation of the user B. The avatar 4B has a left hand that interlocks with the operation of the controller 320L of the user terminal 1B that indicates the operation of the left hand of the user B, a right hand that interlocks with the operation of the controller 320R of the user terminal 1B that indicates the operation of the right hand of the user B, and a facial expression. It has a face that is linked to the facial expression of the user B. The face of the avatar 4B has a plurality of face parts (for example, eyes, eyebrows, mouth, etc.).

In the virtual space 200A provided to the user A, the avatar 4A may not be visualized. In this case, the avatar 4A arranged in the virtual space 200A includes at least the virtual camera 300 that is interlocked with the movement of the HMD 110 of the user terminal 1A.

Further, the positions of the avatars 4A and 4B may be specified according to the positions of the HMD 110 of the user terminals 1A and 1B. Similarly, the orientation of the faces of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Furthermore, the hand motion of the avatars 4A and 4B may be specified according to the motion of the external controller 320 of the user terminals 1A and 1B. In particular, the left-hand operation of the avatars 4A and 4B is specified according to the operation of the controller 320L of the user terminals 1A and 1B, and the right-hand operation of the avatars 4A and 4B is specified according to the operation of the controller 320R of the user terminals 1A and 1B. May be specified. The facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B.

Further, a virtual camera 300 (FIG. 6) may be arranged on each eye of the avatars 4A and 4B. In particular, the left-eye virtual camera may be arranged on the left eyes of the avatars 4A and 4B, and the right-eye virtual camera may be arranged on the right eyes of the avatars 4A and 4B. In the following description, the virtual camera 300 is assumed to be placed in the eyes of the avatars 4A and 4B. Therefore, it is assumed that the visual field CV of the avatar 4A matches the visual field CV of the virtual camera 300 arranged on the avatar 4A (see FIG. 7). Similarly, it is assumed that the visual field CV of the avatar 4B matches the visual field CV of the virtual camera 300 arranged on the avatar 4B (see FIG. 7).

As shown in the state (b) of FIG. 10, the virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

In the virtual space 200B provided to the user B, the avatar 4B may not be visualized. In this case, the avatar 4B arranged in the virtual space 200B includes at least the virtual camera 300 that is interlocked with the movement of the HMD 110 of the user terminal 1B.

Next, referring to FIG. 11, in step S10, the control unit 121 of the user terminal 1A generates voice data of the user A. For example, when the user A inputs a voice into the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. After that, the microphone 118 transmits the generated voice data to the control unit 121 via the I/O interface 124.

Next, in step S11, the control unit 121 of the user terminal 1A generates control information of the avatar 4A, and then voice data indicating the control information of the generated avatar 4A and the voice of the user A (voice data of the user A). ) Is transmitted to the server 2. After that, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12).

Here, the control information of the avatar 4A is information necessary for controlling the operation of the avatar 4A. The control information of the avatar 4A includes information about the position of the avatar 4A (position information), information about the orientation of the face of the avatar 4A (face orientation information), and information about the state of the hand (left hand and right hand) of the avatar 4A (hand information). ) And information regarding the facial expression of the avatar 4A (face information).

The face information of the avatar 4A has information indicating the states of a plurality of face parts. The information indicating the state of the plurality of face parts includes information indicating the state of the eyes of the avatar 4A (shape of the white eye and the position of the black eye with respect to the white eye) and eyebrow information of the avatar 4A (position and shape of the eyebrow). Etc.) (eyebrow information) and information (mouth information) indicating the mouth state (mouth position and shape, etc.) of the avatar 4A.

Specifically, the eye information of the avatar 4A has information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyebrow information of the avatar 4A has information indicating the state of the left eyebrow of the avatar 4A and information indicating the state of the right eyebrow of the avatar 4A.

The control unit 121 of the user terminal 1A acquires an image showing the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user A from the face camera 113 mounted on the HMD 110, and then the acquired image. And the state of the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user A are specified based on a predetermined image processing algorithm. Next, the control unit 121, based on the identified eye and eyebrow state of the user A, information indicating the state of the eyes (left eye and right eye) of the avatar 4A and the eyebrow (left eyebrow and right eyebrow) of the avatar 4A. Each piece of information indicating the state is generated.

Similarly, the control unit 121 of the user terminal 1A acquires an image showing the mouth of the user A and its surroundings from the face camera 117, and then the mouth of the user A based on the acquired image and a predetermined image processing algorithm. Identify the state of. Next, the control unit 121 generates information indicating the mouth state of the avatar 4A based on the identified mouth state of the user A.

As described above, the control unit 121 of the user terminal 1A controls the eye information of the avatar 4A corresponding to the eyes of the user A, the eyebrow information of the avatar 4A corresponding to the eyebrows of the user A, and the avatar 4A corresponding to the mouth of the user A. It is possible to generate the mouth information of.

In addition, the control unit 121 controls the image captured by the face camera 113, the image captured by the face camera 117, and a plurality of types of faces stored in the storage unit 123 or the memory based on a predetermined image processing algorithm. The facial expression (eg, smile) of the user A may be specified from the facial expressions (eg, smile, sad expression, no expression, angry expression, surprised expression, troubled expression, etc.). After that, the control unit 121 can generate face information indicating the facial expression (for example, a smile) of the avatar 4A based on the identified facial expression of the user A. In this case, facial expression data including a plurality of types of facial expressions of the avatar 4A and facial information of the plurality of avatars 4A, each of which is associated with one of the plurality of types of facial expressions, is stored in the storage unit 123. May be.

For example, when the facial expression of the avatar 4A is specified as a smile, the control unit 121 acquires face information indicating the smiling face of the avatar 4A based on the facial expression (smile) of the avatar 4A and the facial expression data. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information, and mouth information of the avatar 4A when the facial expression of the avatar 4A is a smile.

Next, in step S13, the control unit 121 of the user terminal 1B generates control information of the avatar 4B, and then transmits the generated control information of the avatar 4B to the server 2. After that, the control unit 23 of the server 2 receives the control information of the avatar 4B from the user terminal 1B (step S14).

Here, the control information of the avatar 4B is information necessary for controlling the operation of the avatar 4B. The control information of the avatar 4B includes information regarding the position of the avatar 4B (position information), information regarding the orientation of the face of the avatar 4B (face orientation information), and information regarding the state of the hand (left hand and right hand) of the avatar 4B (hand information). ) And information about the facial expression of the avatar 4B (face information).

The face information of the avatar 4B has information indicating the states of a plurality of face parts. The information indicating the states of the plurality of face parts includes information indicating the state of the eyes of the avatar 4B (the shape of the white eye and the position of the black eye with respect to the white eyes), and the eyebrow information of the avatar 4B (the position and the shape of the eyebrow). Etc.) (eyebrows information) and information (mouth information) indicating the mouth state (mouth position and shape, etc.) of the avatar 4B. The method of acquiring the face information of the avatar 4B is the same as the method of acquiring the face information of the avatar 4A.

Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S15), while transmitting the control information of the avatar 4A and the voice data of the user A to the user terminal 1B (step S19). After that, the control unit 121 of the user terminal 1A receives the control information of the avatar 4B in step S16, then updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B, and then the virtual space 200A. The virtual space data indicating the state (see state (a) in FIG. 10) is updated (step S17).

Specifically, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientations of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the face information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

Then, the control unit 121 of the user terminal 1A specifies the visual field CV of the avatar 4A (virtual camera 300) according to the position and the inclination of the HMD 110, and then sets the updated virtual space data and the visual field CV of the avatar 4A. Based on this, the visual field image displayed on the HMD 110 is updated (step S18).

On the other hand, the control unit 121 of the user terminal 1B updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B after receiving the control information of the avatar 4A and the voice data of the user A in step S20. Above, the virtual space data indicating the virtual space 200B (see the state (b) in FIG. 10) is updated (step S21).

Specifically, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientations of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the face information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

Then, the control unit 121 of the user terminal 1B specifies the visual field CV of the avatar 4B (virtual camera 300) according to the position and the inclination of the HMD 110, and then sets the updated virtual space data and the visual field CV of the avatar 4B. Based on this, the visual field image displayed on the HMD 110 is updated (step S22).

After that, the control unit 121 of the user terminal 1B uses the received voice data of the user A, information about the position of the avatar 4A included in the control information of the avatar 4A, information about the position of the avatar 4B, and a predetermined voice processing algorithm. Based on this, the voice data of the user A is processed. After that, the control unit 121 transmits the processed audio data to the headphones 116 (audio output unit), and then the headphones 116 outputs the audio of the user A based on the processed audio data (step S23). In this way, voice chat (VR chat) between users (between avatars) can be realized in the virtual space.

In the present embodiment, after the user terminals 1A and 1B respectively transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, while the avatar 4B is transmitted. The control information of 1 is transmitted to the user terminal 1A. In this way, the movements of the avatars 4A and 4B can be synchronized between the user terminal 1A and the user terminal 1B.

Next, the information processing method according to the present embodiment when the HMD 110 of the user A is removed will be described by mainly referring to FIGS. 12 and 14. FIG. 12 is an example of a flowchart for explaining the information processing method according to the present embodiment when the user A removes the HMD 110. FIG. 14 is a diagram showing an example of the virtual space 200B provided to the user B for explaining the information processing method according to the present embodiment. Note that in the flowchart shown in FIG. 12, the process in which the control unit 121 of the user terminal 1A sends the control information of the avatar 4A to the server 2 and the process of the control unit 121 of the user terminal 1B sends the control information of the avatar 4B to the server 2. Are omitted for convenience of explanation.

In this embodiment, as a precondition, it is assumed that the avatar 4A and the avatar 4B share the same virtual space, as shown in FIG. That is, the users A and B share one virtual space via the communication network 3.

The virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The control unit 121 of the user terminal 1B updates the virtual space data indicating the virtual space 200B. In this description, for convenience of description, the illustration of the virtual space 200A provided to the avatar 4A is omitted.

Referring to FIG. 12, in step S30, the control unit 121 of the user terminal 1A determines whether the user A has removed the HMD 110. In this regard, the control unit 121 may determine whether the user A has removed the HMD 110 based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the mounting sensor 115. The HMD 114 and the mounting sensor 115 function as a sensor configured to detect whether or not the HMD 110 is mounted.

A process of determining whether the user A has removed the HMD 110 using the HMD sensor 114 will be described with reference to the state (a) of FIG. 13. The state (a) of FIG. 13 is an example of a flowchart for explaining the process for determining whether the user A has removed the HMD 110 using the HMD sensor 114. As shown in the state (a) of FIG. 13, the control unit 121 receives tilt information indicating the tilt (roll angle, yaw angle, pitch angle) of the HMD 110 from the HMD sensor 114 (step S301). Next, the control unit 121 determines whether the inclination of the HMD 110 is larger than a predetermined inclination based on the received inclination information (S302). Here, the predetermined inclination may be appropriately set by the user. It is assumed that the inclination of the HMD 110 does not become larger than the predetermined inclination while the user A is wearing the HMD 110. That is, while the user A is wearing the HMD 110, the inclination of the HMD 110 is less than or equal to the predetermined inclination, while when the user A removes the HMD 110, the HMD 110 is greatly inclined, so that the inclination of the HMD 110 is less than the predetermined inclination. It may exceed. From this point of view, the predetermined inclination is set.

For example, when the pitch angle θ of the HMD 110 is larger than the predetermined angle θth (YES in step S302), the control unit 121 removes the HMD 110 by the user A (in other words, as an example of the inclination of the HMD 110). , User A does not wear the HMD 110), and then the process of step S31 is executed. On the other hand, when the control unit 121 determines that the pitch angle θ of the HMD 110 is equal to or smaller than the predetermined angle θth (NO in step S302), it determines that the user A is wearing the HMD 110, and then the HMD sensor. It waits until it receives the tilt information of the HMD 110 from 114.

Next, with reference to the state (b) of FIG. 13, a process of determining whether or not the user A has removed the HMD 110 using the mounting sensor 115 will be described. The state (b) of FIG. 13 is an example of a flowchart for explaining a process for determining whether or not the user A has removed the HMD 110 by using the mounting sensor 115. As shown in the state (b) of FIG. 13, the control unit 121 determines whether or not the event information is received from the mounting sensor 115 (step S303). Here, the event information transmitted from the wearing sensor 115 is information indicating that the state of the HMD 110 of the user terminal 1A has changed between the wearing state and the non-wearing state. For example, when the user A removes the HMD 110 (when the state of the HMD 110 transits from the mounted state to the non-mounted state), the mounting sensor 115 outputs event information. When the user A wears the HMD 110 (when the state of the HMD 110 transits from the non-wearing state to the wearing state), the wearing sensor 115 outputs event information.

When the control unit 121 determines that the event information is received from the wearing sensor 115 (YES in step S303), the user A removes the HMD 110 (in other words, the user A does not wear the HMD 110). After specifying, the process of step S31 is executed. On the other hand, when the control unit 121 determines that the event information is not received from the mounting sensor 115 (NO in step S303), the user A specifies that the HMD 110 is mounted and then mounts the event information. It waits until it receives from the sensor 115.

Next, returning to FIG. 12, when the control unit 121 determines that the user A has removed the HMD 110 (in other words, the user A does not wear the HMD 110) (step S30 YES), the user A has the HMD 110. Information (non-wearing information) indicating that the wearer is not wearing is generated (step S31). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S32).

Next, the server 2 transmits the non-wearing information to the user terminal 1B via the communication network 3 after receiving the non-wearing information from the user terminal 1A (step S33). After that, the control unit 121 of the user terminal 1B receives the non-wearing information from the server 2 and then sets the facial expression of the avatar 4A placed in the virtual space 200B to the default facial expression (in other words, the default setting has been made). (Facial expression) (step S34). Here, the default facial expression of the avatar 4A means the facial expression of the avatar 4A in the initial state before the updating of the facial expression of the avatar 4A is started. When the default facial expression is smiling, the facial expression of the avatar 4A is set to smiling, as shown in FIG. The control unit 121 also updates the action of the avatar 4B (including the facial expression of the avatar 4B) based on the control information of the avatar 4B.

Note that in step S34, the control unit 121 may set the facial expression of the avatar 4A to the facial expression preselected by the user A as a mode indicating that the user A is not wearing the HMD 110. In this case, the information regarding the facial expression of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B and then stored in the storage unit 123 of the user terminal 1B in advance.

Next, as shown in FIG. 14, the control unit 121 generates a balloon object 42A associated with the avatar 4A in the virtual space 200B. Information indicating that the user A is not wearing the HMD 110 (for example, “User A is leaving”.) is displayed on the balloon object 42A.

The control unit 121 may generate the telop object 43A in the virtual space 200B instead of the balloon object 42A (see FIG. 15). As shown in FIG. 15, the telop object 43A displays information indicating that the user A does not wear the HMD 110 (for example, “User A is leaving”.). The telop object 43A may be arranged near the avatar 4A, or may be interlocked with the movement of the visual field CV of the avatar 4B so as to be always arranged within the visual field CV of the avatar 4B. Furthermore, a telop indicating information indicating that the user A does not wear the HMD 110 may be superimposed on the visual field image displayed on the HMD 110.

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B and also updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the visual field image data on the basis of the updated virtual flexion data and the updated visual field CV of the avatar 4B, and then sets the visual field image on the HMD 110 based on the updated visual field image data. It is displayed (step S36).

On the other hand, the control unit 121 of the user terminal 1A shifts from the active mode to the sleep mode after transmitting the non-wearing information to the server 2 via the communication network 3 (step S37). In this case, it is preferable that the mode of the control unit 121 is the sleep mode, while the mode of the sensor (the mounting sensor 115 or the HMD sensor 114) is the active mode.

According to this embodiment, when the user A does not wear the HMD 110, the user terminal 1B presents to the user B that the user A does not wear the HMD 110. In particular, according to this embodiment, when the user A does not wear the HMD 110, information indicating that the user A does not wear the HMD 110 is visualized on the visual field image displayed on the HMD 110 of the user terminal 1B. It At this point, the avatar 4A and the balloon object 42A whose facial expressions are set to the default facial expressions are visualized on the visual field image displayed on the HMD 110 of the user terminal 1B.

As described above, the user B visually recognizes the default facial expression (for example, smile) of the avatar 4A and the information displayed on the balloon object 42A while communicating with the user A in the virtual space 200B. It is possible to easily understand that the user A does not wear the HMD 110. Therefore, it is possible to provide the user with a rich virtual experience.

In particular, if the user B does not know that the user A does not wear the HMD 110, the user B may feel uncomfortable with the user A who does not show the reaction. In this way, by making the user B understand that the user A does not wear the HMD 100, the situation in which the user B feels strange (for example, an eerie valley phenomenon) to the user A who does not show the reaction is avoided. It becomes possible to do.

Further, according to this embodiment, the user A may not wear the HMD 110 according to the information (tilt information or event information) transmitted from the sensor (HMD sensor 114 or wearing sensor 115) of the user terminal 1A. Specified. After that, non-wearing information indicating that the user A does not wear the HMD 110 is transmitted to the server 2. As described above, it is possible to automatically specify that the user A does not wear the HMD 110 according to the information output from the sensor of the user terminal 1A.

In the present embodiment, the user terminal 1A transmits the non-wearing information to the user terminal 1B via the server 2, but instead of the non-wearing information, the tilt information transmitted from the HMD sensor 114 is used by the user terminal. It may be transmitted to 1B. In this case, the control unit 121 of the user terminal 1B specifies that the user A does not wear the HMD 110 based on the received tilt information, and then executes the processes defined in steps S34 to S36. Good.

Next, the information processing method according to the present embodiment when the user A who has removed the HMD 110 reattaches the HMD 110 will be described by mainly referring to FIGS. 16 and 17. FIG. 16 is an example of a flowchart for explaining the information processing method according to the present embodiment when the user A reattaches the HMD 110. FIG. 17 is a diagram showing an example of the virtual space 200B provided to the user B for explaining the information processing method according to the present embodiment.

Referring to FIG. 16, in step S40, the control unit 121 of the user terminal 1A determines whether the user A wears the HMD 110 again. At this point, the control unit 121 may determine whether the user A reattaches the HMD 110 based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the attachment sensor 115. .. For example, when the control unit 121 determines that the event information is received from the mounting sensor 115, the control unit 121 specifies that the user A has mounted the HMD 110, and then executes the process of step S41. On the other hand, when the control unit 121 determines that the event information is not received from the mounting sensor 115, the control unit 121 specifies that the user A does not wear the HMD 110, and then receives the event information from the mounting sensor 115. stand by.

If the decision result in the step S40 is YES, the control section 121 shifts from the sleep mode to the active mode (step S41). Next, the control unit 121 that has transitioned to the active mode generates control information for the avatar 4A and also generates mounting information indicating that the user A has mounted the HMD 110 (step S42). After that, in step S43, the control unit 121 transmits the control information and the wearing information of the avatar 4A to the server 2 via the communication network 3 (step S43).

Next, the server 2 receives the control information and the wearing information of the avatar 4A from the user terminal 1A, and then transmits the control information and the wearing information of the avatar 4A to the user terminal 1B via the communication network 3 (step S44). Then, in step S45, the control unit 121 of the user terminal 1B updates the operation of the avatar 4A based on the control information of the avatar 4A, and also updates the operation of the avatar 4B based on the control information of the avatar 4B. In particular, the control unit 121 updates the facial expression of the avatar 4A based on the face information of the avatar 4A included in the control information of the avatar 4A, and based on the face information of the avatar 4B included in the control information of the avatar 4B. Then, the facial expression of the avatar 4B is updated.

Next, as shown in FIG. 17, the control unit 121 generates a balloon object 44A associated with the avatar 4A in the virtual space 200B (step S46). The balloon object 44A displays information indicating that the user A wears the HMD 110 (for example, “User A has returned to the seat.”).

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B and also updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the visual field image data on the basis of the updated virtual flexion data and the updated visual field CV of the avatar 4B, and then sets the visual field image on the HMD 110 based on the updated visual field image data. It is displayed (step S47).

According to this embodiment, when the user A removes the HMD 110 and attaches it again, the user terminal 1B presents to the user B that the user A has attached the HMD 110. In this way, the user B can easily recognize that the user A has reattached the HMD 110 by visually checking the updated expression of the avatar 4A and the balloon object 44A. Therefore, it is possible to provide the user with a rich virtual experience.

In the description of the present embodiment, information indicating that the user A wears or does not wear the HMD 110 is visualized on the visual field image, but the present embodiment is not limited to this. For example, a voice guidance indicating that the user A does not wear the HMD 110 (for example, “User A has left the seat.”) or a voice guidance indicating that the user A wears the HMD 110 (for example, “User A You have returned to your seat.") may be output from the headphones 116 (audio output unit) of the user terminal 1B. In this case, the control unit 121 of the user terminal 1A transmits the voice guidance data to the user terminal 1B together with the non-wearing information (or the wearing information) via the server 2.

Further, in the description of this embodiment, it is assumed that the virtual space data indicating the virtual space 200B is updated on the user terminal 1B side, but the virtual space data may be updated on the server 2 side. Furthermore, it is assumed that the view image data corresponding to the view image is updated on the user terminal 1B side, but the view image data may be updated on the server 2 side. In this case, the user terminal 1B displays the visual field image on the HMD 110 based on the visual field image data transmitted from the server 2.

Further, the order of processing defined by each step shown in FIGS. 12 and 16 is merely an example, and the order of these steps can be changed appropriately.

Further, in order to realize various processes executed by the control unit 121 of the user terminal 1 by software, a control program for causing a computer (processor) to execute various processes may be incorporated in the storage unit 123 or the memory in advance. Good. Alternatively, the control program is a magnetic disk (HDD, floppy disk), optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), magneto-optical disk (MO, etc.), flash memory (SD card, USB memory, SSD). Etc.) etc. may be stored in a computer-readable storage medium. In this case, by connecting the storage medium to the control device 120, the control program stored in the storage medium is incorporated in the storage unit 123. Then, the control program incorporated in the storage unit 123 is loaded on the RAM, and the processor executes the loaded program, whereby the control unit 121 executes various processes.

Further, the control program may be downloaded from a computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded control program is similarly incorporated in the storage unit 123.

Although the embodiments of the present disclosure have been described above, the technical scope of the present invention should not be limitedly interpreted by the description of the present embodiments. This embodiment is an example, and it is understood by those skilled in the art that various embodiments can be modified within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalent scope thereof.

1, 1A, 1B: user terminal 2: server 3: communication network 4A, 4B: avatar 21: communication interface 22: storage unit 23: control unit 24: bus 42A, 44A: balloon object 43A: telop object 100: virtual space distribution System (delivery system)
110: HMD (head mount device)
112: display unit 113: face camera 114: HMD sensor 115: mounting sensor 116: headphone 117: face camera 118: microphone 120: control device 123: storage unit 124: I/O interface 125: communication interface 126: bus 130: position Sensor 140: Gaze sensors 200, 200A, 200B: Virtual space 210: Center position 300: Virtual camera 320: External controller 320L: Left-hand external controller (controller)
320R: External controller for right hand (controller)
U, A, B: User

Claims (8)

A program executed in a computer having a processor,
The program causes the processor to
Providing a virtual space including a first avatar corresponding to a first user wearing a first head mounted device;
Performing a first control for controlling a motion of the first avatar in the virtual space based on a motion of the first user;
Detecting that the first user has removed the first head mount device;
Executing a second control of performing a predetermined control on the first avatar when the removal is detected,
A program in which the second control is different from the first control. The detecting step includes receiving non-wearing information indicating that the first user has removed the first head mounted device,
The second control is
When receiving the non-wearing information, comprising the first user presents in the virtual space information indicating that not wearing the first head-mounted device,
The program according to claim 1. When the face information indicating the facial expression of the first avatar is received , which is information based on the action of the first user, the first control is based on the face information, and the first control is performed in the virtual space. includes updating the facial expressions of the first avatar,
The second control is
Updating the facial expression of the first avatar in the virtual space based on a first predetermined aspect when detecting the removal.
The program according to claim 1. The program according to claim 3, wherein the first aspect is a facial expression that has been preselected as an aspect indicating that the first user does not wear the first head mount device. The computer is used by a second user,
The first computer, which is used by the first user and has the first head mount device, further includes a mounting sensor configured to detect whether the first user wears the first head mount device. Prepare,
When the attachment sensor detects that the first user has not attached the first head mount device, the first computer indicates that the first user has removed the first head mount device. to transmit non-wearing information, wherein the processor in the sensing steps, to receive the non-mounting information via the communication program as claimed in any one of claims 1 4. The processor,
If the first head-mounted device receives an attachment information indicating that it is mounted again after being detached from the first user, information indicating that the first user is wearing the first head-mounted device The program according to any one of claims 1 to 5 , which causes the step of presenting a program in the virtual space to be executed . An information processing method executed in a computer including a processor,
The information processing method, in the processor,
Providing a virtual space including a first avatar corresponding to a first user wearing a first head mounted device;
Performing a first control for controlling a motion of the first avatar in the virtual space based on a motion of the first user;
Detecting that the first user has removed the first head mount device;
And performing a second control for performing a predetermined control on the first avatar when detecting the removal,
The information processing method, wherein the second control is different from the first control. An information processing device including a processor,
The processor is
Providing a virtual space including a first avatar corresponding to a first user wearing a first head mounted device,
Performing first control for controlling the movement of the first avatar in the virtual space based on the movement of the first user,
Detecting that the first user has removed the first head mount device,
When the removal is detected, a second control for performing a predetermined control on the first avatar is performed.
The information processing apparatus, wherein the second control is different from the first control.