JP7041484B2 - Programs, information processing devices, information processing systems, and information processing methods - Google Patents

Description

The present disclosure relates to programs, information processing devices, information processing systems, and information processing methods.

As in Non-Patent Document 1, there is known a service that allows a plurality of users sharing a virtual space to enjoy communication with body language using a virtual hand or the like.

"Social VR Demo --Selfie Stick and 360 Photo Spheres --Oculus", [online], April 13, 2016, Oculus, [Search on August 10, 2017], Internet <https://youtu.be/ -pumFtAjgLY? t = 5 ＞

However, in the service as described in Non-Patent Document 1, there is room for improvement in order to improve the virtual experience of the user.

The present disclosure is intended to improve the virtual experience of the user.

According to one aspect of the program presented in the present disclosure.
On a computer to provide a virtual experience to the user,
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
A step of executing the second action for the other user based on the execution of the first action, and
To execute.

According to the present disclosure, it is possible to improve the virtual experience of the user.

It is a schematic diagram which shows the virtual space distribution system which concerns on embodiment of this invention (hereinafter, simply referred to as this embodiment). It is a schematic diagram which shows the user terminal. It is a figure which shows the head of the user who wears an HMD. It is a figure which shows the hardware configuration of a control device. It is a flowchart which shows the process of displaying a field of view image on an HMD. It is an xyz space diagram which shows an example of a virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the field of view image displayed on the HMD. It is a figure which shows the hardware configuration of the server shown in FIG. The state (a) is a diagram showing a virtual space provided to the user A. The state (b) is a diagram showing a virtual space provided to the user B. It is a sequence diagram for demonstrating an example of the process of synchronizing the movement of each avatar between user terminals. This is an example of a flowchart for explaining the information processing method according to the present embodiment when the user A removes the HMD. The state (a) is an example of a flowchart for explaining a process for determining whether or not the user A has removed the HMD by using the HMD sensor. The state (b) is an example of a flowchart for explaining a process for determining whether or not the user A has removed the HMD by using the wearing sensor. It is a figure which shows an example of the virtual space provided to the user B for explaining the information processing method which concerns on this embodiment. It is a figure which shows another example of the virtual space provided to the user B for explaining the information processing method which concerns on this embodiment. This is an example of a flowchart for explaining the information processing method according to the present embodiment when the user A reattaches the HMD. It is a figure which shows another example of the virtual space provided to the user B for explaining the information processing method which concerns on this embodiment. This is an example of a flowchart for explaining the information processing method according to the first modification. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 1. FIG. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 1. FIG. It is a figure which shows the display example of the user terminal 5 associated with the user A for demonstrating the information processing method which concerns on modification 1. FIG. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 1. FIG. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 1. FIG. This is an example of a flowchart for explaining the information processing method according to the first modification when the user A does not wear the HMD within a certain period of time after receiving the read information. It is an example of the flowchart for explaining the information processing method which concerns on modification 2. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 2. FIG. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 2. FIG. It is a figure which shows an example of the virtual space 200B provided to the user B for explaining the information processing method which concerns on modification 2. FIG.

[Explanation of Embodiments Shown in the Present Disclosure]
An outline of the embodiments shown in the present disclosure will be described.
(1) A first user terminal having a first head mount device mounted on the head of a first user and a second user terminal having a second head mount device mounted on the head of a second user. It is an information processing method executed by a computer in a virtual space distribution system provided.
The information processing method is
(A) A step of generating virtual space data defining a virtual space including the first avatar associated with the first user and the second avatar associated with the second user.
(B) A step of updating the field image displayed on the second head mount device based on the movement of the second head mount device and the virtual space data.
(D) When the first user receives the non-wearing information indicating that the first head mount device is not worn, the first user does not wear the first head mount device. An information processing method including a step of presenting to the second user by the second user terminal.

According to the above method, when the non-wearing information indicating that the first user is not wearing the first head mount device (hereinafter referred to as the first HMD) is received, the first user is the first HMD by the second user terminal. It is presented to the second user that the device is not worn. In this way, the second user can easily grasp that the first user is not wearing the first HMD while communicating with the first user in the virtual space. Therefore, it is possible to provide the user with a rich virtual experience.

In particular, the second user may feel uncomfortable with the first user who does not show a reaction unless he / she knows that the first user is not wearing the first HMD. In this way, by letting the second user know that the first user is not wearing the first HMD, it is possible to avoid a situation in which the second user feels uncomfortable with the first user who does not show a reaction. It will be possible.

(2) The step (d) is
An item (including a step of visualizing on a field image displayed on the second head mount device that the first user is not wearing the first head mount device when the non-wearing information is received. The information processing method described in 1).

According to the above method, when the non-wearing information indicating that the first user is not wearing the first head mount device (first HMD) is received, the first user is not wearing the first HMD. The information shown is visualized on the field image displayed on the second head mount device (second HMD). In this way, when the second user is communicating with the first user in the virtual space, the second user visually recognizes the information indicating that the first user is not wearing the first HMD through the second HMD. It can be easily grasped that one user does not wear the first HMD.

(3) (e) Further includes a step of updating the facial expression of the first avatar based on the facial information when the facial information indicating the facial expression of the first avatar is received.
The step (d) is
Item (1) or (including the step of visualizing the first avatar whose facial expression is set in the first aspect on the field image displayed on the second head mount device when the non-wearing information is received. The information processing method described in 2).

According to the above method, when the non-wearing information indicating that the first user is not wearing the first head mount device (first HMD) is received, the first avatar whose facial expression is set to the first aspect is received. Is visualized on the field image displayed on the second head mount device (second HMD). In this way, when the second user is communicating with the first user in the virtual space, the first user visually recognizes that the facial expression of the first avatar is a predetermined facial expression. Can easily grasp that the first HMD is not attached.

In particular, when the first user does not wear the first HMD, the facial expression of the first avatar is not updated at all. Therefore, the second user may feel uncomfortable with the first user unless he / she knows that the first user is not wearing the first HMD. In this way, by letting the second user know that the first user is not wearing the first HMD, it is possible to avoid a situation in which the second user feels uncomfortable with the first user who does not show a reaction. It will be possible.

(4) The information processing method according to item (3), wherein the first aspect is a facial expression of the first avatar set by default.

According to the above method, when the non-wearing information indicating that the first user is not wearing the first head mount device (first HMD) is received, the facial expression of the first avatar is set by default. 1 It becomes the facial expression of the avatar. In this way, the second user visually recognizes that the facial expression of the first avatar is the default facial expression when communicating with the first user in the virtual space. , It can be easily grasped that the first user does not wear the first HMD.

(5) The information processing method according to item (3), wherein the first aspect is a facial expression selected in advance as an aspect indicating that the first user is not wearing the first head mount device.

According to the above method, when the non-wearing information indicating that the first user is not wearing the first head mount device (first HMD) is received, the facial expression of the first avatar is the first user. 1 The facial expression selected in advance by the first user (hereinafter referred to as a selected facial expression) is set as an embodiment indicating that the head mount device is not attached. In this way, when the second user is communicating with the first user in the virtual space, the first user visually recognizes that the facial expression of the first avatar is the selected facial expression, so that the first user can use it. It can be easily grasped that the first HMD is not attached.

(6) The first user terminal further includes a mounting sensor configured to detect whether or not the first user is wearing the first head mount device.
When the mounting sensor detects that the first user is not wearing the first head mount device, the first user terminal outputs the non-wearing information, and the second user terminal outputs the non-wearing information. The information processing method according to any one of items (1) to (5), wherein the non-wearing information is received from the first user terminal.

According to the above method, it is specified that the first user is not wearing the first head mount device (first HMD) according to the information transmitted from the wearing sensor of the first user terminal. After that, non-wearing information indicating that the first user is not wearing the first HMD is transmitted. In this way, it is possible to automatically identify that the first user is not wearing the first HMD according to the information output from the wearing sensor of the first user terminal.

(7) (f) When the first user receives the mounting information indicating that the first head mount device has been removed from the first user and then mounted again, the first user wears the first head mount device. The information processing method according to any one of items (1) to (6), further comprising a step of presenting the information processing to the second user by the second user terminal.

According to the above method, when the mounting information indicating that the first head mount device (first HMD) has been removed from the first user and then mounted again is received, the first user is sent to the first HMD by the second user terminal. Is presented to the second user. In this way, the second user can easily know that the first user has reattached the first HMD. Therefore, it is possible to provide the user with a rich virtual experience.

(8) A program for causing a computer to execute the information processing method according to any one of items (1) to (7).

According to the above program, a rich virtual experience can be provided to the user.

(9) A first user terminal having a first head mount device mounted on the head of a first user and a second user terminal having a second head mount device mounted on the head of a second user. A virtual space distribution system configured to execute the information processing method according to any one of items (1) to (7).

According to the virtual space distribution system, it is possible to provide a rich virtual experience to the user.

(10) Processor and
A device equipped with a memory for storing computer-readable instructions.
A device that, when the computer-readable instruction is executed by the processor, executes the information processing method according to any one of items (1) to (7).

According to the above device, it is possible to provide a rich virtual experience to the user.

(11) A computer for providing a virtual experience to the user
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
A step of executing the second action for the other user based on the execution of the first action, and
A program to execute.

According to the above program, the virtual experience of the user can be improved. In particular, by executing an action among a plurality of users who share a virtual space by intuitive operation using an operation object, communication in the virtual space can be smoothly performed without impairing the immersive feeling.

(12) The program according to item (11), which executes the second action when the first action is executed in a state where the virtual object is not controlled by the other user.

According to the above program, another user can be attracted to the virtual space by executing the second action on the virtual object in the uncontrolled state.

(13) The program according to item (11) or (12), wherein the first action includes an action of touching the virtual object in the process of repetitive motion of the operation object.

(14) The program according to item (13), wherein the first action includes a contact operation with the virtual object in the process of the repetitive motion at an acceleration of a certain value or more.

(15) The program according to item (13) or (14), wherein the first action includes an action of passing through relative coordinates associated with the virtual object a predetermined number of times or more within a certain period of time.

According to these programs, it is possible to prevent erroneous detection of the operation of the operation object by not considering the case where the operation object accidentally touches the virtual object.

(16) The program according to any one of items (11) to (15), wherein the second action includes a notification for encouraging the other user to participate in the virtual space.

According to the above program, it is possible to easily make other users who share the virtual space recognize that, for example, multiplayer is being consulted, and it is possible to promote the use of multiplayer.

(17) A step of transitioning the virtual object from the first state to the second state when a reaction to the second action is performed.
If the virtual object is controlled by the other user within a certain period of time after the transition to the second state, the step of transitioning the virtual object to the third state and the step.
The program according to item (16) for further causing the computer to execute the program.

According to the above program, the user who has executed the first action can easily grasp that there has been a reaction from another user and that the virtual object has been controlled by the other user.

(18) If the virtual object is not controlled by the other user within a certain period of time after the transition to the second state, the step of returning the virtual object to the first state is further described. The program according to item (17) for causing a computer to execute.

According to the above program, the user who executed the first action can easily grasp that the virtual object is not controlled by another user even though the second action is executed.

(19) Any of items (16) to (18), wherein the content of the notification differs depending on at least one of the intensity of the operation of the first action and the type of the first action with respect to the virtual object. The program described in.

According to the above method, other users can grasp the degree of request for multiplayer from the user who executed the first action, and decide whether to participate in the virtual space according to the degree of request. can do.

(20) A step of executing the first action by the operation object with respect to another virtual object whose operation in the virtual space can be controlled by a computer.
A step of transitioning the game provided in the virtual space from the first scene to the second scene based on the execution of the first action.
The program according to any one of items (11) to (19) for further causing the computer to execute the program.

According to the above program, the game can be advanced without impairing the user's immersive feeling in the virtual space.

(21) An information processing device for providing a virtual experience to a user.
By controlling the processor included in the information processing device
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
A step of executing the second action for the other user based on the execution of the first action, and
Is executed, an information processing device.

According to the information processing device, the virtual experience of the user can be improved.

(22) An information processing system for providing a virtual experience to a user.
By controlling a plurality of processors included in the information processing system
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
A step of executing the second action for the other user based on the execution of the first action, and
Is executed, an information processing system.

According to the above information processing system, the virtual experience of the user can be improved.

(23) An information processing method executed by a computer to provide a virtual experience to a user.
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
A step of executing the second action for the other user based on the execution of the first action, and
Information processing methods, including.

According to the above information processing method, the virtual experience of the user can be improved.

(24) On a computer to provide a virtual experience to the user,
The step to execute the first action in the virtual space,
Based on the execution of the first action, a step of executing a second action for a user different from the user, and a step of executing the second action.
A step of executing the third action in the virtual space based on the reaction to the second action, and
A program to execute.

According to the above program, the virtual experience of the user can be improved. In particular, by executing the third action in the virtual space based on the reaction to the second action executed for other users, a new virtual experience is provided in which the boundary between the real space and the virtual space becomes seamless. can do.

(25) The first action is an action for identifying the other user, and the third action is an action executed by a virtual object associated with the other user in the virtual space. The program described in item (24).

According to the above program, communication in the virtual space can be smoothly performed by executing an action between a plurality of users who share the virtual space.

(26) The computer is further made to perform a step of moving an operation object in the virtual space according to the movement of a part of the user's body.
The program according to item (24) or (25), wherein the first action is executed using the operation object.

According to the above program, actions can be executed by intuitive operations using operation objects among a plurality of users sharing a virtual space, and the user's immersive feeling is not impaired.

(27) The program according to any one of items (24) to (26), wherein the second action includes a notification for encouraging the other user to participate in the virtual space.

According to the above program, it is possible to easily make other users who share the virtual space recognize that, for example, multiplayer is being consulted, and it is possible to promote the use of multiplayer.

(28) From the item (24), the third action includes transitioning an action executed by a virtual object associated with the other user in the virtual space from the first state to the second state. The program according to any one of (27).

According to the above method, the user who has executed the first action can easily grasp whether or not there is a reaction by another user.

(29) An information processing device for providing a virtual experience to a user.
By controlling the processor included in the information processing device
The step to execute the first action in the virtual space,
Based on the execution of the first action, a step of executing a second action for a user different from the user, and a step of executing the second action.
A step of executing the third action in the virtual space based on the reaction to the second action, and
Is executed, an information processing device.

According to the information processing device, the virtual experience of the user can be improved.

(30) An information processing system for providing a virtual experience to a user.
By controlling a plurality of processors included in the information processing system
The step to execute the first action in the virtual space,
Based on the execution of the first action, a step of executing a second action for a user different from the user, and a step of executing the second action.
A step of executing the third action in the virtual space based on the reaction to the second action, and
Is executed, an information processing system.

According to the above information processing system, the virtual experience of the user can be improved.

(31) An information processing method executed by a computer to provide a virtual experience to a user.
The step to execute the first action in the virtual space,
Based on the execution of the first action, a step of executing a second action for a user different from the user, and a step of executing the second action.
A step of executing the third action in the virtual space based on the reaction to the second action, and
Information processing methods, including.

According to the above information processing method, the virtual experience of the user can be improved.

[Details of Embodiments Shown in the Present Disclosure]
Hereinafter, embodiments shown in the present disclosure will be described with reference to the drawings. For convenience of explanation, the description of the element having the same reference number as the element already described in the description of the present embodiment will not be repeated.

First, the outline of the configuration of the virtual space distribution system 100 will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter, simply referred to as a distribution system 100). As shown in FIG. 1, the distribution system 100 includes a user terminal 1A (first user terminal) operated by a user A (first user) equipped with a head-mounted device (HMD) 110 (first HMD), and an HMD 110 (first user terminal). It includes a user terminal 1B (second user terminal) operated by a user B (second user) equipped with a second HMD), and a server 2. The user terminals 1A and 1B are communicably connected to the server 2 via a communication network 3 such as the Internet. In the present embodiment, the virtual space includes a VR (Virtual Reality) space. In the following, for convenience of explanation, the user terminals 1A and 1B may be collectively referred to as the user terminal 1. Further, each user A and B may be collectively referred to as a user U. Further, in the present embodiment, it is assumed that the user terminals 1A and 1B have the same configuration.

Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic view showing a user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head-mounted device (HMD) 110 mounted on the head of the user U, a headphone 116, a face camera 117, a microphone 118, a position sensor 130, and an external controller. It includes 320 and a control device 120.

The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, a face camera 113, and a mounting sensor 115. The display unit 112 includes a non-transparent display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. As a result, the user U can immerse himself in the virtual space by seeing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the left eye of the user U, and a right-eye display unit configured to provide an image to the right eye of the user U. May be good. Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance.

The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (angular velocity sensor, gyro sensor, etc.), and detects various movements (tilts, etc.) of the HMD 110 mounted on the head of the user U. be able to.

The gaze sensor 140 has an eye tracking function for detecting the line of sight of the user U. The gaze sensor 140 may include, for example, a gaze sensor for the right eye and a gaze sensor for the left eye. The gaze sensor for the right eye acquires information on the angle of rotation of the eyeball of the right eye by irradiating the user U's right eye with, for example, infrared light and detecting the reflected light reflected from the right eye (particularly the cornea or iris). You may. On the other hand, the gaze sensor for the left eye irradiates the left eye of the user U with infrared light, for example, and detects the reflected light reflected from the left eye (particularly the cornea or the iris) to obtain information on the angle of rotation of the eyeball of the left eye. May be obtained.

The face camera 113 is configured to acquire an image (particularly a moving image) in which the eyes (left eye and right eye) and eyebrows (left eyebrows and right eyebrows) of the user U are displayed while the HMD 110 is attached to the user U. Has been done. The face camera 113 is arranged at a predetermined position inside the HMD 110 so that the eyes and eyebrows of the user U can be imaged. The frame rate of the moving image acquired by the face camera 113 may be larger than the frame rate of the field image (described later) displayed on the HMD 110.

The mounting sensor 115 is configured to generate event information to indicate whether the user U is wearing the HMD 110. Specifically, when the user U removes the HMD 110 (when the state of the HMD 110 changes from the mounted state to the non-mounted state), the mounting sensor 115 generates event information and then controls the event information. It is transmitted to the device 120. Further, when the user U wears the HMD 110 (when the state of the HMD 110 changes from the non-wearing state to the wearing state), the mounting sensor 115 generates event information and then transfers the event information to the control device 120. Send. Further, the configuration of the mounting sensor 115 is not particularly limited. For example, it is assumed that the elastic force of the spring provided in the HMD 110 changes when the user U wears the HMD 110. In this case, the mounting sensor 115 may generate event information based on the change in the elastic force of the spring. Further, it is assumed that when the user U wears the HMD 110, the current flowing between the two pads in contact with a part of the body (nose portion) provided on the HMD 110 changes. In this case, the mounting sensor 115 may generate event information based on the change in the current flowing between the two pads.

The headphone 116 (audio output unit) is attached to the left ear and the right ear of the user U, respectively. The headphone 116 is configured to receive voice data (electrical signal) from the control device 120 and output voice based on the received voice data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Further, the microphone 118 is configured to transmit voice data to the control device 120.

The face camera 117 is configured to acquire an image (particularly, a moving image) in which the mouth of the user U and its surroundings are displayed. The face camera 117 may be arranged at a position facing the mouth of the user U so that the mouth of the user U and its surroundings can be imaged. Further, the face camera 117 may be connected to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be larger than the frame rate of the field image displayed on the HMD 110.

The position sensor 130 is configured, for example, by a position tracking camera and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is wirelessly or wiredly connected to the control device 120 and is configured to detect information regarding the position, inclination, or emission intensity of a plurality of detection points (not shown) provided on the HMD 110. .. Further, the position sensor 130 is configured to detect information regarding the positions, tilts and / or emission intensities of a plurality of detection points (not shown) provided on the external controller 320. The detection point is, for example, a light emitting unit that emits infrared rays or visible light. Further, the position sensor 130 may include an infrared sensor and a plurality of optical cameras.

The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, and the hand of the user U in this embodiment), so that the hand of the avatar displayed in the virtual space is displayed. Used to control behavior. The external controller 320 is referred to as a right-handed external controller 320R operated by the right hand of the user U (hereinafter, simply referred to as a controller 320R) and a left-handed external controller 320L operated by the left hand of the user U (hereinafter, simply referred to as a controller 320L). ) And. The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the fingers of the right hand. Further, the right hand of the avatar existing in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the fingers of the left hand. Further, the left hand of the avatar existing in the virtual space moves according to the movement of the controller 320L.

The control device 120 is a computer configured to control the HMD 110. The control device 120 identifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and based on the specified position information, the position of the virtual camera (the position of the avatar) in the virtual space and the reality. The position of the user U wearing the HMD 110 in space can be accurately associated. Further, the control device 120 identifies the operation of the external controller 320 based on the information acquired from the sensor built in the position sensor 130 and / or the external controller 320, and is based on the operation of the specified external controller 320. Therefore, the movement of the avatar's hand displayed in the virtual space can be accurately associated with the movement of the external controller 320 in the real space. In particular, the control device 120 identifies the operation of the controller 320L based on the information acquired from the sensor built in the position sensor 130 and / or the controller 320L, and virtualizes based on the operation of the specified controller 320L. The movement of the left hand of the avatar displayed in the space and the movement of the controller 320L in the real space (the movement of the left hand of the user U) can be accurately associated with each other. Similarly, the control device 120 identifies the operation of the controller 320R based on the information acquired from the position sensor and / the sensor built in the controller 320R, and the virtual space is based on the operation of the specified controller 320R. The movement of the right hand of the avatar displayed inside and the movement of the controller 320R in the real space (the movement of the right hand of the user U) can be accurately associated with each other.

Further, the control unit 121 can associate the facial expression (state) of the user U with the facial expression (state) of the avatar of the user U based on the moving images acquired by the face cameras 113 and 117. The details of the process of associating the facial expression of the user U with the facial expression of the avatar will be described later.

Further, the control device 120 identifies the line of sight of the user U's right eye and the line of sight of the left eye based on the information transmitted from the gaze sensor 140 (the gaze sensor for the left eye and the gaze sensor for the right eye), and sets the line of sight of the right eye. The gaze point, which is the intersection of the line of sight of the left eye, can be specified. Further, the control device 120 can specify the line of sight of both eyes of the user U (the line of sight of the user U) based on the specified gaze point. Here, the line of sight of the user U is the line of sight of both eyes of the user U, and coincides with the direction of the straight line passing through the midpoint of the line segment connecting the right eye and the left eye of the user U and the gazing point.

Next, a method of acquiring information regarding the position and inclination of the HMD 110 will be described with reference to FIG. FIG. 3 is a diagram showing the head of the user U wearing the HMD 110. Information regarding the position and tilt of the HMD 110 linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands upright is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the v-axis and the direction orthogonal to the w-axis are defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 has an angle around each uvw axis (that is, a yaw angle indicating rotation about the v-axis, a pitch angle indicating rotation about the u-axis, and a w-axis as the center. The tilt determined by the roll angle indicating rotation) is detected. The control device 120 determines the angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram showing a hardware configuration of the control device 120. As shown in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are communicably connected to each other via the bus 126.

The control device 120 may be configured as a personal computer, a smartphone, a phablet, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. Further, a part of the functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

The control unit 121 includes a memory and a processor. The memory is composed of, for example, a ROM (Read Only Memory) in which various programs and the like are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs and the like executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit) and / or a GPU (Graphics Processing Unit), and expands a program specified from various programs embedded in the ROM onto the RAM and develops the RAM. It is configured to execute various processes in collaboration with.

In particular, the control unit 121 may control various operations of the control device 120 by the processor deploying the control program on the RAM and executing the control program in cooperation with the RAM. The control unit 121 displays the field of view image on the display unit 112 of the HMD 110 based on the field of view image data. As a result, the user U can immerse himself in the virtual space.

The storage unit (storage) 123 is, for example, a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, and a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 123 may store data related to the user U authentication program and various images and objects (for example, an avatar). Further, a database including a table for managing various data may be constructed in the storage unit 123.

The I / O interface 124 is configured to connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118 to the control device 120 so as to be communicable. For example, it is composed of a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control device 120 may be wirelessly connected to the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118, respectively.

The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as a server 2 via the communication network 3 and various processing circuits for wireless connection, and is for communicating via the communication network 3. It is configured to meet the communication standards of.

Next, a process for displaying the field image on the HMD 110 will be described with reference to FIGS. 5 to 8. FIG. 5 is a flowchart showing a process of displaying a field image on the HMD 110. FIG. 6 is an xyz space diagram showing an example of the virtual space 200. The state (a) of FIG. 7 is a yx plan view of the virtual space 200 shown in FIG. The state (b) of FIG. 7 is a zx plan view of the virtual space 200 shown in FIG. FIG. 8 is a diagram showing an example of the field image V displayed on the HMD 110.

As shown in FIG. 5, in step S1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 6, the virtual space 200 is defined as a spherical image centered on the central position 210 (in FIG. 6, only the upper half of the celestial sphere is shown). Further, in the virtual space 200, an xyz coordinate system with the center position 210 as the origin is set. The virtual camera 300 defines a visual axis L for specifying the visual field image V (see FIG. 8) displayed on the HMD 110. The uvw coordinate system that defines the field of view of the virtual camera 300 is determined to be linked to the uvw coordinate system defined around the head of the user U in the real space. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

Next, in step S2, the control unit 121 specifies the field of view CV (see FIG. 7) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on the data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 specifies the position and orientation of the virtual camera 300 in the virtual space 200 based on the information regarding the position and inclination of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300, and identifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a part of the virtual space 200 that can be visually recognized by the user U wearing the HMD 110 (in other words, a part of the virtual space 200 displayed on the HMD 110). Equivalent to). Further, the visual field CV is set to the first region CVa set as the angle range of the polar angle α about the visual axis L in the xy plane shown in the state (a) of FIG. 7 and the state (b) of FIG. In the xz plane shown, it has a second region CVb set as an angle range of the azimuth angle β about the visual axis L. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the information regarding the specified line of sight of the user U and the position and inclination of the HMD 110. , The orientation of the virtual camera 300 (the visual axis L of the virtual camera) may be determined. Further, as will be described later, the control unit 121 may determine the orientation of the face of the user U's avatar based on the information regarding the position and inclination of the HMD 110.

In this way, the control unit 121 can specify the field of view CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U equipped with the HMD 110 moves, the control unit 121 updates the field of view CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. be able to. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the field of view CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change in the line of sight of the user U.

Next, in step S3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on the virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

Next, in step S4, the control unit 121 displays the visual field image V on the display unit 112 of the HMD 110 based on the visual field image data (see FIG. 7). In this way, the field of view CV of the virtual camera 300 changes according to the movement of the user U wearing the HMD 110, and the field of view image V displayed on the display unit 112 of the HMD 110 changes, so that the user U is in the virtual space. You can immerse yourself in 200.

The virtual camera 300 may include a virtual camera for the left eye and a virtual camera for the right eye. In this case, the control unit 121 generates left-eye field-of-view image data showing the field-of-view image for the left eye based on the virtual space data and the field of view of the left-eye virtual camera. Further, the control unit 121 generates right-eye field image data showing a field image for the right eye based on the virtual space data and the field of view of the right-eye virtual camera. After that, the control unit 121 displays the visual field image for the left eye on the left eye display unit based on the left eye visual field image data, and displays the visual field image for the right eye on the right eye display unit based on the right eye visual field image data. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally by the parallax between the visual field image for the left eye and the visual field image for the right eye. The virtual camera may be placed at the eye position of the avatar operated by the user, as will be described later. For example, the left-eye virtual camera may be placed in the left eye of the avatar, while the right-eye virtual camera may be placed in the right eye of the avatar.

Further, the processes of steps S1 to S4 shown in FIG. 5 may be executed for each frame (still image constituting the moving image). For example, when the frame rate of the moving image is 90 fps, the processes of steps S1 to S4 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). In this way, since the processes of steps S1 to S4 are repeatedly executed at predetermined intervals, the field of view of the virtual camera 300 is updated according to the operation of the HMD 110, and the field of view image V displayed on the display unit 112 of the HMD 110 is updated. Is updated.

Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 9 is a diagram showing a hardware configuration of the server 2. As shown in FIG. 9, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are connected to each other so as to be able to communicate with each other via the bus 24. The control unit 23 includes a memory and a processor, and the memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

The storage unit (storage) 22 is, for example, a large-capacity HDD or the like. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 22 may store user management information for managing each user and data related to various images and objects (for example, an avatar). The communication interface 21 is configured to connect the server 2 to the communication network 3.

Next, an example of the process of synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to FIGS. 1, 10 and 11. The state (a) of FIG. 10 is a diagram showing a virtual space 200A provided to the user A. The state (b) of FIG. 10 is a diagram showing a virtual space 200B provided to the user B. FIG. 11 is a sequence diagram for explaining an example of a process of synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as a prerequisite, as shown in FIG. 10, the avatar 4A (first avatar) associated with the user terminal 1A (user A) and the avatar 4B (first avatar) associated with the user terminal 1B (user B). It is assumed that 2 avatars) share the same virtual space. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

As shown in the state (a) of FIG. 10, the virtual space 200A of the user A includes the avatar 4A and the avatar 4B. The avatar 4A is operated by the user A and is linked to the operation of the user A. The avatar 4A has a left hand (a part of the virtual body of the avatar 4A) linked to the operation of the controller 320L of the user terminal 1A (the operation of the left hand of the user A) and the operation of the controller 320R of the user terminal 1A (the right hand of the user A). It has a right hand linked to the operation) and a face whose facial expression is linked to the facial expression of the user A. The face of Avatar 4A has a plurality of face parts (eg, eyes, eyebrows, mouth, etc.). The avatar 4B is operated by the user B and is linked to the operation of the user B. The avatar 4B has a left hand linked to the operation of the controller 320L of the user terminal 1B indicating the operation of the left hand of the user B and a right hand linked to the operation of the controller 320R of the user terminal 1B indicating the operation of the right hand of the user B. It has a face linked to the facial expression of the user B. The face of Avatar 4B has a plurality of face parts (for example, eyes, eyebrows, mouth, etc.).

In the virtual space 200A provided to the user A, it is assumed that the avatar 4A is not visualized. In this case, the avatar 4A arranged in the virtual space 200A includes at least a virtual camera 300 linked to the movement of the HMD 110 of the user terminal 1A.

Further, the positions of the avatars 4A and 4B may be specified according to the positions of the HMD 110s of the user terminals 1A and 1B. Similarly, the orientation of the face of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Further, the hand movements of the avatars 4A and 4B may be specified according to the movements of the external controllers 320 of the user terminals 1A and 1B. In particular, the operation of the left hand of the avatars 4A and 4B is specified according to the operation of the controllers 320L of the user terminals 1A and 1B, and the operation of the right hand of the avatars 4A and 4B according to the operation of the controllers 320R of the user terminals 1A and 1B. May be specified. Further, the facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B.

Further, a virtual camera 300 (FIG. 6) may be arranged in each eye of the avatars 4A and 4B. In particular, a virtual camera for the left eye may be arranged in the left eye of the avatars 4A and 4B, and a virtual camera for the right eye may be arranged in the right eye of the avatars 4A and 4B. In the following description, it is assumed that the virtual camera 300 is placed in the eyes of the avatars 4A and 4B. Therefore, the field of view CV of the avatar 4A is assumed to match the field of view CV of the virtual camera 300 arranged in the avatar 4A (see FIG. 7). Similarly, the field of view CV of the avatar 4B is assumed to match the field of view CV (see FIG. 7) of the virtual camera 300 arranged in the avatar 4B.

As shown in the state (b) of FIG. 10, the virtual space 200B of the user B includes the avatar 4A and the avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

In the virtual space 200B provided to the user B, it is assumed that the avatar 4B is not visualized. In this case, the avatar 4B arranged in the virtual space 200B includes at least a virtual camera 300 linked to the movement of the HMD 110 of the user terminal 1B.

Next, referring to FIG. 11, in step S10, the control unit 121 of the user terminal 1A generates the voice data of the user A. For example, when the user A inputs voice to the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. After that, the microphone 118 transmits the generated voice data to the control unit 121 via the I / O interface 124.

Next, in step S11, the control unit 121 of the user terminal 1A generates the control information of the avatar 4A, and then the control information of the generated avatar 4A and the voice data indicating the voice of the user A (voice data of the user A). ) Is sent to the server 2. After that, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12).

Here, the control information of the avatar 4A is information necessary for controlling the operation of the avatar 4A. The control information of the avatar 4A includes information on the position of the avatar 4A (position information), information on the orientation of the face of the avatar 4A (face orientation information), and information on the state of the hands (left hand and right hand) of the avatar 4A (hand information). ) And information (face information) regarding the facial expression of the avatar 4A may be included.

The face information of the avatar 4A has information indicating the state of a plurality of face parts. Information indicating the state of a plurality of face parts includes information indicating the state of the eyes of the avatar 4A (shape of the white eyes and the position of the black eyes with respect to the white eyes) (eye information) and information on the eyebrows of the avatar 4A (position and shape of the eyebrows). Etc.) and information (eyebrow information) indicating the state of the mouth of the avatar 4A (position and shape of the mouth, etc.).

Specifically, the eye information of the avatar 4A has information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyebrow information of the avatar 4A has information indicating the state of the left eyebrow of the avatar 4A and information indicating the state of the right eyebrow of the avatar 4A.

The control unit 121 of the user terminal 1A acquires an image showing the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user A from the face camera 113 mounted on the HMD 110, and then obtains the acquired image. And, based on a predetermined image processing algorithm, the states of the eyes (left eye and right eye) and eyebrows (left eyebrows and right eyebrows) of the user A are specified. Next, the control unit 121 uses information indicating the state of the eyes (left eye and right eye) of the avatar 4A and the eyebrows (left eyebrow and right eyebrow) of the avatar 4A based on the specified eye and eyebrow states of the user A. Generate information indicating the status respectively.

Similarly, the control unit 121 of the user terminal 1A acquires an image showing the mouth of the user A and its surroundings from the face camera 117, and then based on the acquired image and a predetermined image processing algorithm, the mouth of the user A. Identify the state of. Next, the control unit 121 generates information indicating the mouth state of the avatar 4A based on the mouth state of the specified user A.

As described above, the control unit 121 of the user terminal 1A has the eye information of the avatar 4A corresponding to the eyes of the user A, the eyebrow information of the avatar 4A corresponding to the eyebrows of the user A, and the avatar 4A corresponding to the mouth of the user A. It becomes possible to generate the mouth information of.

Further, the control unit 121 has an image captured by the face camera 113, an image captured by the face camera 117, and a plurality of types of facial expressions stored in the storage unit 123 or the memory based on a predetermined image processing algorithm. The facial expression (for example, smile) of the user A may be specified from the facial expressions (for example, smile, sad expression, expressionless expression, angry expression, surprised expression, troubled expression, etc.). After that, the control unit 121 can generate facial information indicating the facial expression (for example, a smile) of the avatar 4A based on the specified facial expression of the user A. In this case, facial expression data including facial expressions of a plurality of types of the avatar 4A and facial information of the plurality of avatars 4A each associated with one of the facial expressions of the plurality of types is stored in the storage unit 123. You may.

For example, when the facial expression of the avatar 4A is specified as a smile, the control unit 121 acquires face information indicating the smile of the avatar 4A based on the facial expression (smile) of the avatar 4A and the facial expression data. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information, and mouth information of the avatar 4A when the facial expression of the avatar 4A is a smile.

Next, in step S13, the control unit 121 of the user terminal 1B generates the control information of the avatar 4B, and then transmits the control information of the generated avatar 4B to the server 2. After that, the control unit 23 of the server 2 receives the control information of the avatar 4B from the user terminal 1B (step S14).

Here, the control information of the avatar 4B is information necessary for controlling the operation of the avatar 4B. The control information of the avatar 4B includes information on the position of the avatar 4B (position information), information on the orientation of the face of the avatar 4B (face orientation information), and information on the state of the hands (left hand and right hand) of the avatar 4B (hand information). ) And information (face information) regarding the facial expression of the avatar 4B may be included.

The face information of the avatar 4B has information indicating the state of a plurality of face parts. The information indicating the state of a plurality of face parts includes information indicating the state of the eyes of the avatar 4B (shape of the white eyes and the position of the black eyes with respect to the white eyes) (eye information) and information on the eyebrows of the avatar 4B (position and shape of the eyebrows). Etc.) and information (eyebrow information) indicating the state of the mouth of the avatar 4B (position and shape of the mouth, etc.). The method of acquiring the face information of the avatar 4B is the same as the method of acquiring the face information of the avatar 4A.

Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S15), while transmitting the control information of the avatar 4A and the voice data of the user A to the user terminal 1B (step S19). After that, the control unit 121 of the user terminal 1A receives the control information of the avatar 4B in step S16, updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B, and then the virtual space 200A. The virtual space data indicating (see the state (a) in FIG. 10) is updated (step S17).

Specifically, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

After that, the control unit 121 of the user terminal 1A identifies the field of view CV of the avatar 4A (virtual camera 300) according to the position and tilt of the HMD 110, and then sets the updated virtual space data and the field of view CV of the avatar 4A. Based on this, the visual field image displayed on the HMD 110 is updated (step S18).

On the other hand, the control unit 121 of the user terminal 1B updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B after receiving the control information of the avatar 4A and the voice data of the user A in step S20. Above, the virtual space data indicating the virtual space 200B (see the state (b) in FIG. 10) is updated (step S21).

Specifically, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

After that, the control unit 121 of the user terminal 1B identifies the field of view CV of the avatar 4B (virtual camera 300) according to the position and tilt of the HMD 110, and then sets the updated virtual space data and the field of view CV of the avatar 4B. Based on this, the visual field image displayed on the HMD 110 is updated (step S22).

After that, the control unit 121 of the user terminal 1B uses the received voice data of the user A, the information regarding the position of the avatar 4A included in the control information of the avatar 4A, the information regarding the position of the avatar 4B, and a predetermined voice processing algorithm. Based on this, the voice data of user A is processed. After that, the control unit 121 transmits the processed voice data to the headphone 116 (voice output unit), and the headphone 116 outputs the voice of the user A based on the processed voice data (step S23). In this way, voice chat (VR chat) between users (between avatars) can be realized in the virtual space.

In the present embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, while the avatar 4B. Control information is transmitted to the user terminal 1A. In this way, it is possible to synchronize the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B.

Next, by mainly referring to FIGS. 12 and 14, the information processing method according to the present embodiment when the HMD 110 of the user A is removed will be described. FIG. 12 is an example of a flowchart for explaining an information processing method according to the present embodiment when the user A removes the HMD 110. FIG. 14 is a diagram showing an example of a virtual space 200B provided to the user B for explaining the information processing method according to the present embodiment. In the flowchart shown in FIG. 12, the control unit 121 of the user terminal 1A transmits the control information of the avatar 4A to the server 2, and the control unit 121 of the user terminal 1B transmits the control information of the avatar 4B to the server 2. Is omitted for convenience of explanation.

In this embodiment, as a precondition, it is assumed that the avatar 4A and the avatar 4B share the same virtual space as shown in FIG. That is, it is assumed that the users A and B share one virtual space via the communication network 3.

The virtual space 200B of the user B includes the avatar 4A and the avatar 4B. The control unit 121 of the user terminal 1B updates the virtual space data indicating the virtual space 200B. In this description, for convenience of explanation, the illustration of the virtual space 200A provided to the avatar 4A is omitted.

Referring to FIG. 12, in step S30, the control unit 121 of the user terminal 1A determines whether or not the user A has removed the HMD 110. In this respect, the control unit 121 may determine whether or not the user A has removed the HMD 110 based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the mounting sensor 115. The HMD 114 and the mounting sensor 115 function as sensors configured to detect whether or not the HMD 110 is mounted.

With reference to the state (a) of FIG. 13, a process of determining whether or not the user A has removed the HMD 110 by using the HMD sensor 114 will be described. The state (a) in FIG. 13 is an example of a flowchart for explaining a process for determining whether or not the user A has removed the HMD 110 by using the HMD sensor 114. As shown in the state (a) of FIG. 13, the control unit 121 receives tilt information indicating the tilt (roll angle, yaw angle, pitch angle) of the HMD 110 from the HMD sensor 114 (step S301). Next, the control unit 121 determines whether or not the inclination of the HMD 110 is larger than the predetermined inclination based on the received inclination information (S302). Here, the predetermined inclination may be appropriately set by the user. While the user A is wearing the HMD 110, it is assumed that the inclination of the HMD 110 does not become larger than the predetermined inclination. That is, while the user A is wearing the HMD 110, the tilt of the HMD 110 is equal to or less than the predetermined tilt, while when the user A removes the HMD 110, the HMD 110 is greatly tilted, so that the tilt of the HMD 110 has a predetermined tilt. May exceed. From this point of view, a predetermined inclination is set.

For example, when the control unit 121 determines that the pitch angle θ of the HMD 110 is larger than the predetermined angle θth (YES in step S302) as an example of the inclination of the HMD 110, the user A removes the HMD 110 (in other words,). , User A is not wearing the HMD 110), and then the process of step S31 is executed. On the other hand, when the control unit 121 determines that the pitch angle θ of the HMD 110 is equal to or less than the predetermined angle θth (NO in step S302), the control unit 121 identifies that the user A is wearing the HMD 110, and then the HMD sensor. It waits until the tilt information of the HMD 110 from 114 is received.

Next, with reference to the state (b) of FIG. 13, a process of determining whether or not the user A has removed the HMD 110 by using the mounting sensor 115 will be described. The state (b) in FIG. 13 is an example of a flowchart for explaining a process for determining whether or not the user A has removed the HMD 110 by using the mounting sensor 115. As shown in the state (b) of FIG. 13, the control unit 121 determines whether or not the event information has been received from the mounting sensor 115 (step S303). Here, the event information transmitted from the wearing sensor 115 is information indicating that the state of the HMD 110 of the user terminal 1A has changed between the wearing state and the non-wearing state. For example, when the user A removes the HMD 110 (when the state of the HMD 110 changes from the mounted state to the non-mounted state), the mounting sensor 115 outputs event information. Further, when the user A wears the HMD 110 (when the state of the HMD 110 changes from the non-wearing state to the wearing state), the wearing sensor 115 outputs the event information.

When the control unit 121 determines that the event information has been received from the mounting sensor 115 (YES in step S303), the control unit 121 determines that the user A has removed the HMD 110 (in other words, the user A has not mounted the HMD 110). After specifying, the process of step S31 is executed. On the other hand, when the control unit 121 determines that the event information has not been received from the mounting sensor 115 (NO in step S303), the user A identifies that the HMD 110 is mounted, and then mounts the event information. It waits until it receives from the sensor 115.

Next, returning to FIG. 12, when the control unit 121 determines that the user A has removed the HMD 110 (in other words, the user A does not wear the HMD 110) (step S30YES), the user A determines that the HMD 110 is not attached (step S30YES). Generates information (non-wearing information) indicating that the device is not mounted (step S31). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S32).

Next, the server 2 receives the non-wearing information from the user terminal 1A, and then transmits the non-wearing information to the user terminal 1B via the communication network 3 (step S33). After that, the control unit 121 of the user terminal 1B receives the non-wearing information from the server 2, and then sets the facial expression of the avatar 4A arranged in the virtual space 200B as the default facial expression (in other words, the default setting). (Facial expression) is set (step S34). Here, the default facial expression of the avatar 4A means the facial expression of the avatar 4A in the initial state before the update of the facial expression of the avatar 4A is started. When the default facial expression is a smile, the facial expression of Avatar 4A is set to smile, as shown in FIG. Further, the control unit 121 updates the operation of the avatar 4B (including the facial expression of the avatar 4B) based on the control information of the avatar 4B.

In step S34, the control unit 121 may set the facial expression of the avatar 4A to the facial expression selected in advance by the user A as an embodiment indicating that the user A is not wearing the HMD 110. In this case, information regarding the facial expression of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B and then stored in advance in the storage unit 123 of the user terminal 1B.

Next, as shown in FIG. 14, the control unit 121 creates a balloon object 42A associated with the avatar 4A in the virtual space 200B. Information indicating that the user A is not wearing the HMD 110 (for example, "User A is absent") is displayed on the balloon object 42A.

The control unit 121 may generate the telop object 43A in the virtual space 200B instead of the balloon object 42A (see FIG. 15). As shown in FIG. 15, the telop object 43A displays information indicating that the user A is not wearing the HMD 110 (for example, "user A is absent"). The telop object 43A may be placed near the avatar 4A, or may be interlocked with the movement of the field of view CV of the avatar 4B so that it is always placed within the field of view CV of the avatar 4B. Further, a telop indicating information indicating that the user A is not wearing the HMD 110 may be superimposed on the visual field image displayed on the HMD 110.

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S36).

On the other hand, the control unit 121 of the user terminal 1A shifts from the active mode to the sleep mode after transmitting the non-wearing information to the server 2 via the communication network 3 (step S37). In this case, it is preferable that the mode of the control unit 121 is the sleep mode, while the mode of the sensor (mounted sensor 115 or the HMD sensor 114) is the active mode.

According to the present embodiment, when the user A is not wearing the HMD 110, the user terminal 1B presents to the user B that the user A is not wearing the HMD 110. In particular, according to the present embodiment, when the user A is not wearing the HMD 110, information indicating that the user A is not wearing the HMD 110 is visualized on the field image displayed on the HMD 110 of the user terminal 1B. To. In this respect, the avatar 4A and the balloon object 42A whose facial expressions are set to the default facial expressions are visualized on the visual field image displayed on the HMD 110 of the user terminal 1B.

In this way, the user B visually recognizes the default facial expression (for example, a smile) of the avatar 4A and the information displayed on the balloon object 42A while communicating with the user A on the virtual space 200B. It can be easily grasped that the user A does not wear the HMD 110. Therefore, it is possible to provide the user with a rich virtual experience.

In particular, the user B may feel uncomfortable with the user A who does not show a reaction unless he / she knows that the user A is not wearing the HMD 110. In this way, by letting the user B know that the user A is not wearing the HMD 100, it is possible to avoid a situation in which the user B feels uncomfortable (for example, the uncanny valley phenomenon) with respect to the user A who does not show a reaction. It becomes possible to do.

Further, according to the present embodiment, the user A does not wear the HMD 110 according to the information (tilt information or event information) transmitted from the sensor (HMD sensor 114 or the mounting sensor 115) of the user terminal 1A. Be identified. After that, non-wearing information indicating that the user A is not wearing the HMD 110 is transmitted to the server 2. In this way, it is possible to automatically identify that the user A is not wearing the HMD 110 according to the information output from the sensor of the user terminal 1A.

In the present embodiment, the user terminal 1A transmits the non-wearing information to the user terminal 1B via the server 2, but instead of the non-wearing information, the user terminal 1A sends the tilt information transmitted from the HMD sensor 114 to the user terminal. It may be transmitted to 1B. In this case, the control unit 121 of the user terminal 1B may execute the processes specified in steps S34 to S36 after specifying that the user A is not wearing the HMD 110 based on the received tilt information. good.

Next, by mainly referring to FIGS. 16 and 17, the information processing method according to the present embodiment when the user A who has removed the HMD 110 reattaches the HMD 110 will be described. FIG. 16 is an example of a flowchart for explaining an information processing method according to the present embodiment when the user A reattaches the HMD 110. FIG. 17 is a diagram showing an example of a virtual space 200B provided to the user B for explaining the information processing method according to the present embodiment.

Referring to FIG. 16, in step S40, the control unit 121 of the user terminal 1A determines whether or not the user A wears the HMD 110 again. In this respect, the control unit 121 may determine whether or not the user A has remounted the HMD 110 based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the mounting sensor 115. .. For example, when it is determined that the event information has been received from the mounting sensor 115, the control unit 121 identifies that the user A has mounted the HMD 110, and then executes the process of step S41. On the other hand, when the control unit 121 determines that the event information is not received from the mounting sensor 115, the user A specifies that the HMD 110 is not mounted, and then receives the event information from the mounting sensor 115. stand by.

If the determination result in step S40 is YES, the control unit 121 shifts from the sleep mode to the active mode (step S41). Next, the control unit 121 that has transitioned to the active mode generates control information for the avatar 4A and also generates wearing information indicating that the user A has worn the HMD 110 (step S42). After that, the control unit 121 transmits the control information and the wearing information of the avatar 4A to the server 2 via the communication network 3 (step S43).

Next, the server 2 receives the control information and the wearing information of the avatar 4A from the user terminal 1A, and then transmits the control information and the wearing information of the avatar 4A to the user terminal 1B via the communication network 3 (step S44). After that, the control unit 121 of the user terminal 1B updates the operation of the avatar 4A based on the control information of the avatar 4A, and updates the operation of the avatar 4B based on the control information of the avatar 4B (step S45). In particular, the control unit 121 updates the facial expression of the avatar 4A based on the face information of the avatar 4A included in the control information of the avatar 4A, and is based on the face information of the avatar 4B included in the control information of the avatar 4B. Then, the facial expression of Avatar 4B is updated.

Next, as shown in FIG. 17, the control unit 121 creates a balloon object 44A associated with the avatar 4A in the virtual space 200B (step S46). Information indicating that the user A has attached the HMD 110 (for example, "User A has returned to his / her seat") is displayed on the balloon object 44A.

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S47).

According to the present embodiment, when the user A removes the HMD 110 and then reattaches it, the user terminal 1B indicates to the user B that the user A is wearing the HMD 110. In this way, the user B can easily grasp that the user A has reattached the HMD 110 by visually recognizing the updated facial expression of the avatar 4A and the balloon object 44A. Therefore, it is possible to provide the user with a rich virtual experience.

In the description of the present embodiment, information indicating that the user A has or has not worn the HMD 110 is visualized on the visual field image, but the present embodiment is not limited to this. For example, a voice guidance indicating that the user A does not wear the HMD 110 (for example, "User A has left") or a voice guidance indicating that the user A wears the HMD 110 (for example, "User A has left"). "I have returned to my seat." Etc.) may be output from the headphone 116 (audio output unit) of the user terminal 1B. In this case, the control unit 121 of the user terminal 1A transmits voice guidance data to the user terminal 1B together with the non-wearing information (or wearing information) via the server 2.

Further, in the description of the present embodiment, it is assumed that the virtual space data indicating the virtual space 200B is updated on the user terminal 1B side, but the virtual space data may be updated on the server 2 side. Further, although it is assumed that the visual field image data corresponding to the visual field image is updated on the user terminal 1B side, the visual field image data may be updated on the server 2 side. In this case, the user terminal 1B displays the field of view image on the HMD 110 based on the field of view image data transmitted from the server 2.

Further, the order of processing defined in each step shown in FIGS. 12 and 16 is only an example, and the order of these steps can be changed as appropriate.

(Modification 1)
Next, with reference to FIGS. 18 to 25, a modification 1 of the information processing method according to the present embodiment when the HMD 110 of the user A is removed will be described. FIG. 18 is an example of a flowchart for explaining the information processing method according to the modified example 1. 19, 20, 22 and 23 are diagrams showing an example of the virtual space 200B provided to the user B for explaining the information processing method according to the modification 1. FIG. 21 is a diagram showing a display example of a user terminal 5 associated with the user A for explaining the information processing method according to the modification 1. FIG. 24 is an example of a flowchart for explaining an information processing method according to the first modification when the user A does not wear the HMD within a certain period of time after receiving the read information.

Referring to FIG. 18, the control unit 121 of the user terminal 1A determines whether or not the user A has removed the HMD 110 (step S50). Since the process of determining whether or not the user A has removed the HMD 110 by using the HMD sensor 114 is as described in the above embodiment, the detailed description thereof will be omitted.

When it is determined in step S50 that the user A has removed the HMD 110 (in other words, the user A has not attached the HMD 110) (YES in step S50), the control unit 121 has the user A attached the HMD 110. Information (non-attachment information) indicating that the information is not installed is generated (step S51). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S52). In step S51, the control unit 121 may generate positional relationship information indicating the positional relationship between the HMD 110 and the external controller 320, and transmit the positional relationship information to the server 2 together with the non-wearing information.

Next, the server 2 receives the non-wearing information from the user terminal 1A, and then transmits the non-wearing information to the user terminal 1B via the communication network 3 (step S53).

On the other hand, the control unit 121 of the user terminal 1A shifts from the active mode to the sleep mode after transmitting the non-wearing information to the server 2 via the communication network 3 (step S54). In this case, it is preferable that the mode of the control unit 121 is the sleep mode, while the mode of the sensor (mounted sensor 115 or the HMD sensor 114) is the active mode.

Next, the control unit 121 of the user terminal 1B sets the posture of the avatar 4A arranged in the virtual space 200B as the default posture (in other words, the default set posture) based on the non-wearing information received from the server 2. (Step S55). In the first modification, the virtual space will be described assuming that it is a room where a multiplayer partner is searched for, but the present invention is not limited to this. Here, the default posture of the avatar 4A means the posture in the initial state of the avatar 4A. When the default posture is the lying state, the posture of the avatar 4A is set to the lying state as shown in FIG.

In step S55, the control unit 121 may set the posture of the avatar 4A to the posture previously selected by the user A as an embodiment indicating that the user A is not wearing the HMD 110. In this case, the information regarding the posture of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B and then stored in advance in the storage unit 123 of the user terminal 1B.

Further, when the user 1B has received the positional relationship information indicating the positional relationship between the HMD 110 and the external controller 320, in step S55, the control unit 121 changes the posture of the avatar 4A based on the positional relationship information. It may be set. That is, when the user A does not wear the HMD 110, the HMD 110 and the external controller 320 are often placed on the floor or desk in the real space. In this case, the avatar 4A of the virtual space 200B may be set to have a posture of being crushed on the floor based on the positional relationship information between the HMD 110 and the external controller 320.

Next, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A arranged in the virtual space 200B to the default facial expression (in other words, the default facial expression) (step S56). ). When the default facial expression is the state with the eyes closed, the facial expression of the avatar 4A is set to the state with the eyes closed, as shown in FIG.

In step S56, the control unit 121 may set the facial expression of the avatar 4A to the facial expression selected in advance by the user A as an embodiment indicating that the user A is not wearing the HMD 110. In this case, information regarding the facial expression of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B and then stored in advance in the storage unit 123 of the user terminal 1B.

Next, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S57).

In this way, in the virtual space 200B, the posture and face display of the avatar 4A are displayed in the default setting state (the state in which the eyes are closed and lying down), so that the user B wears the HMD 110 and the user A wears the HMD 110. It can be easily recognized that the user does not participate in the content provided in the virtual space 200B.

Next, the control unit 121 of the user terminal 1B updates the field image in step S56, and then, in the virtual space 200B, the avatar 4B associated with the user B performs a specific action (the avatar 4B associated with the user A) with respect to the avatar 4A associated with the user A. It is determined whether or not (an example of the first action) has been executed (step S58).

The specific action performed by Avatar 4B on Avatar 4A includes, for example, an action involving physical interference between Avatar 4A and Avatar 4B. The physical interference between the avatar 4A and the avatar 4B is detected by using a collider (collider object) for detecting a collision with another object. Although not shown, the collider is associated with the bodies of the avatars 4A and 4B, and is used for collision determination (hit determination) between the avatar 4A and the avatar 4B. For example, when the virtual hand 400 of the avatar 4B in which the collider is set comes into contact with the avatar 4A in which the collider is set, it is determined that the avatar 4B and the avatar 4A are in contact with each other. The collider may be set in the entire body range of the avatars 4A, 4B, or may be set in a specific part of the body of the avatars 4A, 4B (for example, hands, feet, head, shoulders, hips, etc.). In particular, in the virtual hand 400, it is preferable that a collider is set for each fingertip and the collision determination is also detected independently for each finger. This makes it possible to accurately determine the collision between another object and the finger. The collider is configured as a transparent body and is not displayed in the visual field image V (FIG. 8).

In the above example, the control unit 121 specifies the collision determination of the avatar 4B with respect to the avatar 4A based on the movement of the avatar 4B (for example, the movement of the virtual hand 400), but the present invention is not limited to this example. For example, the control unit 121 may specify the collision determination for the avatar 4A based on the movement of the user B (for example, the movement of the hand of the user B).

Specific actions including actions involving physical interference include, for example, the action of the avatar 4B touching the avatar 4A using the virtual hand 400, the action of grabbing and shaking the body of the avatar 4A, and the action of hitting the avatar 4A (for example,). It may include an action (slap the face or shoulder of Avatar 4A) and the like. The specific action may be an action including a repetitive motion, that is, an motion of touching the avatar 4A in the process of the repetitive motion of the virtual hand 400. By doing so, even if the avatar 4B accidentally touches the avatar 4A due to the erroneous operation of the user B, it can be detected that it is not a specific action, and the erroneous detection of the operation of the virtual hand 400 can be prevented. Can be done.

For example, as shown in FIG. 20, when the virtual hand 400 is repeatedly moved while the avatar 4B is holding the body (for example, the torso) of the avatar 4A by the virtual hand 400, the control unit 121 specifies. Identify that the shaking action was performed as an action. In grasping (selecting) the body of the avatar 4A by the virtual hand 400, for example, the user B operates a key (not shown) of the external controller 320 in a state where the collider of the virtual hand 400 and the collider of the body of the avatar 4A are in contact with each other. This can be achieved by pressing (pressing). This shaking motion may include a contact motion with the avatar 4A in the process of repetitive motion of the virtual hand 400 at an acceleration of a certain value or more. For example, in the state shown in FIG. 20, the virtual hand 400 is reciprocated in the z-axis direction while the body of the avatar 4A is grasped by the virtual hand 400, the acceleration in the + z-axis direction is equal to or higher than a certain value, and the virtual hand 400 is reciprocated. When the state in which the acceleration in the −z-axis direction is equal to or higher than a certain value continues for a predetermined number of times or more, the shaking operation may be performed. However, the direction in which the virtual hand 400 is reciprocated is not limited to the z-axis direction, and may be any direction. Further, the shaking motion may include an motion in which the virtual hand 400 reciprocates one or more relative coordinates associated with the avatar 4A a predetermined number of times or more within a fixed time. For example, in the state shown in FIG. 20, the relative coordinates of 1 or more are the first relative coordinates provided at arbitrary positions in the + z-axis direction with respect to the body portion of the avatar 4A grasped by the virtual hand 400. , And a second relative coordinate provided at an arbitrary position in the −z axis direction. In this example, while the body of the avatar 4A is grasped by the virtual hand 400, the virtual hand 400 is reciprocated in the z-axis direction, the first relative coordinates are reciprocated a predetermined number of times or more within a certain time, and the second. When the relative coordinates of the above are reciprocated more than a predetermined number of times within a certain time, the shaking operation may be performed. However, the direction in which the virtual hand 400 is reciprocated is not limited to the z-axis direction, and may be any direction. In this case, the relative coordinates with respect to the virtual hand 400 may also be provided in any direction.

It is preferable to determine that the specific action has been executed for the operation performed by the user B himself / herself as a shaking. That is, since the shaking direction is not uniform and is assumed to differ depending on the user, the moving direction of the virtual hand 400 may be any direction and does not need to be limited to a specific direction. However, when it is necessary to strictly determine the operation of a specific action or to give reality to the operation of the avatar 4B (for example, to prevent the virtual hand 400 from getting stuck in the avatar 4A), the control unit 121 May determine that there was a shaking motion when the repetitive motion was performed in a specific direction in consideration of the moving direction of the virtual hand 400.

On the other hand, when the avatar 4B does not grasp the body of the avatar 4A by the virtual hand 400 and the virtual hand 400 contacts the avatar 4A (collider collision) at a certain acceleration or higher, the control unit 121 strikes as a specific action. Identify that the action was performed. In order to prevent erroneous detection of the motion of the virtual hand 400, the slap motion is particularly a repetitive motion of the virtual hand 400 having a certain acceleration or more, and a plurality of contact motions (colider collision) with the avatar 4A in the process of the repetitive motion. ) Is preferably included (specifically, reciprocating slap operation). For example, in the state shown in FIG. 19, the virtual hand 400 is reciprocated in the y-axis direction, and the acceleration in the + y-axis direction is at least a certain value and the acceleration in the −y-axis direction is at least a certain value. When the collider of the virtual hand 400 and the collider of the face of the avatar 4A come into contact with each other more than a predetermined number of times, the tapping motion may be performed. However, the direction in which the virtual hand 400 is reciprocated is not limited to the y-axis direction, and may be any direction. In this round-trip slap operation, for example, the virtual hand 400 of the avatar 4B reciprocates the relative coordinates associated with the avatar 4A a predetermined number of times or more within a fixed time, and the virtual hand 400 touches the avatar 4A a plurality of times within a fixed time. It may be specified by the fact that there is a hit determination). For example, in the state shown in FIG. 19, the relative coordinates of 1 or more are the first relative coordinates provided at an arbitrary position in the + y-axis direction with respect to the face of the avatar 4A, and any arbitrary coordinates in the −y-axis direction. A second relative coordinate provided at the position can be mentioned. In this example, the virtual hand 400 is reciprocated in the y-axis direction, the first relative coordinates are reciprocated a predetermined number of times or more within a fixed time, and the second relative coordinates are reciprocated a predetermined number of times or more within a fixed time. In the state, when the collider of the virtual hand 400 and the collider of the face of the avatar 4A occur more than a predetermined number of times, the tapping operation may be performed. However, the direction in which the virtual hand 400 is reciprocated is not limited to the y-axis direction, and may be any direction. In this case, the relative coordinates with respect to the virtual hand 400 may also be provided in any direction. However, the control unit 121 may determine that the tapping motion is performed when the repetitive motion is performed in a specific direction in consideration of the moving direction of the virtual hand 400.

The avatar 4A may be set to follow the movement of the avatar 4B. For example, as shown in FIG. 20, when the avatar 4B executes the shaking motion with respect to the avatar 4A, the control unit 121 moves the avatar 4A so that the avatar 4A is shaken following the shaking motion. Can be set. However, when the avatar 4B performs a shaking motion or a tapping motion with respect to the avatar 4A, the avatar 4A may be set not to follow the motion, that is, to not move.

Next, when the control unit 121 determines that the avatar 4B has executed the specific action on the avatar 4A (YES in step S58), the information indicating that the specific action has been executed by the user B (action execution). Information) is generated (step S60). After that, the control unit 121 transmits the action execution information to the server 2 via the communication network 3 (step S61).

Next, the server 2 uses information regarding the participation consultation notification for asking the user A to participate in the content provided in the virtual space 200 based on the action execution information received from the user terminal 1B (participation consultation notification information). Is generated (step S62). The participation consultation notification is a notification for asking the user A who has removed the HMD 110 to participate in the content provided in the virtual space 200. The participation consultation notification information includes, for example, the identification information of the avatar 4B that executed the specific action, the identification information of the avatar 4A that received the specific action, the type of the specific action, the strength of the action of the specific action, and the like. May be good.

Next, the server 2 identifies the personal information of the user A based on the generated participation consultation notification information (step S63). For example, the personal information of the user A (for example, an e-mail address, an SNS (Social Networking Service) address, etc.) is registered in advance in an information table stored in the storage unit 22 of the server 2, and is managed in association with the identification information of the avatar 4A. Has been done. In step S63, the server 2 refers to the information table stored in the storage unit 22 and acquires the personal information of the user A based on the identification information of the avatar 4A included in the participation consultation notification information. However, the personal information of the user A may be managed by a server different from the server 2. Examples of the server different from the server 2 include, for example, a server capable of distributing a game program to a user in which personal information is registered, that is, a server playing the role of a game distribution platform, and the like, but the server is limited to this. It's not something.

Next, the server 2 transmits the participation consultation notification information to the destination (for example, the email address of the user A) acquired in step S63 (step S64). As a method of transmitting the participation consultation notification information to the user A, for example, a communication tool such as e-mail, chat, or SNS can be used.

Next, the user terminal 5 receives the participation consultation notification information by the communication tool, and displays the reception notification of the participation consultation notification information on the display unit 6 (step S65). The user terminal 5 is a terminal device owned by the user A, and may be a portable terminal such as a smartphone, a PDA (Personal Digital Assistant), a tablet or a fablet type computer, a wearable device, or a personal computer. , A video game device, or the like.

Next, the user terminal 5 activates the communication tool based on the input operation by the user A who sees the reception notification, and displays the message M1 on the display unit 6 as shown in FIG. 21 (step S66). The message M1 is information indicating that the user B is asking the user A to participate in the virtual space 200 (for example, "just now [content name (for example, the name of the content provided in the virtual space 200)). ], The communication is received from [the other player's name (for example, user B)]. Start [Content] and go to see you! "). If the user terminal 5 is a personal computer or a video game device, and the display of these devices also displays an image of the content provided in the virtual space 200, a message regarding the participation consultation notification is displayed on the display. May be displayed. Further, the message M1 may be a message prompting the user A to attach the HMD 110 of the user terminal 1A.

Next, the user terminal 5 determines that the user A has read the message M1 based on the fact that the message M1 is displayed on the display unit 6, and generates read information indicating that the message M1 has been read. (Step S67). The read information is generated based on the reaction of the user A to the message M1, and may be generated, for example, based on a mail reply, a "like" point, or the like. After that, the user terminal 5 transmits the read information to the server 2 via the communication network (step S68).

Next, the server 2 receives the read information from the user terminal 5, and then transmits the read information to the user terminal 1B via the communication network 3 (step S69).

Next, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A arranged in the virtual space 200B to the updated facial expression based on the read information received from the server 2 (step S70). ). For example, as shown in FIG. 22, the control unit 121 updates the facial expression of the avatar 4A, which was in the state where the eyes are closed (an example of the first state), to the state where the eyes are open (an example of the second state). ..

In step S70, the control unit 121 may update the posture of the avatar 4A based on the read information received from the server 2. For example, the posture of the avatar 4A in the lying state may be updated to the sitting state.

Next, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S71). As a result, the user B can easily grasp that the user A has made a reaction to the participation consultation notification by visually recognizing that the avatar 4A is in the open state.

On the other hand, in step S72, the server 2 transmits the read information to the user terminal 1A in addition to the user terminal 1B.

Next, the control unit 121 of the user terminal 1A determines whether or not the user A wears the HMD 110 within a certain period of time after receiving the read information (step S73). The control unit 121 may determine whether or not the user A has attached the HMD 110 based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the attachment sensor 115. The determination of whether or not the user A has attached the HMD 110 is based on the inclination information indicating the inclination (roll angle, yaw angle, pitch angle) of the HMD 110 acquired by the HMD sensor 114, and the inclination of the HMD 110 is larger than the predetermined inclination. It can be judged by whether it is small or not. As described above, while the user A is wearing the HMD 110, the inclination of the HMD 110 is equal to or less than the predetermined inclination. Therefore, when the inclination of the HMD 110 is smaller than the predetermined inclination, the user A wears the HMD 110. Is identified.

When it is determined in step S73 that the user A has attached the HMD 110 (YES in step S73), the control unit 121 of the user terminal 1A shifts from the sleep mode to the active mode (step S74). Next, the control unit 121 that has transitioned to the active mode generates control information for the avatar 4A and also generates wearing information indicating that the user A has worn the HMD 110 (step S75). After that, the control unit 121 transmits the control information and the wearing information of the avatar 4A to the server 2 via the communication network 3 (step S76).

Next, the server 2 receives the control information and the wearing information of the avatar 4A from the user terminal 1A, and then transmits the control information and the wearing information of the avatar 4A to the user terminal 1B via the communication network 3 (step S77).

Next, the control unit 121 of the user terminal 1B updates the operation of the avatar 4A based on the control information of the avatar 4A, and updates the operation of the avatar 4B based on the control information of the avatar 4B (step S78). In particular, the control unit 121 updates the posture of the avatar 4A based on the wearing information of the avatar 4A. Further, the control unit 121 may update the facial expression of the avatar 4B based on the face information of the avatar 4B included in the control information of the avatar 4B. Specifically, the control unit 121 sets the posture of the avatar 4A to a raised state (an example of a third state), for example, as shown in FIG. 23.

As in the embodiment shown in FIG. 17, the control unit 121 may generate a balloon object associated with the avatar 4A in the virtual space 200B. Information indicating that the user A has attached the HMD 110 (for example, "User A has returned to the seat", etc.) is displayed in the balloon object.

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S79).

On the other hand, if it is determined in step S73 that the user A is not wearing the HMD 110 within a certain period of time from the reception of the read information (NO in step S73), the process proceeds to FIG. 24. Next, as shown in FIG. 24, the control unit 121 generates non-wearing information indicating that the user A is not wearing the HMD 110 (step S80). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S81).

Next, the server 2 receives the non-wearing information from the user terminal 1A, and then transmits the non-wearing information to the user terminal 1B via the communication network 3 (step S82).

Next, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A arranged in the virtual space 200B to the default facial expression based on the non-wearing information received from the server 2 (step S83). For example, when the default facial expression is a state in which the eyes are closed, the facial expression of the avatar 4A is updated to the state in which the eyes are closed, as shown in FIG. In the process of step S83, if the control unit 121 of the user terminal 1B measures a certain time from the reception of the read information and does not receive the control information and the wearing information of the avatar 4A within a certain time, the avatar 4A The facial expression may be set to the default facial expression. If the eyes of the avatar 4A are open based on the read information in step S70 of FIG. 18, the facial expression of the avatar 4A is returned to the closed state in the step S83. On the other hand, when the participation consultation notification has not been read (NO in step S66), the facial expression of the avatar 4A is changed in this step S83 because the avatar 4A is kept in a closed state. Not changed.

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S84).

As described above, according to the information processing method according to the first modification, the step of moving the virtual hand 400 in the virtual space 200 according to the movement of the user B and the step in the virtual space 200 based on the movement of the user A. Based on the step of executing an action (first action) such as a shaking motion by the virtual hand 400 on the avatar 4A whose movement can be controlled, and the execution of the action, the user terminal 5 associated with the user A (is associated with). ) To send a participation consultation notification (second action) to ask for participation in the virtual space 200, and the like. According to this method, communication in the virtual space 200 can be performed by executing a predetermined action between a plurality of users A and B sharing the virtual space 200 by an intuitive operation using the virtual hand 400. It can be performed smoothly without impairing the user's immersive feeling. As described above, according to the first modification, the user's virtual experience is provided by providing a user interface (so-called dietic UI) that does not require menu operations or key operations for communication between users. Can be improved.

In particular, by transmitting the participation consultation notification from the user B to the user terminal 5 associated with the user A based on the specific action executed for the avatar 4A, the content provided from the user B to the virtual space 200 can be obtained. The user A can easily recognize that the participation has been consulted. This makes it possible to promote the use of content (for example, a multiplayer game) that needs to be played by a plurality of users at the same time in the virtual space 200.

Further, in the information processing method according to the first modification, the user B executes a specific action on the avatar 4A in a state where the avatar 4A is not controlled by the user A (that is, a state in which the user A does not wear the HMD 110). If so, the participation consultation notification is sent to the user A. Therefore, the user A in the state where the HMD 110 is not attached can be attracted to the content provided in the virtual space 200, and the participation in the content can be further promoted. In particular, in a VR game, when there are few users (avatars) waiting in the room, that is, when there are few candidates for multiplayer opponents, the users (avatars) enter the room for the time being, but multiplayer candidates appear. Since the HMD is often removed and the user waits until, such a waiting user can be effectively attracted to the multiplayer of the VR game.

Further, in the information processing method according to the modification 1, when the user A reacts to the participation consultation notification (for example, when the message M1 is read), the avatar 4A in the virtual space 200B is used. , The step of transitioning from the closed eyes state (first state) to the open eyes state (second state), and after the transition of the avatar 4A to the open eyes state, the avatar by the user A within a certain period of time. When the 4A is in the controlled state, the step of transitioning the avatar 4A in the virtual space 200B to the upright state (third state) may be further included. As a result, the user B can easily grasp that there is a reaction from the user A and that the user A wears the HMD 110 and the avatar 4A is controlled.

On the other hand, if the avatar 4A is not controlled by the user A within a certain period of time after the avatar 4A transitions to the open state, the avatar 4A in the virtual space 200B is closed. You may return to. As a result, the user B can easily grasp that the avatar 4A is not controlled by the user A even though the participation consultation notification is transmitted (that is, the user A does not participate in the content). Can be done. Therefore, the user B shakes the avatar 4A again to ask for participation in the virtual space again, or the user B participates in the virtual space for an avatar associated with a user other than the user A who shares the virtual space 200. You can consult with them.

The state in which the avatar 4A is not controlled by the user A includes not only the case where the user A has removed the HMD 110 but also the case where the user A has not logged in to the content provided in the virtual space 200 in the first place. obtain. If the user A is not logged in to the content, the avatar 4A does not normally exist in the virtual space 200B provided by the user terminal 1B of the user B, but the user B should register the user A as a friend. For example, the avatar 4A of the user A can exist in the virtual space 200B even if the user A is not logged in to the content. Whether or not this friend is registered may be registered in the server 2 or the like together with the personal information of each user, for example. In this case, the control unit 121 of the user terminal 1B can set the posture and facial expression of the avatar 4A arranged in the virtual space 200B by default based on the fact that the user A is not logged in to the content (FIG. 18). Step S54). In this way, the avatar 4A associated with the user A who is not logged in to the content is made to exist in the virtual space 200B, and the user A is consulted based on the execution of the specific action from the avatar 4B to the avatar 4A. By sending the notification, the motivation for user A to log in to the content can be provided.

Further, the content of the participation consultation notification (message M1) may be different depending on the strength of the action of the specific action for the avatar 4A by the avatar 4B. For example, a plurality of threshold values may be set stepwise with respect to the acceleration of the operation by the virtual hand 400, and the content of the message M1 may be changed according to the threshold value of the acceleration. For example, the message M1 may include information on the degree of request for participation, and the information on the degree of request for participation may be changed according to the strength of the shaking motion. Specifically, when the shaking motion is intense, the content indicating that the participation request from the user B is high (for example, "participation request: ☆☆☆ (3 stars)") is displayed as the message M1. You may do so. It should be noted that a predetermined threshold value may be set for the number of times the virtual hand 400 passes through the relative coordinates associated with the avatar 4A, and the intensity of the action of the specific action may be specified according to the threshold value of the number of times of passing. ..

Further, when a plurality of avatars simultaneously (or collaborate) perform a specific action on the avatar 4A, the content of the message M1 may be changed. In this case, it is preferable to provide a dedicated collider for each specific part of the body of the avatar A so that the avatar A can be grasped by a plurality of avatars at the same time. This enables the shaking operation of the avatar 4A by a plurality of avatars. In this case, it is preferable that the larger the number of users who perform a specific action on the avatar 4A, the higher the degree of request for participation.

Further, the content of the message M1 may be different depending on the type of the specific action executed on the avatar 4A. For example, when the avatar 4B slaps the avatar 4A, the degree of participation request displayed in the message M1 may be higher than when the avatar 4B shakes the avatar 4A.

In this way, by making the content of the participation consultation notification different according to the strength of the action of the specific action for the avatar 4B by the avatar 4B and the type of the specific action, the user A is requested to participate in the content from the user B. You can grasp the height of. Therefore, the user A can decide whether or not to participate in the content according to the high degree of request.

Further, in the present embodiment, an action accompanied by physical interference is exemplified as a specific action for the avatar 4A by the avatar 4B, but the action is not limited to this example. The specific action may be an action for identifying the user A. For example, the specific action executed from the avatar 4B to the avatar 4A is a line of sight, a voice, a laser pointer, a facial expression (smile, glaring, etc.). , Applause, etc. may be specified. Further, the user B may select an avatar (user) who wants to ask for participation in the content from the avatar list (user list) displayed in the virtual space 200B. That is, the specific action executed by the avatar 4B may be a direct action or an indirect action on the avatar 4A. Further, a direct action and an indirect action may be combined.

Further, the transmission timing of the participation consultation notification may be different depending on the content of the content provided in the virtual space 200. For example, when the number of users required to start multiplayer is three or more, the control unit 121 of the user terminal 1B is the last avatar among the plurality of avatars required to start multiplayer. When it is determined that the specific action has been executed, the control unit 121 of the user terminal 1B collectively sends a participation consultation notification to each user associated with the plurality of avatars that have executed the specific action. You may do so. Even if the number of users required to start multiplayer is three or more, the control unit 121 of the user terminal 1B performs a specific action on each avatar at the timing when the avatar 4B performs a specific action on each avatar. You may also send the participation consultation notification individually.

Further, in the above embodiment, the facial expression of the avatar 4A with closed eyes is displayed based on the reaction of the user A to the participation consultation notification (for example, the message M1 regarding the participation consultation notification has been read). , I'm updating with my eyes open, but it's not limited to this example. For example, the avatar 4A may be updated with the eyes open based on the execution of the specific action on the avatar 4A by the avatar 4B. As a result, the user B can easily grasp that the specific action has been appropriately executed for the avatar 4A. It should be noted that, based on the fact that the user A wears the HMD 110, the avatar 4A may be updated with the eyes open.

Further, the information processing method according to the modification 1 includes a step of executing a predetermined action (an example of the first action) accompanied by physical interference from the user B to the user A in the virtual space 200, and execution of the action. Based on the step of executing the participation consultation notification to the user A (an example of the second action) and the reaction of the user A to the participation consultation notification (for example, the message M1 regarding the participation consultation notification has been read). A step of executing the update of the posture and / or the facial expression of the avatar 4A (an example of the third action) in the virtual space 200 is included. According to this method, the posture and facial expression of the avatar 4A associated with the user A in the virtual space 200B are updated based on the reaction of the user A to the participation consultation notification transmitted to the user A, thereby updating the virtual space. Communication between users A and B within 200 can be smoothly performed. In this way, by reflecting the reaction of the user A to the participation consultation notification in the real space in the virtual space 200, it is possible to provide a seamless virtual experience that blurs the boundary between the real space and the virtual space. ..

(Modification 2)
Next, a modification 2 of the information processing method according to the present embodiment will be described with reference to FIGS. 25 to 28. Modification 2 exemplifies an information processing method at the start of a game when a game (so-called single play game) played only by user B is provided in the virtual space 200 (200B). FIG. 25 is an example of a flowchart for explaining the information processing method according to the modified example 2. 26 to 28 are diagrams showing an example of the virtual space 200B provided to the user B for explaining the information processing method according to the modification 2. As shown in FIG. 26, in the second modification, the virtual space 200B includes the avatar 4B and the avatar 4C. The avatar 4C is an avatar 4C (so-called Non Player Character (NPC)) that can be controlled by the server 2.

With reference to FIG. 25, first, the control unit 121 of the user terminal 1B accepts the game start input based on the operation by the user B (step S100). Next, the control unit 121 transmits the game start input information to the server 2 via the communication network 3 (step S101).

Next, the server 2 reads the data of the opening movie from the storage unit 22 based on the game start input information received from the control unit 121 (step S102). The opening movie is a video reproduced in the virtual space 200B as a scenario scene (an example of the first scene) introduced at the start of the game. Next, the server 2 transmits the data of the opening movie to the control unit 121 of the user terminal 1B via the communication network 3 (step S103).

Next, the control unit 121 of the user terminal 1B plays the opening movie in the virtual space 200B based on the data of the opening movie received from the server 2 (step S104).

During or after the opening movie is being played back, the control unit 121 of the user terminal 1B sets the posture and facial expression of the avatar 4C to the default posture and facial expression (step S105). Assuming that the default posture is lying down and the default facial expression is the closed eyes, the posture of Avatar 4C is set to lying down and the face of Avatar 4C is set as shown in FIG. The facial expression is set with the eyes closed.

Next, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S106).

Next, after updating the visual field image in step S106, the control unit 121 causes the avatar 4B associated with the user B to perform a specific action on the avatar 4C associated with the server 2 (an example of the first action). Is determined (step S107). Since the determination process of the specific action is the same as that of the first modification, the detailed description thereof will be omitted. In order to encourage the avatar 4B to execute a specific action on the avatar 4C, the control unit 121 displays a message such as "Please wake up the avatar 4C" in the virtual space 200B after playing the opening movie in step S104. You may.

Next, when the control unit 121 determines that the avatar 4B has executed the specific action on the avatar 4C (YES in step S107), the control unit 121 generates action execution information indicating that the specific action has been executed by the user B. (Step S108). After that, the control unit 121 transmits the action execution information to the server 2 via the communication network 3 (step S109).

Next, the server 2 generates scene transition information based on the action execution information received from the user terminal 1B (step S110). The scene transition information is information for transitioning the game provided in the virtual space 200B from the scenario scene to the game scene (an example of the second scene). Next, the server 2 transmits the scene transition information to the control unit 121 of the user terminal 1B via the communication network 3 (step S111).

Next, the control unit 121 of the user terminal 1B updates the posture and facial expression settings of the avatar 4C based on the scene transition information received from the server 2 (step S112). Specifically, for example, as shown in FIG. 28, the posture of the avatar 4C is set to the raised state, and the facial expression of the avatar 4C is set to the state of opening the eyes.

Next, the control unit 121 creates a balloon object 45A associated with the avatar 4C in the virtual space 200B (step S113). Information indicating that the avatar 4C has joined the group of the avatar 4B (for example, "Avatar 4C has joined the group") is displayed on the balloon object 45A.

After that, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the visual field CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and then displays the field of view image on the HMD 110 based on the updated field of view image data. (Step S114).

After that, the control unit 121 starts displaying the game scene based on the scene transition information (step S115).

As described above, the information processing method according to the modification 2 is specified as a step of executing a specific action such as a shaking motion by the virtual hand 400 for the avatar 4C whose operation in the virtual space 200B can be controlled by a computer. It includes a step of transitioning the game provided in the virtual space 200B from the scenario scene (first scene) to the game scene (second scene) based on the execution of the action. Conventionally, in order to start a game scene, a user interface for advancing the game after playing the scenario scene (for example, "Press Any Button") is displayed, and the user presses a predetermined button on the external controller 320. The game scene was started by. On the other hand, according to the method according to the second modification, the game scene is started based on the execution of the specific action of the user B with respect to the avatar 4C, so that the user B's immersive feeling in the virtual space 200B is not impaired. You can advance the game. In the second modification, the case where the processing of the second modification is executed in a system having the server 2 and the user terminal 1B has been described as an example, but the present invention is not limited to this, and the processing may be executed by the user terminal 1B alone. good.

Further, the information processing method for advancing the game according to the second modification is not limited to the scene when the game is started, but can also be applied to the scene where the game is restarted in the middle of the game. For example, when a game played by a party composed of a plurality of avatars (for example, avatar 4B and avatar 4C) is provided to the virtual space 200B, a specific action is executed from the avatar 4B to the avatar 4C after the game progress is interrupted. If it is determined that the game has been played, the game may be restarted. Specifically, in the scene where the party stayed at the inn, it is assumed that the avatar 4C is sleeping even the next morning. In this state, the avatar 4B may perform a specific action such as a shaking motion or a tapping motion on the avatar 4C so that the avatar 4C awakens and the progress of the game is restarted. According to such a method, the progress of the game can be restarted without impairing the immersive feeling of the user B in the virtual space 200B.

Further, the order of processing defined in each step shown in FIGS. 18, 24 and 25 is only an example, and the order of these steps can be changed as appropriate.

Further, in the description of the above-described embodiment and modification, it is premised that the presence / absence of mounting of the HMD 110 is detected by the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the mounting sensor 115. , Not limited to this example. For example, the control unit 121 of the user terminal detects the movement of the HMD 110 by the position sensor 130 and / or the HMD sensor 114, and when the movement follows a predetermined trajectory, the user wears the HMD 110, or the HMD 110. You may specify that you have removed it. Further, the control unit 121 detects the presence or absence of movement of the HMD 110 by the position sensor 130 and / or the HMD sensor 114, and for example, when the position of the HMD 110 does not change at all within a certain period of time, the user wears the HMD 110. It may be specified that it has not been done. Further, the control unit 121 may detect whether or not the HMD 110 is attached by detecting the relative positional relationship between the HMD 110 and the external controller 320. For example, when the HMD 110 and the external controller 320 are separated by a certain distance or more, the control unit 121 may specify that the user does not wear the HMD 110.

Further, in the description of the above-described embodiment and modification, it is assumed that the virtual space data indicating the virtual space 200B is updated on the user terminal 1B side, but even if the virtual space data is updated on the server 2 side. good. Further, although it is assumed that the visual field image data corresponding to the visual field image is updated on the user terminal 1B side, the visual field image data may be updated on the server 2 side. In this case, the user terminal 1B displays the field of view image on the HMD 110 based on the field of view image data transmitted from the server 2.

Further, the order of processing defined in each step shown in FIGS. 12 and 16 is only an example, and the order of these steps can be changed as appropriate.

Further, in the present embodiment, the virtual space (VR space) in which the user is immersed by the HMD device has been described as an example, but a transmissive HMD device may be adopted as the HMD device. In this case, an augmented reality (AR) space or mixed reality is output by outputting a view image obtained by synthesizing a part of the image constituting the virtual space in the real space visually recognized by the user via the transmissive HMD device. (MR: Mixed Reality) A virtual experience in space may be provided to the user. In this case, instead of the virtual hand, an action on the target object in the virtual space may be generated based on the movement of the user's hand. Specifically, the processor may specify the coordinate information of the position of the user's hand in the real space, and may define the position of the target object in the virtual space in relation to the coordinate information in the real space. As a result, the processor can grasp the positional relationship between the user's hand in the real space and the target object in the virtual space 2, and can execute the process corresponding to the collision control and the like described above between the user's hand and the target object. Become. As a result, it becomes possible to give an action to the target object based on the movement of the user's hand.

Further, in order to realize various processes executed by the control unit 121 of the user terminal 1 by software, even if a control program for causing a computer (processor) to execute various processes is preliminarily incorporated in the storage unit 123 or the memory. good. Alternatively, the control program includes a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray (registered trademark) disk, etc.), a magneto-optical disk (MO, etc.), a flash memory (SD card, SD card, etc.). It may be stored in a computer-readable storage medium such as a USB memory, SSD, etc.). In this case, by connecting the storage medium to the control device 120, the control program stored in the storage medium is incorporated into the storage unit 123. Then, the control program incorporated in the storage unit 123 is loaded onto the RAM, and the processor executes the loaded program, so that the control unit 121 executes various processes.

Further, the control program may be downloaded from the computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded control program is incorporated into the storage unit 123.

Although the embodiments of the present disclosure have been described above, the technical scope of the present invention should not be construed as being limited by the description of the present embodiments. This embodiment is an example, and it is understood by those skilled in the art that various embodiments can be modified within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the scope thereof.

1,1A, 1B: User terminal 2: Server 3: Communication network 4A, 4B: Avatar 21: Communication interface 22: Storage unit 23: Control unit 24: Bus 42A, 44A, 45A: Blowout object 43A: Telop object 100: Virtual Spatial distribution system (distribution system)
110: HMD (Head Mounted Device)
112: Display unit 113: Face camera 114: HMD sensor 115: Wearing sensor 116: Headphones 117: Face camera 118: Microphone 120: Control device 123: Storage unit 124: I / O interface 125: Communication interface 126: Bus 130: Position Sensor 140: Gaze sensor 200, 200A, 200B: Virtual space 210: Center position 300: Virtual camera 320: External controller 320L: External controller for left hand (controller)
320R: External controller for right hand (controller)
400: Virtual hand U, A, B: User M1: Message

Claims (12)

To a computer for providing a virtual experience to the user via an image display device associated with the user's head .
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
When the first action is executed when the other user is not wearing the second image display device associated with the head of the other user, the second action for the other user is performed. And the steps to perform
A program to execute. The program according to claim 1 , wherein the first action includes an action of contacting the virtual object in the process of repetitive motion of the operation object. The program according to claim 2 , wherein the first action includes a contact operation with the virtual object in the process of the repetitive motion at an acceleration of a certain value or more. The program according to claim 2 or 3 , wherein the first action includes an operation of passing through relative coordinates associated with the virtual object a predetermined number of times or more within a certain period of time. The program according to any one of claims 1 to 4 , wherein the second action includes a notification for encouraging the other user to participate in the virtual space. A step of transitioning the virtual object from the first state to the second state when a reaction to the second action is performed.
If the virtual object is controlled by the other user within a certain period of time after the transition to the second state, the step of transitioning the virtual object to the third state and the step.
The program according to claim 5 , further causing the computer to execute the program. If the virtual object is not controlled by the other user within a certain period of time after the transition to the second state, the computer further executes a step of returning the virtual object to the first state. The program according to claim 6 . The program according to any one of claims 5 to 7 , wherein the content of the notification differs depending on at least one of the intensity of the operation of the first action and the type of the first action with respect to the virtual object. .. A step of executing the first action by the operation object with respect to another virtual object whose operation in the virtual space can be controlled by a computer.
A step of transitioning the game provided in the virtual space from the first scene to the second scene based on the execution of the first action.
The program according to any one of claims 1 to 8 , further causing the computer to execute the program. An information processing device for providing a virtual experience to the user via an image display device associated with the user's head .
By controlling the processor included in the information processing device
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
When the first action is executed when the other user is not wearing the second image display device associated with the head of the other user, the second action for the other user is performed. And the steps to perform
Is executed, an information processing device. An information processing system for providing a virtual experience to the user via an image display device associated with the user's head .
By controlling a plurality of processors included in the information processing system
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
When the first action is executed when the other user is not wearing the second image display device associated with the head of the other user, the second action for the other user is performed. And the steps to perform
Is executed, an information processing system. A method of information processing performed by a computer to provide a virtual experience to the user via an image display device associated with the user's head .
The step of moving the operation object in the virtual space according to the movement of a part of the user's body,
A step of executing a first action by the operation object on a virtual object whose operation in the virtual space can be controlled based on the movement of another user different from the user.
When the first action is executed when the other user is not wearing the second image display device associated with the head of the other user, the second action for the other user is performed. And the steps to perform
Information processing methods, including.

Images