JP2019211868A - Program, computer and method for providing user with virtual experience via image display device - Google Patents

Abstract

To provide a novel technique for suppressing withdrawal from a game not intended by a user in the game in which a plurality of users participate.SOLUTION: A program according to an embodiment of the present disclosure causes a computer for providing users with a virtual experience in a game in which a plurality of the users participate via respective image display devices to execute steps of: defining a virtual space for providing the virtual experience; providing the users with the virtual space via the respective image display devices; detecting a removal action of the user to remove the image display device associated with a head of the user; and generating output that prompts the user whose removal action is detected, to perform input related to returning to the game.SELECTED DRAWING: Figure 11

Description

  The present disclosure relates to a program, a computer, and a method for providing a virtual experience to a user via an image display device. More specifically, the present disclosure relates to a program, a computer, and a method for executing processing related to a user leaving in a game.

  Patent Document 1 discloses a method in which, in a game in which a plurality of users participate, each user who participates in the game evaluates manners in the game of other users who participate in the game. Note that the game described in Patent Document 1 is not a game to which a virtual reality (VR) technology is applied. According to the method of Patent Document 1, each user is restricted from participating in the game or suffers a disadvantage in the game based on the manner evaluation result for the user.

  The technique of Patent Literature 1 indirectly suppresses the user from performing bad manners such as illegal withdrawal by restricting the user's participation in the game or giving a disadvantage. Here, “unauthorized withdrawal” means that the user disconnects the communication connection of the game terminal or the game home screen so that the user can end the game halfway when the situation of the game becomes disadvantageous for the user. Or to go back to.

  According to the method of Patent Document 1, when a certain user leaves illegally, other users cannot determine whether or not this illegal leaving is intentional. Therefore, both intentional fraud and unintentional fraud leave receive the same manner rating. In this case, unintentional unauthorized departure, for example, unintentional unauthorized departure of a user who is not familiar with the game cannot be suppressed without giving a restriction or disadvantage to participation in the game.

Japanese Patent No. 5679227

  An object of the present disclosure is to provide a novel technique for suppressing unauthorized leaving of a user in a virtual experience in a virtual space in which a plurality of users participate.

According to an embodiment of the present disclosure, the first user performs a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user. On the computer to provide to
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
A step of causing the image display device to output a UI (User Interface) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected;
A program for executing is provided.

According to another embodiment of the present disclosure, a virtual experience in a virtual space in which a plurality of users including the first user participates via the image display device associated with the head of the first user. A computer for providing to a user,
By control by a processor included in the computer,
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
When the removal operation is detected, there is provided a computer in which a step of causing the image display device to output a UI (User Interface) for prompting an input as to whether or not to end the virtual experience is executed.

According to another embodiment of the present disclosure, a virtual experience in a virtual space in which a plurality of users including the first user participates via the image display device associated with the head of the first user. A computer-implemented method for providing to a user comprising:
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
Causing the image display device to output a user interface (UI) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected.

  According to the embodiment of the present disclosure, it is possible to provide a novel technique for suppressing withdrawal from a game not intended by the user in a game in which a plurality of users are involved in play.

  Other features and advantages of the present disclosure will become apparent from the following description of embodiments, the accompanying drawings, and the appended claims.

1 is a diagram schematically showing a configuration of an HMD system. FIG. 3 is a block diagram illustrating an example of a basic hardware configuration of a computer according to an embodiment of the present disclosure. It is a figure which represents notionally the uvw visual field coordinate system set to HMD according to one Embodiment. FIG. 3 is a diagram conceptually illustrating one aspect of expressing a virtual space according to an embodiment. It is the figure showing the head of the user who wears HMD from the top according to one embodiment. It is a figure showing the yz cross section which looked at the visual recognition area from the x direction in virtual space. It is a figure showing the xz cross section which looked at the visual recognition area from the y direction in virtual space. It is a figure showing the schematic structure of a controller according to one Embodiment. It is a figure showing the schematic structure of a controller according to one Embodiment. FIG. 3 is a block diagram illustrating computer functionality including a game departure control function for providing a virtual experience to a user via an HMD system, according to an embodiment of the present disclosure. It is a flowchart of the general process for displaying the image of the virtual space where a user immerses on a display part. 3 is a flowchart of a method according to an embodiment of the present disclosure. It is the schematic which shows a condition when the user A and the user B are performing online match play. It is a figure which illustrates a mode when the user A puts HMD with which the user A is mounted | worn or the controller which it has in the hand on the near desk. It is a figure which illustrates roughly the mode of the removal operation | movement detection using the locus | trajectory of the movement of HMD when removing HMD. It is a figure which illustrates roughly a mode that HMD removal operation | movement detection is performed using the attachment / detachment sensor provided in HMD. It is a figure which illustrates roughly a mode when a removal operation | movement is detected based on positions, such as HMD. It is a figure which illustrates roughly a mode when removing operation | movement detection is performed based on both motion and positions, such as HMD. It is a figure which illustrates roughly a mode when removing operation | movement detection is performed based on both motion and positions, such as HMD. It is a figure which illustrates roughly a mode when performing removal operation | movement detection based on a user's eyes. It is a figure which illustrates roughly a mode when performing removal operation | movement detection based on a user's eyes. It is a figure which illustrates roughly a mode when removing operation | movement detection is performed based on the positional relationship between HMD and a controller. It is a figure which illustrates roughly a mode when removing operation | movement detection is performed based on the positional relationship between HMD and a controller. It is a figure which shows an example of the alert provided as a user interface displayed on the display part 112 of HMD110. FIG. 6 is a schematic diagram illustrating several ways of providing additional alerts.

[Description of Embodiment of Present Disclosure]
First, configurations of exemplary embodiments of the present disclosure will be listed and described. The method, the program, and the computer according to the embodiments of the present disclosure may include the following configurations.

(Item 1)
A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user.
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
A step of causing the image display device to output a UI (User Interface) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected;
A program that executes

(Item 2)
The step of detecting the removal operation includes:
Detecting a first part included in the head of the first user;
The program according to item 1, comprising: a step of determining that the removal operation has been performed when the first part cannot be detected.

(Item 3)
The step of detecting the removal operation includes:
Tracking the image display device;
The program according to item 1, comprising the step of determining that the removal operation has been performed when the image display device is stationary or when the image display device moves along a predetermined trajectory.

(Item 4)
The step of detecting the removal operation includes:
Tracking an operating device used by the first user;
The step of determining that the removal operation has been performed when the operation device is stationary or when a positional relationship between the operation device and the image display device satisfies a predetermined condition. Program.

(Item 5)
In the computer,
Receiving an input to end the virtual experience;
Detaching the first user from the virtual experience in response to receiving an input to end the virtual experience;
5. The program according to any one of items 1 to 4, wherein the program is executed.

(Item 6)
In the computer,
Receiving an input that does not end the virtual experience;
Returning the first user to the virtual experience in response to receiving an input that does not end the virtual experience;
6. The program according to any one of items 1 to 5, wherein the program is executed.

(Item 7)
In the computer,
From the item 1, the step of determining that the first user has illegally left the virtual experience in response to receiving neither the input for ending the virtual experience nor the input not ending the virtual experience is received. The program according to any one of 6 above.

(Item 8)
The UI includes items 1 to 7 including a warning that the first user is deemed to have illegally left the virtual experience when the first user does not return to the virtual experience or perform a normal leave. The program according to any one of the above.

(Item 9)
Including the step of causing at least one of an audio output device and an image output device associated with the image display device to perform an output for encouraging normal departure from the virtual experience when the removal operation is detected. The program according to any one of 1 to 8.

(Item 10)
A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user,
By control by a processor included in the computer,
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
And a step of causing the image display device to output a UI (User Interface) for prompting an input as to whether or not to end the virtual experience when the removal operation is detected.

(Item 11)
Executed by a computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user. It is a method,
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
Causing the image display device to output a UI (User Interface) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected.

[Details of Embodiment of the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, similar elements are denoted by the same reference numerals. The names and functions are also the same. A duplicate description of such elements is omitted.

  The configuration of the image display apparatus system 100 will be described with reference to FIG. Hereinafter, a system in which a head-mounted device (HMD) is used as an example of an image display device will be described in detail. However, those skilled in the art will appreciate that various devices for providing a virtual experience to a user, as well as HMDs, can be applied to embodiments of the present disclosure.

  FIG. 1 schematically shows the configuration of the system 100. In one example, the system 100 is provided as a home system or a business system. The HMD may be a so-called head mounted display including a display unit, or may be a head mounted device on which a terminal such as a smartphone having the display unit can be attached.

  The system 100 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a display unit 112 and a gaze sensor 140. The controller 160 may include a motion sensor 130.

  In one example, the computer 200 may be connectable to a network 192 such as the Internet, and may be communicable with a computer such as a server 150 connected to the network 192. In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120.

  The HMD 110 may be worn on the head of the user 190 and provide a virtual space to the user during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the display unit 112, respectively. When each eye of the user visually recognizes each image, the user can recognize the image as a three-dimensional image based on the parallax of both eyes.

  The display unit 112 is realized as, for example, a non-transmissive display device. In one example, the display unit 112 is disposed on the main body of the HMD 110 so as to be positioned in front of both eyes of the user. Therefore, when the user visually recognizes the three-dimensional image displayed on the display unit 112, the user can be immersed in the virtual space. In an embodiment, the virtual space includes, for example, a background, an object that can be operated by the user, an image of a menu that can be selected by the user, and the like. In an embodiment, the display unit 112 may be realized as a liquid crystal display unit or an organic EL (Electro Luminescence) display unit included in an information display terminal such as a smartphone.

  In one example, the display unit 112 may include a sub display unit for displaying an image for the right eye and a sub display unit for displaying an image for the left eye. In another aspect, the display unit 112 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the display unit 112 includes a high-speed shutter. The high-speed shutter operates so that an image for the right eye and an image for the left eye can be displayed alternately so that the image is recognized only by one of the eyes.

  In one example, the HMD 110 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. More specifically, the HMD sensor 120 may read a plurality of infrared rays emitted from the HMD 110 and detect the position and inclination of the HMD 110 in the real space.

  In an aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and inclination of the HMD 110 by executing image analysis processing using image information of the HMD 110 output from the camera.

  In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120 as a position detector. The HMD 110 can detect the position and inclination of the HMD 110 itself using the sensor 114. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 detects its own position and inclination using any one of these sensors instead of the HMD sensor 120. obtain. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects angular velocities around the three axes of the HMD 110 in real space over time. The HMD 110 calculates a temporal change in the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates an inclination of the HMD 110 based on the temporal change in the angle. The HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance. Moreover, the view field image may include a configuration for presenting the real space in a part of the image configuring the virtual space. For example, an image captured by a camera mounted on the HMD 110 may be displayed so as to be superimposed on a part of the field-of-view image, or a part of the field-of-view image is set by setting a high transmittance of a part of the transmissive display device. Real space may be visible from a part.

  The gaze sensor 140 detects the direction (line of sight) in which the line of sight of the user 190's right eye and left eye is directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, the gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris with respect to the irradiated light. . The gaze sensor 140 can detect the line of sight of the user 190 based on each detected rotation angle.

  Server 150 may send a program to computer 200. In another aspect, the server 150 may communicate with other computers 200 for providing virtual reality to HMDs used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's operation with another computer 200, and a plurality of users are common in the same virtual space. Allows you to enjoy the game.

  The controller 160 is connected to the computer 200 by wire or wireless. The controller 160 receives input of commands from the user 190 to the computer 200. In an aspect, the controller 160 is configured to be gripped by the user 190. In another aspect, the controller 160 is configured to be wearable on the body of the user 190 or a portion of clothing. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the computer 200. In another aspect, the controller 160 receives an operation from the user 190 for controlling the position and movement of an object arranged in the virtual space.

  In an aspect, the motion sensor 130 is attached to the user's hand to detect movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, rotation speed, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove-type controller 160A. In some embodiments, for safety in real space, it is desirable that the controller 160A is attached to something that does not fly easily by being attached to the hand of the user 190, such as a glove shape. In another aspect, a controller 160B having a general structure having a plurality of operation buttons may be used. In another aspect, a sensor that is not worn by the user 190 may detect the movement of the user's 190 hand. Optical sensors that detect the position, orientation, direction of movement, distance of movement, etc. of various parts of the body of the user 190 may be used. For example, a signal from a camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190. For example, the motion sensor 130 and the computer 200 are connected to each other wirelessly. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

A computer 200 according to an embodiment of the present disclosure will be described with reference to FIG. FIG. 2 is a block diagram illustrating an example of a basic hardware configuration of the computer 200 according to an embodiment of the present disclosure. The computer 200 includes a processor 202, a memory 204, a storage 206, an input / output interface 208, and a communication interface 210 as main components. Each component is connected to the bus 212.
The processor 202 executes a series of instructions included in a program stored in the memory 204 or the storage 206 based on a signal given to the computer 200 or when a predetermined condition is satisfied. In one aspect, the processor 202 is realized as a device such as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or an FPGA (Field-Programmable Gate Array).

  The memory 204 temporarily stores programs and data. The program is loaded from the storage 206, for example. The data includes data input to the computer 200 and data generated by the processor 202. In one aspect, the memory 204 is implemented as a volatile memory such as a RAM (Random Access Memory).

  The storage 206 holds programs and data permanently. The storage 206 is realized, for example, as a nonvolatile storage device such as a ROM (Read-Only Memory), a hard disk device, or a flash memory. Programs stored in the storage 206 include a program for providing a virtual space in the system 100, a simulation program, a game program, a user authentication program, a program for realizing communication with another computer 200, and the like. Data stored in the storage 206 includes data and objects for defining the virtual space.

  In another aspect, the storage 206 may be realized as a removable storage device such as a memory card. In still another aspect, a configuration using a program and data stored in an external storage device may be used instead of the storage 206 built in the computer 200. According to such a configuration, for example, in a scene where a plurality of systems 100 are used like an amusement facility, it is possible to update programs and data in a lump.

  In some embodiments, the input / output interface 208 communicates signals between the HMD 110, the HMD sensor 120, and the motion sensor 130. In one aspect, the input / output interface 208 is realized using a terminal such as USB (Universal Serial Bus, USB), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or the like. The input / output interface 208 is not limited to the above.

  In certain embodiments, the input / output interface 208 may further communicate with the controller 160. For example, the input / output interface 208 receives signals output from the controller 160 and the motion sensor 130. In another aspect, the input / output interface 208 sends instructions output from the processor 202 to the controller 160. The command instructs the controller 160 to vibrate, output sound, emit light, and the like. When the controller 160 receives the command, the controller 160 executes vibration, sound output, light emission, and the like according to the command.

  The communication interface 210 is connected to the network 192 and communicates with other computers (for example, the server 150) connected to the network 192. In one aspect, the communication interface 210 is implemented as a wired communication interface such as a LAN (Local Area Network), or a wireless communication interface such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), or NFC (Near Field Communication). Is done. The communication interface 210 is not limited to the above.

  In an aspect, the processor 202 accesses the storage 206, loads one or more programs stored in the storage 206 into the memory 204, and executes a series of instructions included in the program. The one or more programs may include an operating system of the computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space, and the like. The processor 202 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 208. The HMD 110 displays an image on the display unit 112 based on the signal.

  In the example shown in FIG. 2, the computer 200 is provided outside the HMD 110. However, in another aspect, the computer 200 may be embedded in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including the display unit 112 may function as the computer 200.

  Further, the computer 200 may be configured to be used in common for a plurality of HMDs 110. According to such a configuration, for example, since the same virtual space can be provided to a plurality of users, each user can enjoy the same application as other users in the same virtual space.

  In an embodiment, in the system 100, a global coordinate system is preset. The global coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-rear direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, vertical direction (up-down direction), and front-rear direction in the global coordinate system are defined as an x-axis, a y-axis, and a z-axis, respectively. More specifically, in the global coordinate system, the x axis is parallel to the horizontal direction of the real space. The y axis is parallel to the vertical direction of the real space. The z axis is parallel to the front-rear direction of the real space.

  In certain aspects, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects the infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect temporal changes in the position and inclination of the HMD 110 using each value detected over time.

  The global coordinate system is parallel to the real space coordinate system. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination around the three axes of the HMD 110 in the global coordinate system. The HMD sensor 120 sets the uvw visual field coordinate system to the HMD 110 based on the inclination of the HMD 110 in the global coordinate system. The uvw visual field coordinate system set in the HMD 110 corresponds to a viewpoint coordinate system when the user 190 wearing the HMD 110 views an object in the virtual space.

  The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually showing the uvw visual field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and inclination of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 202 sets the uvw visual field coordinate system to the HMD 110 based on the detected value.

  As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system with the head (origin) of the user wearing the HMD 110 as the center (origin). More specifically, the HMD 110 includes a horizontal direction, a vertical direction, and a front-rear direction (x-axis, y-axis, and z-axis) that define the global coordinate system by an inclination around each axis of the HMD 110 in the global coordinate system. Three directions newly obtained by tilting around the axis are set as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw visual field coordinate system in the HMD 110.

  In an aspect, when the user 190 wearing the HMD 110 stands upright and is viewing the front, the processor 202 sets the uvw visual field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch direction (u-axis) and yaw direction (v-axis) of the uvw visual field coordinate system in the HMD 110. And the roll direction (w-axis).

  After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (the amount of change in inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system, respectively, as the inclination of the HMD 110. The pitch angle (θu) represents the inclination angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the inclination angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the inclination angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

  The HMD sensor 120 sets the uvw visual field coordinate system in the HMD 110 after the HMD 110 has moved to the HMD 110 based on the detected tilt angle of the HMD 110. The relationship between the HMD 110 and the uvw visual field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

  In one aspect, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between the points (for example, the distance between the points). The position in the real space may be specified as a relative position to the HMD sensor 120. Further, the processor 202 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

  The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually illustrating one aspect of expressing a virtual space 400 according to an embodiment. The virtual space 400 has a spherical shape that covers the entire 360 ° direction of the center 406. In FIG. 4, the upper half of the celestial sphere in the virtual space 400 is illustrated in order not to complicate the description. In the virtual space 400, each mesh is defined. The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system defined in the virtual space 400. The computer 200 associates each partial image constituting content (still image, moving image, etc.) that can be developed in the virtual space 400 with each corresponding mesh in the virtual space 400 so that a virtual space image that can be visually recognized by the user is obtained. A virtual space 400 to be developed is provided to the user.

  In one aspect, the virtual space 400 defines an xyz coordinate system with the center 406 as the origin. The xyz coordinate system is, for example, parallel to the global coordinate system. Since the xyz coordinate system is a kind of viewpoint coordinate system, the horizontal direction, vertical direction (vertical direction), and front-rear direction in the xyz coordinate system are defined as the x-axis, y-axis, and z-axis, respectively. Therefore, the x axis (horizontal direction) of the xyz coordinate system is parallel to the x axis of the global coordinate system, the y axis (vertical direction) of the xyz coordinate system is parallel to the y axis of the global coordinate system, and the xyz coordinate system The z-axis (front-rear direction) is parallel to the global coordinate system z-axis.

  When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 404 is disposed at the center 406 of the virtual space 400. In an aspect, the processor 202 displays an image captured by the virtual camera 404 on the display unit 112 of the HMD 110. The virtual camera 404 similarly moves in the virtual space 400 in conjunction with the movement of the HMD 110 in the real space. Thereby, changes in the position and orientation of the HMD 110 in the real space can be similarly reproduced in the virtual space 400.

  As in the case of the HMD 110, the virtual camera 404 has a uvw visual field coordinate system. The uvw visual field coordinate system of the virtual camera 404 in the virtual space 400 is defined so as to be linked to the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 404 changes accordingly. The virtual camera 404 can also move in the virtual space 400 in conjunction with the movement of the user wearing the HMD 110 in the real space.

  The processor 202 of the computer 200 defines the visual field area 410 in the virtual space 400 based on the arrangement position of the virtual camera 404 and the reference line of sight 408. The visual field area 410 corresponds to an area of the virtual space 400 that is visually recognized by the user wearing the HMD 110.

  The line of sight of the user 190 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 190 visually recognizes the object. The uvw visual field coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the display unit 112. Further, the uvw visual field coordinate system of the virtual camera 404 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, the system 100 according to an aspect can regard the line of sight of the user 190 detected by the gaze sensor 140 as the line of sight of the user in the uvw visual field coordinate system of the virtual camera 404.

  The determination of the user's line of sight will be described with reference to FIG. FIG. 5 is a top view of the head of a user 190 wearing the HMD 110 according to an embodiment.

  In one aspect, the gaze sensor 140 detects each line of sight of the user's 190 right eye and left eye. In an aspect, when the user 190 is looking nearby, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

  When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 specifies the gazing point N1 that is the intersection of the lines of sight R1 and L1. On the other hand, when the detected values of the lines of sight R2 and L2 are received from the gaze sensor 140, the computer 200 specifies the intersection of the lines of sight R2 and L2 as the point of sight. The computer 200 specifies the line of sight N0 of the user 190 based on the specified position of the gazing point N1. For example, the computer 200 detects, as the line of sight N0, the extending direction of the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1. The line of sight N0 is a direction in which the user 190 is actually pointing the line of sight with both eyes. The line of sight N0 corresponds to the direction in which the user 190 is actually pointing the line of sight with respect to the field of view area 410.

  In another aspect, the system 100 may include a microphone and a speaker in any part of the system 100. The user can give a voice instruction to the virtual space 400 by speaking to the microphone.

  In another aspect, system 100 may include a television broadcast receiving tuner. According to such a configuration, the system 100 can display a television program in the virtual space 400.

  In yet another aspect, the system 100 may include a communication circuit for connecting to the Internet or a call function for connecting to a telephone line.

  The visual field area 410 will be described with reference to FIGS. FIG. 6 is a diagram illustrating a yz cross section of the visual field area 410 viewed from the x direction in the virtual space 400. FIG. 7 is a diagram illustrating an xz cross section of the visual field area 410 viewed from the y direction in the virtual space 400.

  As shown in FIG. 6, the field-of-view region 410 in the yz section includes a region 602. The area 602 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the yz section of the virtual space 400. The processor 202 defines a range including the polar angle α around the reference line of sight 408 in the virtual space as a region 602.

  As shown in FIG. 7, the field-of-view region 410 in the xz section includes a region 702. The area 702 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the xz cross section of the virtual space 400. The processor 202 defines a range including the azimuth angle β around the reference line of sight 408 in the virtual space 400 as a region 702. The polar angles α and β are determined according to the arrangement position of the virtual camera 404 and the orientation of the virtual camera 404.

  In an aspect, the system 100 provides the user 190 with a field of view in the virtual space by causing the display unit 112 to display a field of view image based on a signal from the computer 200. The view image corresponds to a portion of the virtual space image 402 that is superimposed on the view region 410. When the user 190 moves the HMD 110 worn on the head, the virtual camera 404 moves in conjunction with the movement. As a result, the position of the visual field area 410 in the virtual space 400 changes. As a result, the view image displayed on the display unit 112 is updated to an image that is superimposed on the view region 410 in the direction in which the user faces in the virtual space 400 in the virtual space image 402. The user can visually recognize a desired direction in the virtual space 400.

  Thus, the orientation (tilt) of the virtual camera 404 corresponds to the user's line of sight (reference line of sight 408) in the virtual space 400, and the position where the virtual camera 404 is arranged corresponds to the user's viewpoint in the virtual space 400. Therefore, by moving the virtual camera 404 (including the operation of changing the arrangement position and the operation of changing the orientation), the image displayed on the display unit 112 is updated, and the field of view (including the viewpoint and the line of sight) of the user 190 is moved. Is done.

  While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 402 developed in the virtual space 400 without visually recognizing the real world. Therefore, the system 100 can give the user a high sense of immersion in the virtual space 400.

  In an aspect, the processor 202 may move the virtual camera 404 in the virtual space 400 in conjunction with movement of the user 190 wearing the HMD 110 in real space. In this case, the processor 202 specifies an image region (that is, a view region 410 in the virtual space 400) projected on the display unit 112 of the HMD 110 based on the position and orientation of the virtual camera 404 in the virtual space 400.

  According to an embodiment, the virtual camera 404 may include two virtual cameras: a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Also, appropriate parallax may be set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 400.

  A controller 160A, which is an example of the controller 160, will be described with reference to FIG. 8A. FIG. 8A is a diagram illustrating a schematic configuration of a controller 160A according to an embodiment.

  In certain aspects, the controller 160A may include a right controller and a left controller. For ease of explanation, the controller 160A in FIG. 8A represents the right controller. The right controller is operated with the right hand of the user 190. The left controller is operated with the left hand of the user 190. In one aspect, the right controller and the left controller are configured symmetrically as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller and the left hand holding the left controller. In another aspect, the controller 160 may be an integrated controller that accepts operations of both hands. Hereinafter, the right controller 160A will be described.

  The right controller 160A includes a grip 802, a frame 804, and a top surface 806. The grip 802 is configured to be gripped by the right hand of the user 190. For example, the grip 802 can be held by the palm of the right hand of the user 190 and three fingers (middle finger, ring finger, little finger).

  The grip 802 includes buttons 808 and 810 and a motion sensor 130. The button 808 is disposed on the side surface of the grip 802 and receives an operation with the middle finger of the right hand. The button 810 is disposed on the front surface of the grip 802 and receives an operation with the index finger of the right hand. In some embodiments, the buttons 808, 810 are configured as trigger buttons. The motion sensor 130 is built in the housing of the grip 802. Note that the grip 802 may not include the motion sensor 130 when the operation of the user 190 can be detected from around the user 190 by a camera or other device.

  The frame 804 includes a plurality of infrared LEDs 812 arranged along the circumferential direction thereof. The infrared LED 812 emits infrared rays in accordance with the progress of the program during execution of the program using the controller 160A. Infrared rays emitted from the infrared LED 812 can be used to detect the positions and postures (tilt and orientation) of the right controller 160A and the left controller (not shown). In the example shown in FIG. 8, infrared LEDs 812 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. 8. An array of one column or three or more columns may be used.

  The top surface 806 includes buttons 814 and 816 and an analog stick 818. Buttons 814 and 816 are configured as push buttons. Buttons 814 and 816 receive an operation with the thumb of the right hand of user 190. In a certain aspect, the analog stick 818 accepts an operation in an arbitrary direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object arranged in the virtual space 400.

  In an aspect, the right controller 160A and the left controller include a battery for driving a member such as the infrared LED 812. The battery may be either a primary battery or a secondary battery, and the shape thereof may be arbitrary, such as a button type or a dry battery type. In another aspect, the right controller 160A and the left controller may be connected to a USB interface of the computer 200, for example. In this case, the right controller 800 and the left controller may be supplied with power via the USB interface.

  A controller 160B, which is an example of the controller 160, will be described with reference to FIG. 8B. FIG. 8B is a diagram illustrating a schematic configuration of a controller 160B according to an embodiment.

  The controller 160B includes a plurality of buttons 820 (820a, 820b, 820c, 820d) and 822 (822a, 822b, 822c, 822d), and left and right analog sticks 824L and 824R. Each button 820 and 822 is configured as a push button. Buttons 820 and 822 accept an operation with the thumb of the hand of user 190. A trigger-type button (not shown) that can accept an operation with an index finger or a middle finger of the user 190 may be further provided in the controller 160B. The analog sticks 824L and 824R each accept an operation in an arbitrary direction of 360 degrees from the initial position (neutral position) in a certain aspect. The operation includes, for example, an operation for moving an object arranged in the virtual space 400. Buttons 820 (820a, 820b, 820c, 820d) and 822 (822a, 822b, 822c, 822d) and analog sticks 824L and 824R (and, if included, trigger buttons not shown) have separate operations. A command is assigned. The operation command is, for example, a command for giving a command to an object in the virtual space 400, a command for making various settings on a menu screen of a game, and the computer when the user 190 is experiencing the virtual space 400. Any other command that can be entered into 200 is included. The operation command assigned to each button or analog stick may be dynamically changed according to, for example, the progress of the game or the change of the scene.

  In one aspect, the controller 160B may include a plurality of infrared LEDs (not shown) disposed on the outer surface thereof. The infrared LED emits infrared light in accordance with the progress of the program during execution of the program using the controller 160B. Infrared light emitted from the infrared LED can be used to detect the position and orientation (tilt, orientation) of the controller 160B. The controller 160B includes a battery for driving internal electronic components. The battery may be either a primary battery or a secondary battery, and the shape thereof may be arbitrary, such as a button type or a dry battery type. In another aspect, the controller 160B may be connected to a USB interface of the computer 200, for example. In this case, the controller 160B can be powered through the USB interface.

.
FIG. 9 is a block diagram illustrating the functionality of a computer 200 for providing a virtual experience to a user via the system 100 according to one embodiment of the present disclosure. As already mentioned, various image display devices for providing a virtual experience to a user can be used in embodiments of the present disclosure. In the embodiments described below, an HMD is used as an example of an image display device.

  The computer 200 executes various processes mainly based on inputs from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, and the controller 160.

  The computer 200 includes a processor 202, a memory 204, and a communication control unit 205. The processor 202 includes a virtual space specifying unit 902, an HMD motion detection unit 904, a line-of-sight detection unit 906, a reference line-of-sight determination unit 908, a visual field region determination unit 910, a virtual viewpoint specification unit 912, and a visual field image generation unit 914. And a virtual camera control unit 916, a game departure control unit 918, and a view field image output unit 924. Memory 204 may be configured to store various information. In one example, the memory 204 may include virtual space data 926, object data 928, application data 930, removal determination data 932, alert data 934, other data 938, and the like. The memory 204 also includes various data necessary for calculation to provide output information corresponding to inputs from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, the controller 160, etc. to the display unit 112 associated with the HMD 110. But you can. The object data 928 may include data related to various objects arranged in the virtual space. The removal determination data 932 may include various data used to determine the removal operation of the image display device (HMD or the like) by the user. The display unit 112 may be built in the HMD 110 or may be a display of another device (for example, a smartphone) that can be attached to the HMD 110.

  In FIG. 9, the components included in the processor 202 are just one example of expressing the function executed by the processor 202 as a specific module. The functions of multiple components may be realized by a single component. The processor 202 may be configured to perform the functions of all components.

  FIG. 10 is a flowchart of a general process for displaying an image of the virtual space in which the user is immersed on the display unit 112.

  A general process of the system 100 for providing an image of the virtual space will be described with reference to FIGS. 9 and 10. The virtual space 400 may be provided by the interaction of the HMD sensor 120, the gaze sensor 140, the computer 200, and the like.

  Processing begins at step 1002. As an example, a game application included in the application data 930 may be executed by the computer 200. In step 1004, the processor 202 (or the virtual space specifying unit 902) refers to the virtual space data 926 and generates a celestial spherical virtual space image 402 constituting the virtual space 400 in which the user is immersed. The position and inclination of the HMD 110 are detected by the HMD sensor 120. Information detected by the HMD sensor 120 is transmitted to the computer 200. In step 1006, the HMD motion detection unit 904 acquires position information, tilt information, and the like of the HMD 110. In step 1008, the viewing direction is determined based on the acquired position information and tilt information.

  When the gaze sensor 140 detects the movement of the left and right eyeballs of the user, the information is transmitted to the computer 200. In step 1010, the line-of-sight detection unit 906 identifies the direction in which the line of sight of the right eye and the left eye is directed, and determines the line-of-sight direction N0. In step 1012, the reference visual line determination unit 908 determines the visual field direction determined by the inclination of the HMD 110 or the user's visual line direction N0 as the reference visual line 408. The reference line of sight 408 may also be determined based on the position and tilt of the virtual camera 404 that follows the position and tilt of the HMD 110.

  In step 1014, the view area determination unit 910 determines the view area 410 of the virtual camera 404 in the virtual space 400. As shown in FIG. 4, the visual field area 410 is a part of the virtual space image 402 that constitutes the visual field of the user. The field of view area 410 is determined based on the reference line of sight 408. A yz sectional view of the visual field region 410 viewed from the x direction and an xz sectional view of the visual field region 410 viewed from the y direction are shown in FIGS. 6 and 7 described above, respectively.

  In step 1016, the visual field image generation unit 914 generates a visual field image based on the visual field region 410. The field-of-view image includes two two-dimensional images for the right eye and the left eye. These two-dimensional images are superimposed on the display unit 112 (more specifically, the right-eye image is output to the right-eye display unit, and the left-eye image is output to the left-eye display unit). A virtual space 400 as an image is provided to the user. In step 1018, the view image output unit 924 outputs information related to the view image to the display unit 112. The display unit 112 displays the view image based on the received view image information. The process ends at step 1020.

  FIG. 11 is a flowchart of a game departure control method 1100 according to an embodiment of the present disclosure. In an embodiment of the present disclosure, a computer program may cause the processor 202 (or the computer 200) to execute the steps illustrated in FIG. Another embodiment of the present disclosure may also be implemented as a computer that includes at least a processor and executes the method 1100 under the control of the processor.

  Hereinafter, embodiments of the present disclosure will be specifically described. Here, as a specific example to which the embodiment of the present disclosure can be applied, a game is assumed in which a user can immerse and enjoy a virtual space in which an avatar and other objects associated with the user are arranged. . However, embodiments of the present disclosure are not necessarily limited to such an aspect. It will be apparent to those skilled in the art that the embodiments of the present disclosure can take various forms that fall within the scope of the claims.

  FIG. 12 is a schematic diagram illustrating a situation when a plurality of users, for example, two users, user A and user B, are performing a match play of an online game. Note that the number of users participating in the game is not limited to two. The embodiment of the present disclosure can also be applied to a game in which a larger number of users can participate. A system 100 shown in FIG. 9 is installed at each location of the user A and the user B, and these systems are connected to each other via a network 192 and a server 150. In order to simplify the description, in FIG. 12, the HMD 1202 such as the above-described HMD 110 worn by the user A, the controller 1204 such as the above-described controller (movable device related to the game) held in the right hand, and the HMD 1202 Only a personal computer (PC) 1208 on a desk 1206 to which a controller 1204 is connected is shown. However, it will be apparent that other elements shown in FIG. 1 or FIG. 9 may be provided. User B also has a similar device, but the device is omitted for simplicity. Hereinafter, an embodiment of the present disclosure will be described assuming a situation in which the user A and the user B illustrated in FIG. 12 are performing online match play.

  Returning to FIG. 11, the process first proceeds to step 1104. In step 104, the game withdrawal control unit 918 determines whether or not the user A is playing an online match in which other users also participate based on information on the currently played game obtained from the application data 930 or the like. judge. If it is determined in step 1104 that the user A is not playing (“N” in step 1104), the process proceeds to step 1122 and ends. For example, this process is executed when the user A is not in online match play, or when the game being played by the user A or the opponent user B is normally terminated.

  On the other hand, if user A is currently playing (“Y” in step 1104), the process proceeds to step 1106. In step 1106, the game withdrawal control unit 918 determines whether or not the user A has removed the HMD 1202 or the like. Details of detection of the removal operation of the HMD 1202 and the like will be described later with reference to FIGS. If the removal operation is not detected (“N” in step 1106), the process returns to before step 1104. On the other hand, when the removal operation is detected (“Y” in step 1106), the process proceeds to step 1108. In step 1108, the game departure control unit 918 generates an alert for the user A as an output for prompting an input related to returning to the game by referring to the alert data 934 or the like. The generation of this alert may include the generation of a user interface. The alert data 934 may store data such as the content, form, and location of the generated alert. The alert can be generated in the form of display, sound, vibration, light, etc., can be generated in a plurality of forms, or can be generated in duplicate. A specific example of the alert will be described later with reference to FIGS.

  The process proceeds to step 1110, and the game withdrawal control unit 918 determines whether or not the user A has selected to return to play in response to an output prompting an input related to returning to the game. If it is determined that the return to play has been selected (“Y” in step 1110), the process returns to step 1104. If return to play is not selected (“N” in step 1110), the process proceeds to step 1112.

  In step 1112, the game withdrawal control unit 918 determines whether it is selected not to return to play, that is, to end normally. If normal termination is selected (“Y” in step 1112), the process proceeds to step 1114. In step 1114, normal termination processing is performed. In one example, as a result of the user A selecting to leave the match play, the game may be terminated after the user A loses and the opponent user B wins. In a game in which a plurality of users participate, how to determine the game win / loss when some users leave may be arbitrarily determined. For example, even when the user A has a higher score than the user B, when the user A leaves, the user A may be determined to lose.

  When not returning to play (normal end) is not selected (“N” in step 1112), the process proceeds to step 1116. In step 1116, the game withdrawal control unit 918 determines whether the communication connection necessary for playing the game has been disconnected. For example, when the communication cable for connection with the network is disconnected, it is determined that the communication connection is disconnected. If it is determined that the communication connection has been disconnected (“Y” in step 1116), the process proceeds to step 1118. In step 1118, the game leaving control unit 918 performs the illegal leaving process and ends the process. In an example of the illegal leaving process, in addition to the disadvantages imposed on the user A in the normal end process described above for the user A determined not to end the match play normally, restrictions on future game participation, Penalties and other disadvantages may be imposed. What disadvantages are imposed on the user A in the unauthorized leaving process can be arbitrarily determined according to the type of game, the influence of the opponent, and the like.

  On the other hand, if it is determined that the communication connection has not been disconnected (“N” in step 1116), the process proceeds to step 1120. In step 1120, the game departure control unit 918 determines whether a predetermined time has elapsed. The predetermined time may be set, for example, for the time that has elapsed since the removal operation was detected in step 1106. Alternatively, the predetermined time may be set for the time that has elapsed since the alert was generated in step 1108. Alternatively, the predetermined time may be set for the time elapsed from the time point associated with step 1110, 1112 or 1116. These predetermined times may correspond to the time when the user B who is the opponent is waiting for some response from the user A. Therefore, it is not preferable that the predetermined time is too long. As an example, the predetermined time may be set to 30 seconds. Depending on the type of game, the predetermined time may be a longer time or a shorter time. By providing such a time limit, it is possible to prevent the opponent player from being inconvenienced by waiting indefinitely.

  If the predetermined time has not elapsed ("N" in step 1120), the process returns to before step 1110, whether or not return to play is selected, whether or not return to play is selected, and communication is performed. The determination as to whether or not the connection has been disconnected or whether or not a predetermined time has elapsed is repeatedly executed. The process regarding each determination is as described above. If it is determined in step 1120 that the predetermined time has elapsed (“Y” in step 1120), the process proceeds to step 1118, the above-described unauthorized leaving process is executed, and this flow ends.

  Next, a specific example of detection of the removal operation of the HMD or the like in step 1106 of FIG. 11 will be described with reference to FIGS.

  First, an example of detection of an HMD removal operation based on removal of the HMD or releasing the controller will be described with reference to FIG. As shown in the figure, a situation is considered in which the user A removes the HMD 1202 from the user's head and places it on a nearby desk. Here, the HMD placed on the desk is indicated as HMD1302. In this case, the movement of the HMD 1302 is almost or completely stationary. Therefore, by determining whether or not the distance that the HMD moves is equal to or less than the minimum distance, it can be determined whether or not the HMD is in a completely stationary state or a state close thereto. Thereby, it is possible to detect whether or not the user A has removed the HMD 1202. The minimum distance may be stored in advance in the removal determination data 932 for the HMD and the controller. While the user wears the HMD 1202 and plays a game, the HMD moves somewhat. Therefore, by using the minimum distance as a threshold, it can be determined whether or not the HMD is in a completely stationary state or a state close thereto. The position and movement of the HMD 1202 can be detected using the position tracking function of the HMD sensor 120 described above or the HMD position and tilt detection function of the sensor 114. In the case of an HMD without such a function, a sensor such as a gyro sensor may be added. Further, in FIG. 13, the desk is shown as the place where the HMD is placed, but the embodiment of the present disclosure can be similarly applied when the HMD is placed in any other place such as a floor or a chair.

  FIG. 13 also shows a situation where the controller 1204 held by the user A is placed on the desk, and the placed controller is shown as a controller 1304. Also in this case, the controller 1304 is in a completely stationary state or a state close thereto. While the user plays the game with the controller in hand, some movement occurs. Therefore, if it is determined that the controller is in a fully stationary state or a state close thereto, the user has released the controller and placed it somewhere (ie, a removal operation has been performed). The position, movement, etc. of the controller can be detected by, for example, the motion sensor 130 described above. Some users place the controller first before removing the HMD. In such a case, the removal operation by the user can be detected earlier than the processing based on the movement of the HMD by performing the processing based on the stationary state of the controller.

  Next, as another example of the process based on the movement of the HMD, the detection of the HMD removal operation using the locus of the movement of the HMD during the removal operation will be described with reference to FIG. As shown, when user A attempts to remove the HMD 1404 worn on the head 1402, the user A first lifts the HMD 1404 upward along a trajectory 1406 and then somewhere else from the lifted position. The HMD 1404 is lowered downward along the trajectory 1408 for placement (eg, on a desk). In such a case, the trajectory 1406 and the trajectory 1408 are parabolic curves. Therefore, when the movement of the HMD along such a parabolic trajectory is detected, it can be determined that the HMD removal operation is being performed. For comparison with such a case, considering a situation where the user A simply faces upward, the HMD follows a trajectory 1410, for example. Since the distance of the movement of the locus 1410 is short and the amount of movement in the vertical direction is small, it can be seen that it is quite different from the parabolic movement such as the locus 1406 or the locus 1408. Therefore, it is possible to distinguish between an operation of removing the HMD by the user and an operation in which the user simply faces upward. According to this detection method, the removal operation can be detected at a stage before the HMD is placed somewhere and becomes completely stationary.

  As another example, the removal operation of the HMD may be detected by using an unnatural movement of the HMD that is unlikely to occur during game play. For example, such unnatural movement may be stored in the removal determination data 932 as data in advance. By determining whether the detected movement of the HMD is the same as or similar to the movement stored in advance, the removal operation can be detected.

  Next, HMD removal operation detection using the attachment / detachment sensor will be described with reference to FIG. In this example, the HMD 1504 attached to the head 1502 of the user A is provided with an attachment / detachment sensor 1506 schematically indicated by a dotted line. By using the detachable sensor 1506 to determine that the HMD 1504 has been removed from the head when the user A's forehead is no longer detected, the detachment operation can be detected. The detachable sensor 1506 may be, for example, a sensor that uses infrared rays, a mechanical switch type sensor, or any other type of sensor.

  Next, an example in which an HMD removal operation is detected based on the position of the HMD or the like will be described with reference to FIG. Here, the position includes the direction of the HMD or the like. As shown, the HMD 1602 is assumed to be placed face down. In this case, the occurrence of the removal operation can be detected by determining that the HMD 1602 is in a removed state when the downward state of the HMD 1602 continues for a certain period of time or longer. The same processing can be performed when the HMD 1602 is in an upward state or in another direction for a certain period or longer. It is almost impossible in game play that the direction of the HMD remains the same for a certain period of time. Therefore, when such a situation occurs, it can be determined that the HMD is placed on a desk, floor, or the like. Similarly, it is determined that the controller 1604 has been released from the user's hand and placed on a desk or the like when a certain period of time has elapsed while the controller 1604 remains in a specific orientation (for example, landscape orientation). May be.

  In the above, the example which detects the HMD removal operation based on one of the movement and position of the HMD or the like has been described. However, the embodiment of the present disclosure is not limited to such an aspect. The detection of the removal operation may be performed based on both the movement and the position of the HMD or the like. For example, a removal operation based on a movement from a predetermined position due to a predetermined movement, a movement to a predetermined position due to a predetermined movement, a predetermined movement occurring at a predetermined position, etc. May be detected.

  With reference to FIG. 17A and FIG. 17B, an example in which HMD removal operation detection is performed based on both the movement and position of the HMD or the like will be described. In FIG. 17A and FIG. 17B, the direction of the HMD is indicated by a thick white arrow for simplification. As shown, the HMD 1702 of FIG. 17A is moved more than a certain distance in the upward direction in a forward-facing state. This movement can be interpreted as a movement to remove the HMD 1702 overhead. On the other hand, the HMD 1704 in FIG. 17B is moved a certain distance or more downward in the downward state. This movement can be interpreted as a movement of placing the HMD 1704 on the floor or the like. Based on the interpretation of the movement of the HMD as described above, the removal operation of the HMD may be detected.

  In the above, an example in which the removal operation detection of the HMD or the like is performed on the assumption that the HMD or the like is removed from the head or placed on a desk has been described. As another example, if tracking of HMD movement is continued by performing position tracking of HMD, data relating to movement of HMD is accumulated. Then, it is possible to acquire teacher data indicating what kind of movement the user is performing the removal operation and use it for machine learning using artificial intelligence (AI). In that case, it may be possible to detect the removal operation based on the movement and position that were not assumed in advance.

  Next, with reference to FIG. 18A and FIG. 18B, an example in which an HMD removal operation is detected based on at least one of the position and movement of the user's head and eyes will be described. Since the movement and position of the user's head can be specified as the position and movement of the HMD when the HMD is worn, processing based on the position and movement of the eyes will be particularly described here. As shown in FIGS. 18A and 18B, the HMD 1802 includes a gaze sensor 1804 (shown schematically) such as the gaze sensor 140 described above, and performs eye tracking of the user's eyes. In FIG. 18A, it is assumed that the user wears the HMD 1802 on the head. At this time, the eyes of the user can be tracked by the gaze sensor 1804. On the other hand, in FIG. 18B, when the user lifts the HMD above the head to remove the HMD from the head, the gaze sensor 1804 cannot track the user's eyes. Thus, when the position and / or movement of the eyes cannot be detected, it may be determined that the removal operation has been performed. Furthermore, it may be determined that the removal operation has been performed when an unnatural eye movement that cannot normally occur in game play is detected. Data used for such determination may be stored in the removal determination data 932.

  Next, with reference to FIG. 19A and FIG. 19B, an example of detecting the HMD removal operation based on the relative positional relationship between the HMD and the controller will be described. FIG. 19A shows a state where the user A wearing the HMD 1902 has the controller 1904 in his hand. At this time, the maximum distance D1 between the HMD 1902 and the controller 1904 is substantially determined by the length of the hand of the user A. In this situation, when the user A puts the controller 1904 on a desk a little away in preparation for removing the HMD 1902, the distance D2 between the controller 1906 and the HMD 1902 placed on the desk is larger than the maximum distance of D1. growing. Therefore, when the positional relationship between the controller and the HMD satisfies a predetermined condition (in this example, the distance D2 is larger than the maximum value of the distance D1), the HMD removal operation is performed (or performed). Can be determined). The same process can be performed when the controller is placed not at the desk but at the position indicated by reference numeral 1908 and the distance D3 between the controller 1908 and the HMD 1902 is equal to or greater than the maximum distance of D1.

  On the other hand, as shown in FIG. 19B, when the user A moves the controller 1904 to the position indicated by the reference number 1910 and approaches the HMD 1902, the distance between the controller 1910 and the HMD 1902 becomes a certain distance or less. This state can be interpreted as a state in which the user A tries to remove the HMD 1902 and the hand holding the controller is brought close to the HMD 1902. Therefore, it may be determined that the removal operation of the HMD has been performed when a predetermined condition is satisfied such that the distance between the controller and the HMD is equal to or less than the predetermined distance. Alternatively, when it is not necessary to bring the controller close to the HMD due to the nature of the game, it may be determined that the removal operation of the HMD has occurred when the controller is close to the HMD for a certain time or more.

  In FIG. 19A and FIG. 19B, the determination based on whether the positional relationship between the controller and the HMD satisfies a predetermined condition has been described. As another example, a similar determination can be made based on whether the positional relationship between the controller and the user's head satisfies a predetermined condition.

  Next, details of alert generation in step 1108 of FIG. 11 will be described with reference to FIGS.

  FIG. 20 illustrates an example of an alert displayed on the display unit 112 (see FIG. 9) of the HMD 110. The alert 2002 is displayed with detection of the removal operation of the HMD or the like as a trigger, and provides a user interface (UI) for receiving an input from the user. In this example, the alert 2002 includes a “Yes” button 2006 for return selection together with a text display 2004 “Do you want to return to play?”. When selecting return to play in step 1110 in FIG. 11, the user may operate the “Yes” button 2006. The alert 2002 includes a text display 2008 “Do you want to return to play (select normal end)?” And a “Yes” button 2010 for selecting normal end, that is, normal departure. When selecting normal end in step 1112 of FIG. 11, the user may operate the “Yes” button 2010. In addition, the alert 2002 may include a warning statement 2012 for the user, “Please note that if it does not end normally, it will result in illegal withdrawal and a disadvantage for future game play”. The warning message can prompt the user to return to the game or finish the game normally. As another example, the alert 2002 may include a warning message such as “Return within 30 seconds. If not returned, retired”.

  The removal operation detection method described with reference to FIGS. 13 to 19 is a method that can detect the removal operation before removing the HMD or during removal of the HMD (in terms of timing, an alert 2002 displayed on the display unit 112 of the HMD 110 is displayed. A case in which the user can visually recognize) and a method in which the removal operation cannot be detected unless the removal of the HMD is completed (when the alert 2002 displayed on the display unit 112 of the HMD 110 is not visible to the user). obtain.

  FIG. 21 illustrates a duplicate alert according to an embodiment of the present disclosure that can be applied to the techniques described in FIGS. Since the alert 2002 shown in FIG. 20 is displayed on the display unit 112 of the HMD 110, the user cannot see the alert 2002 after removing the HMD. Therefore, during the removal operation of the HMD or the like, an additional alert may be generated in addition to the alert 2002 in order to generate an alert before the removal operation is completed to attract the user's attention more reliably. For example, as shown, text or other types of alerts 2106 may be displayed on the display of the PC 2104 running the VR game placed on the desk 2102 of the user A. This additional alert 2106 may include a warning text such as the warning text 2012 shown in FIG. In another example, the additional alert may be to vibrate the controller 2108 placed on the desk 2102. Alternatively, the additional alert may be to light or blink the lamp 2112 provided on the removed HMD 2110. Further, the additional alert may be to generate an alarm sound and emit the alarm sound via the headphones 2114 used by the user. When a speaker is used instead of the headphones, an alarm sound may be emitted through the speaker.

  By performing the duplicate alert of FIG. 21, even if the removal operation cannot be detected unless the removal of the HMD is completed, the user notices the alert. I can urge you to. Even if the removal operation can be detected before removing the HMD or during removal of the HMD, the user does not always notice the alert, so it is useful to perform the duplicate alert of FIG.

  As mentioned above, although embodiment of this indication was described, it should be understood that these are only illustrations and do not limit the scope of this indication. It should be understood that changes, additions, improvements, and the like of the embodiments can be made as appropriate without departing from the spirit and scope of the present disclosure. The scope of the present disclosure should not be limited by any of the above-described embodiments, but should be defined by the claims and their equivalents.

  DESCRIPTION OF SYMBOLS 100 ... HMD system, 112 ... Display part, 114 ... Sensor, 120 ... HMD sensor, 130 ... Motion sensor, 140 ... Gaze sensor, 150 ... Server, 160A, 160B ... Controller, 190 ... Head, 192 ... Network, 200 ... Computer, 202 ... Processor, 204 ... Memory, 205 ... Communication control unit, 206 ... Storage, 208 ... I / O interface, 210 ... Communication interface, 212 ... Bus, 400 ... Virtual space, 402 ... Virtual space image, 404 ... Virtual camera 406 ... center, 408 ... reference line of sight, 410 ... field of view, 800 ... right controller, 802 ... grip, 804 ... frame, 806 ... top, 808,810,814,816 ... button, 812 ... infrared LED, 818 ... Analog stick, 820, 22 ... button, 824R, 824L ... analog stick, 902 ... virtual space specifying unit, 904 ... HMD motion detecting unit, 906 ... gaze detecting unit, 908 ... reference gaze determining unit, 910 ... visual field region determining unit, 912 ... virtual viewpoint specifying Unit 914 ... view image generation unit 916 ... virtual camera control unit 918 ... game leaving control unit 924 ... view image output unit 926 ... virtual space data, 928 ... object data, 930 ... application data, 932 ... removal determination Data, 934 ... alert data

Claims (11)

A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user.
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
A step of causing the image display device to output a UI (User Interface) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected;
A program that executes The step of detecting the removal operation includes:
Detecting a first part included in the head of the first user;
The program according to claim 1, further comprising: determining that the removal operation has been performed when the first part cannot be detected. The step of detecting the removal operation includes:
Tracking the image display device;
The program according to claim 1, further comprising: determining that the removal operation has been performed when the image display device is stationary or when the image display device moves along a predetermined trajectory. The step of detecting the removal operation includes:
Tracking an operating device used by the first user;
The step of determining that the removal operation has been performed when the operation device is stationary or when a positional relationship between the operation device and the image display apparatus satisfies a predetermined condition. The listed program. In the computer,
Receiving an input to end the virtual experience;
Detaching the first user from the virtual experience in response to receiving an input to end the virtual experience;
The program according to any one of claims 1 to 4, wherein the program is executed. In the computer,
Receiving an input that does not end the virtual experience;
Returning the first user to the virtual experience in response to receiving an input that does not end the virtual experience;
The program according to any one of claims 1 to 5, wherein the program is executed. In the computer,
The step of determining that the first user has illegally left the virtual experience in response to receiving neither an input that ends the virtual experience nor an input that does not end the virtual experience is performed. The program according to any one of 6 to 6.   The UI includes a warning that the first user is deemed to have left the virtual experience if the first user does not return to the virtual experience and does not leave the virtual experience. 8. The program according to any one of items 7. And a step of causing at least one of an audio output device and an image output device associated with the image display device to perform an output for encouraging normal departure from the virtual experience when the removal operation is detected. Item 9. The program according to any one of Items 1 to 8. A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user,
By control by a processor included in the computer,
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
And a step of causing the image display device to output a UI (User Interface) for prompting an input as to whether or not to end the virtual experience when the removal operation is detected. Executed by a computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user. It is a method,
Defining the virtual space;
Providing the virtual space to the first user via the image display device;
Detecting a removal operation in which the first user removes the image display device;
Causing the image display device to output a UI (User Interface) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected.