JP2020064652A - Program, computer and method for providing user with virtual experience via image display device - Google Patents

Abstract

To provide a novel technique for suppressing withdrawal from a game not intended by a user in the game in which a plurality of users participate.SOLUTION: A program according to an embodiment of the present disclosure causes a computer for providing users with a virtual experience in a game in which a plurality of the users participate via respective image display devices to execute steps of: defining a virtual space for providing the virtual experience; providing the users with the virtual space via the respective image display devices; detecting a removal action of the user to remove the image display device associated with a head of the user; and generating output that prompts the user whose removal action is detected, to perform input related to returning to the game.SELECTED DRAWING: Figure 11

Description

  The present disclosure relates to a program, a computer and a method for providing a virtual experience to a user via an image display device. More specifically, the present disclosure relates to a program, a computer, and a method for executing a process related to leaving a user in a game.

  Patent Document 1 discloses a method in which, in a game in which a plurality of users participate, each user who participates in the game evaluates manners in the game of other users who participate in the game. The game described in Patent Document 1 is not a game to which a virtual reality (VR) technique is applied. According to the method of Patent Document 1, each user is restricted from participating in the game or suffers a disadvantage in the game based on the manner evaluation result for itself.

  The method of Patent Document 1 indirectly restricts the user from performing an act of bad manners such as illegal leaving by limiting the user's participation in the game or giving a disadvantage. Here, "illegal departure" means that the user disconnects the communication connection of the game terminal or the home screen of the game in order to end the game in the middle when the situation of the game is disadvantageous to the user. To return to.

  According to the method of Patent Document 1, when a certain user makes an unauthorized departure, another user cannot determine whether this unauthorized departure is intentional or not. Therefore, both intentional and unintentional breaches receive the same manner evaluation. In this case, it is not possible to prevent an unintended illegitimate departure, for example, an unintended illegitimate departure of a user who is not familiar with the game without restricting participation in the game or giving a disadvantage.

Japanese Patent No. 5679227

  An object of the present disclosure is to provide a novel method for suppressing unauthorized departure of a user in a virtual experience in a virtual space in which a plurality of users participate.

According to an embodiment of the present disclosure, the first user experiences a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user. To the computer for delivery to
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
Output a UI (User Interface) to the image display device to prompt input of whether or not to end the virtual experience when the removal operation is detected,
A program is provided to execute.

According to another embodiment of the present disclosure, the first virtual experience in a virtual space in which a plurality of users including the first user participate is performed via the image display device associated with the head of the first user. A computer for providing to a user,
By the control of the processor included in the computer,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
And a step of causing the image display device to output a UI (User Interface) that prompts an input as to whether or not to end the virtual experience when the removal operation is detected.

According to another embodiment of the present disclosure, the first virtual experience in a virtual space in which a plurality of users including the first user participate is performed via the image display device associated with the head of the first user. A computer implemented method for providing to a user, comprising:
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
Outputting a UI (User Interface) to the image display device for prompting an input as to whether to end the virtual experience when the removal operation is detected.

  According to the embodiments of the present disclosure, it is possible to provide a novel method for suppressing a departure from a game that is not intended by users in a game in which a plurality of users participate in play.

  Other features and advantages of the present disclosure will be apparent from the description of the embodiments below, the accompanying drawings and the claims.

It is a figure which shows the structure of a HMD system roughly. FIG. 16 is a block diagram illustrating an example of a basic hardware configuration of a computer according to an embodiment of the present disclosure. FIG. 6 is a diagram conceptually showing a uvw visual field coordinate system set in an HMD according to one embodiment. It is a figure which represents notionally one aspect which represents virtual space according to one embodiment. FIG. 3 is a diagram showing the head of the user wearing the HMD from above, according to one embodiment. It is a figure showing the yz section which looked at the visual recognition region from the x direction in virtual space. It is a figure showing the xz section which looked at the visual recognition field from the y direction in virtual space. It is a figure showing the schematic structure of the controller according to one embodiment. It is a figure showing the schematic structure of the controller according to one embodiment. FIG. 6 is a block diagram illustrating functions of a computer including a game departure control function for providing a virtual experience to a user via an HMD system according to an embodiment of the present disclosure. It is a flowchart of a general process for displaying an image of a virtual space in which a user is immersed in a display unit. 3 is a flowchart of a method according to an embodiment of the present disclosure. It is a schematic diagram showing a situation when user A and user B are performing an online match play. It is a figure which illustrates roughly a mode when a user A puts on the HMD which he is wearing or the controller which he has in his hand on the nearby desk. It is a figure which roughly explains the mode of removal operation | movement detection using the locus | trajectory of the movement of HMD when removing HMD. It is a figure which illustrates roughly a mode that HMD removal operation detection is performed using the attachment / detachment sensor provided in HMD. It is a figure which illustrates roughly a mode when detecting a removal operation based on a position of HMD etc. It is a figure which illustrates roughly a mode when detecting a removal operation based on both movement and a position of HMD etc. It is a figure which illustrates roughly a mode when detecting a removal operation based on both movement and a position of HMD etc. It is a figure explaining roughly a mode when detecting a detachment operation based on a user's eyes. It is a figure explaining roughly a mode when detecting a detachment operation based on a user's eyes. It is a figure which roughly illustrates a mode at the time of performing a removal operation | movement detection based on the positional relationship between HMD and a controller. It is a figure which roughly illustrates a mode at the time of performing a removal operation | movement detection based on the positional relationship between HMD and a controller. It is a figure which shows an example of the alert provided as a user interface displayed on the display part 112 of HMD110. FIG. 6 is a schematic diagram illustrating some methods of providing additional alerts.

[Description of Embodiments of the Present Disclosure]
First, the configurations of exemplary embodiments of the present disclosure will be listed and described. The method, the program, and the computer according to the embodiment of the present disclosure may have the following configurations.

(Item 1)
A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
Output a UI (User Interface) to the image display device to prompt input of whether or not to end the virtual experience when the removal operation is detected,
A program to execute.

(Item 2)
The step of detecting the removal operation includes
Detecting a first part included in the head of the first user;
And a step of determining that the removal operation has been performed when the first part cannot be detected.

(Item 3)
The step of detecting the removal operation includes
Tracking the image display device,
The program according to item 1, comprising a step of determining that the detaching operation is performed when the image display device stands still or when the image display device moves on a predetermined trajectory.

(Item 4)
The step of detecting the removal operation includes
Tracking an operating device used by the first user;
Determining that the removal operation has been performed when the operating device is stationary or when the positional relationship between the operating device and the image display device satisfies a predetermined condition. Program of.

(Item 5)
On the computer,
Accepting an input to end the virtual experience,
Gracefully leaving the first user from the virtual experience in response to receiving an input to end the virtual experience;
5. The program according to any one of items 1 to 4, for executing the program.

(Item 6)
On the computer,
Accepting an input that does not end the virtual experience,
Returning the first user to the virtual experience in response to receiving an input that does not end the virtual experience;
6. The program according to any one of items 1 to 5, for executing the program.

(Item 7)
On the computer,
From item 1, which executes a step of determining that the first user has illegally left the virtual experience in response to neither input of ending the virtual experience nor input of not ending the virtual experience is received. 6. The program according to any one of 6.

(Item 8)
Items 1 to 7, wherein the UI includes a warning that the first user is considered to have been illegally withdrawn from the virtual experience if the first user does not return to the virtual experience or normally exits. The program according to any one of 1.

(Item 9)
And a step of causing at least one of an audio output device and an image output device associated with the image display device to perform an output for urging a normal departure from the virtual experience when the removal operation is detected. The program according to any one of 1 to 8.

(Item 10)
A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user,
By the control of the processor included in the computer,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
And a step of causing the image display device to output a UI (User Interface) prompting an input as to whether or not to end the virtual experience when the removal operation is detected.

(Item 11)
Computer-implemented for providing to the first user a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user. Method,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
Outputting a UI (User Interface) to the image display device for prompting an input as to whether or not to end the virtual experience when the removal operation is detected.

[Details of the embodiment of the present disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, similar elements are given similar reference numerals. Their names and functions are also the same. Overlapping description of such elements is omitted.

  The configuration of the image display device system 100 will be described with reference to FIG. Hereinafter, a system using a head-mounted device (HMD) as an example of the image display device will be specifically described. However, it will be understood by those skilled in the art that not only the HMD but various devices for providing a virtual experience to a user can be applied to the embodiments of the present disclosure.

  FIG. 1 schematically shows the configuration of the system 100. In one example, the system 100 is provided as a home system or a business system. The HMD may be a so-called head-mounted display having a display unit, or may be a head-mounted device to which a terminal such as a smartphone having the display unit can be attached.

  The system 100 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a display unit 112 and a gaze sensor 140. The controller 160 may include the motion sensor 130.

  In one example, the computer 200 may be connectable to a network 192 such as the Internet, or may be capable of communicating with a computer such as the server 150 connected to the network 192. In another aspect, HMD 110 may include sensor 114 instead of HMD sensor 120.

  The HMD 110 may be mounted on the head of the user 190 and provide the user with a virtual space during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the display unit 112, respectively. When each eye of the user visually recognizes each image, the user can recognize the image as a three-dimensional image based on the parallax of both eyes.

  The display unit 112 is realized, for example, as a non-transmissive display device. In one example, the display unit 112 is arranged on the main body of the HMD 110 so as to be located in front of both eyes of the user. Therefore, the user can immerse himself in the virtual space when viewing the three-dimensional image displayed on the display unit 112. In one embodiment, the virtual space includes, for example, a background, objects that the user can operate, images of menus that the user can select, and the like. In an embodiment, the display unit 112 can be realized as a liquid crystal display unit or an organic EL (Electro Luminescence) display unit included in an information display terminal such as a smartphone.

  In one example, the display unit 112 may include a sub display unit for displaying an image for the right eye and a sub display unit for displaying an image for the left eye. In another aspect, the display unit 112 may be configured to integrally display the image for the right eye and the image for the left eye. In this case, the display unit 112 includes a high speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

  In one example, the HMD 110 includes multiple light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. More specifically, the HMD sensor 120 may read a plurality of infrared rays emitted by the HMD 110 and detect the position and inclination of the HMD 110 in the physical space.

  In an aspect, the HMD sensor 120 may be implemented by a camera. In this case, the HMD sensor 120 can detect the position and inclination of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

  In another aspect, the HMD 110 may include a sensor 114 as a position detector instead of the HMD sensor 120. The HMD 110 can detect the position and inclination of the HMD 110 itself using the sensor 114. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 uses any one of these sensors instead of the HMD sensor 120 to detect its own position and inclination. obtain. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in the physical space over time. The HMD 110 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle. Further, the HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting its transmittance. Further, the view field image may include a configuration for presenting the real space in a part of the image forming the virtual space. For example, an image captured by a camera mounted on the HMD 110 may be displayed so as to be superimposed on a part of the view image, or a part of the transmissive display device may be set to have a high transmittance so that the view image The physical space may be made visible from a part.

  The gaze sensor 140 detects a direction (line of sight) in which the right and left eyes of the user 190 are directed. The detection of the direction is realized by a known eye tracking function, for example. The gaze sensor 140 is realized by a sensor having the eye tracking function. In an aspect, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right and left eyes of the user 190 with infrared light, and receives the reflected light from the cornea and the iris with respect to the irradiated light to detect the rotation angle of each eyeball. . The gaze sensor 140 can detect the line of sight of the user 190 based on the detected rotation angles.

  The server 150 may send the program to the computer 200. In another aspect, the server 150 may communicate with other computers 200 for providing virtual reality to HMDs used by other users. For example, when a plurality of users play a participation-type game in an amusement facility, each computer 200 communicates a signal based on the operation of each user with another computer 200 so that the plurality of users share a common virtual space. Allows you to enjoy the game.

The controller 160 is connected to the computer 200 by wire or wirelessly. The controller 160 receives an instruction input from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be grippable by the user 190. In another aspect, the controller 160 is configured to be attachable to a part of the body or clothing of the user 190. In another aspect, the controller 160 may be configured to output vibration, sound, and / or light based on a signal transmitted from the computer 200. In another aspect, the controller 160 accepts an operation from the user 190 for controlling the position and movement of an object arranged in the virtual space.

  In one aspect, the motion sensor 130 is attached to the user's hand to detect movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, the rotation speed, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove type controller 160A. In one embodiment, for safety in the physical space, the controller 160A is preferably attached to something that does not fly easily by being attached to the hand of the user 190, such as a glove type. In another aspect, a controller 160B having a general structure having a plurality of operation buttons may be used. In another aspect, a sensor not worn by the user 190 may detect movement of the user's 190 hand. Optical sensors may be used to detect the position, orientation, direction of movement, distance of movement, etc. of various parts of the body of the user 190. For example, a signal from a camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190. The motion sensor 130 and the computer 200 are wirelessly connected to each other, for example. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication method is used.

The computer 200 according to the embodiment of the present disclosure will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example of a basic hardware configuration of the computer 200 according to an embodiment of the present disclosure. The computer 200 includes a processor 202, a memory 204, a storage 206, an input / output interface 208, and a communication interface 210 as main components. Each component is connected to the bus 212.
The processor 202 executes a series of instructions included in a program stored in the memory 204 or the storage 206 based on a signal given to the computer 200 or when a predetermined condition is satisfied. In one aspect, the processor 202 is realized as a device such as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

  The memory 204 temporarily stores programs and data. The program is loaded from the storage 206, for example. The data includes data input to the computer 200 and data generated by the processor 202. In one aspect, the memory 204 is implemented as a volatile memory such as a RAM (Random Access Memory).

  The storage 206 permanently holds programs and data. The storage 206 is realized, for example, as a non-volatile storage device such as a ROM (Read-Only Memory), a hard disk device, or a flash memory. The programs stored in the storage 206 include a program for providing a virtual space in the system 100, a simulation program, a game program, a user authentication program, a program for realizing communication with another computer 200, and the like. The data stored in the storage 206 includes data and objects for defining the virtual space.

In another aspect, the storage 206 may be realized as a removable storage device such as a memory card. In yet another aspect, instead of the storage 206 built in the computer 200, a configuration using programs and data stored in an external storage device may be used. With such a configuration, for example, in a situation where a plurality of systems 100 are used, such as in an amusement facility, it is possible to collectively update programs and data.

  In certain embodiments, the input / output interface 208 communicates signals with the HMD 110, the HMD sensor 120, and the motion sensor 130. In one aspect, the input / output interface 208 is realized by using a terminal such as a USB (Universal Serial Bus, USB), a DVI (Digital Visual Interface), and a HDMI (registered trademark) (High-Definition Multimedia Interface). The input / output interface 208 is not limited to the above.

  In some embodiments, the input / output interface 208 can also communicate with the controller 160. For example, the input / output interface 208 receives the signals output from the controller 160 and the motion sensor 130. In another aspect, the input / output interface 208 sends the instructions output from the processor 202 to the controller 160. The command instructs the controller 160 to perform vibration, voice output, light emission, or the like. Upon receiving the command, the controller 160 executes vibration, voice output, light emission, etc. according to the command.

  The communication interface 210 is connected to the network 192 and communicates with another computer (for example, the server 150) connected to the network 192. In one aspect, the communication interface 210 is realized as a wired communication interface such as a LAN (Local Area Network), or a wireless communication interface such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication), or the like. To be done. The communication interface 210 is not limited to the above.

  In an aspect, the processor 202 accesses the storage 206, loads one or more programs stored in the storage 206 into the memory 204, and executes the series of instructions contained in the program. The one or more programs may include an operating system of the computer 200, an application program for providing a virtual space, game software executable in the virtual space, and the like. The processor 202 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 208. The HMD 110 displays an image on the display unit 112 based on the signal.

  In the example shown in FIG. 2, the computer 200 is provided outside the HMD 110. However, in another aspect, the computer 200 may be incorporated in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including the display unit 112 may function as the computer 200.

  Moreover, the computer 200 may be configured to be commonly used by a plurality of HMDs 110. With such a configuration, for example, the same virtual space can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space.

  In one embodiment, the system 100 has a preset global coordinate system. The global coordinate system has three reference directions (axes) that are parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-rear direction orthogonal to both the vertical direction and the horizontal direction. In this embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, vertical direction (vertical direction), and front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction in real space. The y-axis is parallel to the vertical direction of the real space. The z axis is parallel to the front-back direction of the physical space.

  In one aspect, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects infrared light emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the physical space according to the movement of the user 190 who wears the HMD 110, based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect temporal changes in the position and inclination of the HMD 110 by using each value detected over time.

  The global coordinate system is parallel to the coordinate system in real space. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination around the three axes of the HMD 110 in the global coordinate system. The HMD sensor 120 sets the uvw visual field coordinate system to the HMD 110 based on the inclination of the HMD 110 in the global coordinate system. The uvw visual field coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 views an object in the virtual space.

  The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually showing a uvw visual field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and inclination of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 202 sets the uvw visual field coordinate system in the HMD 110 based on the detected value.

  As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system with the head of the user wearing the HMD 110 as the center (origin). More specifically, the HMD 110 sets the horizontal direction, the vertical direction, and the front-back direction (x axis, y axis, z axis) that define the global coordinate system by the inclination around each axis of the HMD 110 in the global coordinate system. Three directions newly obtained by inclining about the axis are set as a pitch direction (u axis), a yaw direction (v axis), and a roll direction (w axis) of the uvw visual field coordinate system in the HMD 110.

  In an aspect, when the user 190 wearing the HMD 110 is standing upright and viewing the front, the processor 202 sets the uvw view coordinate system parallel to the global coordinate system in the HMD 110. In this case, the horizontal direction (x axis), vertical direction (y axis), and front-back direction (z axis) in the global coordinate system are the pitch direction (u axis) and yaw direction (v axis) of the uvw visual field coordinate system in the HMD 110. And the roll direction (w axis).

  After the uvw visual field coordinate system is set in the HMD 110, the HMD sensor 120 can detect the inclination (change amount of inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), the yaw angle (θv), and the roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

  The HMD sensor 120 sets the uvw visual field coordinate system in the HMD 110 after the HMD 110 moves to the HMD 110 based on the detected tilt angle of the HMD 110. The relationship between the HMD 110 and the uvw visual field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and the inclination of the HMD 110 change, the position and the inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in association with the change of the position and the inclination.

  In one aspect, the HMD sensor 120 uses the HMD 110 based on the light intensity of infrared light acquired based on the output from the infrared sensor and the relative positional relationship between a plurality of points (for example, the distance between the points). The position in the real space may be specified as the relative position with respect to the HMD sensor 120. Further, the processor 202 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the physical space (global coordinate system) based on the specified relative position.

  The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually showing one aspect of expressing the virtual space 400 according to an embodiment. The virtual space 400 has a spherical structure that covers the entire center 406 in the 360 ° direction. In FIG. 4, the celestial sphere in the upper half of the virtual space 400 is illustrated in order not to complicate the description. Each mesh is defined in the virtual space 400. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system defined in the virtual space 400. The computer 200 associates each partial image forming the content (still image, moving image, etc.) that can be expanded in the virtual space 400 with each corresponding mesh in the virtual space 400, and the virtual space image visually recognizable by the user is displayed. The expanded virtual space 400 is provided to the user.

  In an aspect, in the virtual space 400, an xyz coordinate system whose origin is the center 406 is defined. The xyz coordinate system is, for example, parallel to the global coordinate system. Since the xyz coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the xyz coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. Therefore, the x axis of the xyz coordinate system (horizontal direction) is parallel to the x axis of the global coordinate system, the y axis of the xyz coordinate system (vertical direction) is parallel to the y axis of the global coordinate system, and the x axis of the xyz coordinate system is The z axis (front-back direction) is parallel to the z axis of the global coordinate system.

  When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 404 is arranged at the center 406 of the virtual space 400. In an aspect, the processor 202 displays the image captured by the virtual camera 404 on the display unit 112 of the HMD 110. The virtual camera 404 similarly moves in the virtual space 400 in association with the movement of the HMD 110 in the physical space. Thereby, the change in the position and orientation of the HMD 110 in the physical space can be reproduced in the virtual space 400 as well.

  As in the case of the HMD 110, the uvw visual field coordinate system is defined for the virtual camera 404. The uvw visual field coordinate system of the virtual camera 404 in the virtual space 400 is defined to be linked to the uvw visual field coordinate system of the HMD 110 in the physical space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 404 also changes accordingly. The virtual camera 404 can also move in the virtual space 400 in conjunction with the movement of the user wearing the HMD 110 in the real space.

  The processor 202 of the computer 200 defines the view area 410 in the virtual space 400 based on the arrangement position of the virtual camera 404 and the reference line of sight 408. The view area 410 corresponds to an area of the virtual space 400 that is visually recognized by the user wearing the HMD 110.

  The line of sight of the user 190 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 190 visually recognizes an object. The uvw visual field coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the display unit 112. The uvw visual field coordinate system of the virtual camera 404 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, system 100 according to an aspect can regard the line of sight of user 190 detected by gaze sensor 140 as the line of sight of the user in the uvw visual field coordinate system of virtual camera 404.

  The determination of the line of sight of the user will be described with reference to FIG. FIG. 5 is a diagram showing the head of the user 190 wearing the HMD 110 according to an embodiment from above.

  In one aspect, the gaze sensor 140 detects each line of sight of the right and left eyes of the user 190. In an aspect, the gaze sensor 140 detects the lines of sight R1 and L1 when the user 190 is looking closer. In another aspect, the gaze sensor 140 detects lines of sight R2 and L2 when the user 190 is looking far away. In this case, the angle formed by the sight lines R2 and L2 with respect to the roll direction w is smaller than the angle formed by the sight lines R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

  When the computer 200 receives the detection values of the sight lines R1 and L1 as the detection result of the sight line from the gaze sensor 140, the computer 200 specifies the gaze point N1 which is the intersection of the sight lines R1 and L1 based on the detection value. On the other hand, when the computer 200 receives the detection values of the sight lines R2 and L2 from the gaze sensor 140, the computer 200 specifies the intersection of the sight lines R2 and L2 as the gaze point. The computer 200 specifies the line of sight N0 of the user 190 based on the specified position of the gazing point N1. The computer 200 detects, as the line of sight N0, for example, a direction in which a straight line passing through the midpoint of a straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1 extends. The line of sight N0 is the direction in which the user 190 is actually directing his or her eyes with both eyes. Further, the line of sight N0 corresponds to the direction in which the user 190 is actually directing the line of sight with respect to the view field 410.

  In another aspect, the system 100 may include a microphone and a speaker in any part of the system 100. The user can give a voice instruction to the virtual space 400 by speaking into the microphone.

  In another aspect, the system 100 may include a television broadcast receiving tuner. With such a configuration, the system 100 can display the television program in the virtual space 400.

  In yet another aspect, the system 100 may include a communication circuit for connecting to the Internet or a call function for connecting to a telephone line.

  The visibility region 410 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing a yz cross section of the view field 410 in the virtual space 400 viewed from the x direction. FIG. 7 is a diagram showing an xz cross section of the view field 410 in the virtual space 400 viewed from the y direction.

  As shown in FIG. 6, the view field 410 in the yz section includes a region 602. The area 602 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the yz section of the virtual space 400. The processor 202 defines a range including the polar angle α around the reference line of sight 408 in the virtual space as a region 602.

As shown in FIG. 7, the view field 410 in the xz section includes a region 702. The area 702 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the xz section of the virtual space 400. The processor 202 defines a range including the azimuth angle β around the reference line of sight 408 in the virtual space 400 as a region 702. The polar angles α and β are determined according to the arrangement position of the virtual camera 404 and the orientation of the virtual camera 404.

  In one aspect, the system 100 provides the user 190 with a view in the virtual space by displaying a view image on the display unit 112 based on a signal from the computer 200. The view field image corresponds to a portion of the virtual space image 402 that is superimposed on the view field region 410. When the user 190 moves the HMD 110 mounted on the head, the virtual camera 404 also moves in conjunction with the movement. As a result, the position of the view field 410 in the virtual space 400 changes. As a result, the view image displayed on the display unit 112 is updated to the image of the virtual space image 402 that is superimposed on the view area 410 in the direction in which the user is facing in the virtual space 400. The user can visually recognize a desired direction in the virtual space 400.

  Thus, the orientation (tilt) of the virtual camera 404 corresponds to the user's line of sight (reference line of sight 408) in the virtual space 400, and the position where the virtual camera 404 is arranged corresponds to the user's viewpoint in the virtual space 400. Therefore, by moving the virtual camera 404 (including the operation of changing the arrangement position and the operation of changing the direction), the image displayed on the display unit 112 is updated, and the field of view (including the viewpoint and the line of sight) of the user 190 moves. To be done.

  The user 190 can visually recognize only the virtual space image 402 developed in the virtual space 400 without visually recognizing the real world while wearing the HMD 110. Therefore, the system 100 can give the user a high sense of immersion in the virtual space 400.

  In an aspect, the processor 202 can move the virtual camera 404 in the virtual space 400 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 202 specifies the image area (that is, the view area 410 in the virtual space 400) projected on the display unit 112 of the HMD 110 based on the position and orientation of the virtual camera 404 in the virtual space 400.

  According to certain embodiments, virtual camera 404 may include two virtual cameras, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In addition, appropriate parallax may be set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 400.

  A controller 160A that is an example of the controller 160 will be described with reference to FIG. 8A. FIG. 8A is a diagram showing a schematic configuration of a controller 160A according to an embodiment.

  In an aspect, the controller 160A can include a right controller and a left controller. For ease of explanation, controller 160A in FIG. 8A shows the right controller. The right controller is operated by the right hand of the user 190. The left controller is operated by the left hand of the user 190. In one aspect, the right controller and the left controller are symmetrically configured as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller and the left hand holding the left controller. In another aspect, the controller 160 may be an integrated controller that accepts the operation of both hands. The right controller 160A will be described below.

  The right controller 160A includes a grip 802, a frame 804, and a top surface 806. The grip 802 is configured to be gripped by the right hand of the user 190. For example, the grip 802 may be held by the palm of the user's 190 right hand and three fingers (middle finger, ring finger, little finger).

  Grip 802 includes buttons 808 and 810 and motion sensor 130. The button 808 is arranged on the side surface of the grip 802 and receives an operation by the middle finger of the right hand. The button 810 is arranged on the front surface of the grip 802 and receives an operation by the index finger of the right hand. In one aspect, the buttons 808, 810 are configured as trigger-type buttons. The motion sensor 130 is built in the housing of the grip 802. Note that the grip 802 may not include the motion sensor 130 when the motion of the user 190 can be detected from around the user 190 by a camera or other device.

  The frame 804 includes a plurality of infrared LEDs 812 arranged along the circumferential direction thereof. The infrared LED 812 emits infrared light according to the progress of the program while the program using the controller 160A is being executed. The infrared rays emitted from the infrared LED 812 can be used to detect each position and posture (tilt, direction) of the right controller 160A and the left controller (not shown). In the example shown in FIG. 8, the infrared LEDs 812 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. Arrays with one row or more than two rows may be used.

  The top surface 806 includes buttons 814 and 816 and an analog stick 818. Buttons 814 and 816 are configured as push buttons. Buttons 814 and 816 accept an operation performed by the thumb of the right hand of user 190. In a certain aspect, the analog stick 818 accepts an operation in an arbitrary direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object arranged in the virtual space 400.

  In one aspect, the right controller 160A and the left controller include batteries for driving components such as infrared LEDs 812. The battery may be either a primary battery or a secondary battery, and the shape thereof may be any of button type, dry cell type and the like. In another aspect, the right controller 160A and the left controller may be connected to a USB interface of the computer 200, for example. In this case, the right controller 800 and the left controller may be powered via the USB interface.

  A controller 160B that is an example of the controller 160 will be described with reference to FIG. 8B. FIG. 8B is a diagram illustrating a schematic configuration of the controller 160B according to an embodiment.

  The controller 160B includes a plurality of buttons 820 (820a, 820b, 820c, 820d) and 822 (822a, 822b, 822c, 822d), and left and right analog sticks 824L and 824R. Each button 820 and 822 is configured as a push button. Buttons 820 and 822 accept an operation by the thumb of the user 190's hand. A trigger-type button (not shown) capable of accepting an operation with the index finger or the middle finger of the user 190 may be further provided in the controller 160B. In a certain aspect, the analog sticks 824L and 824R each accept an operation in an arbitrary direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object arranged in the virtual space 400. Buttons 820 (820a, 820b, 820c, 820d) and 822 (822a, 822b, 822c, 822d) and analog sticks 824L and 824R (and trigger-type buttons (not shown) if included) are each operated separately. Command is assigned. The operation command is, for example, a command for giving a command to an object in the virtual space 400, a command for making various settings on a game menu screen, and a computer when the user 190 is experiencing the virtual space 400. Includes any other commands that may be input to 200. The operation command assigned to each button or analog stick may be dynamically changed, for example, according to the progress of the game or the change of the scene.

  In one aspect, the controller 160B may include a plurality of infrared LEDs (not shown) disposed on the outer surface thereof. The infrared LED emits infrared light according to the progress of the program while the program using the controller 160B is being executed. The infrared rays emitted from the infrared LEDs can be used to detect the position and posture (tilt, orientation) of the controller 160B. The controller 160B also includes a battery for driving internal electronic components. The battery may be either a primary battery or a secondary battery, and the shape thereof may be any of button type, dry cell type and the like. In another aspect, the controller 160B may be connected to the USB interface of the computer 200, for example. In this case, the controller 160B may be powered via the USB interface.

  FIG. 9 is a block diagram illustrating the functionality of computer 200 for providing a virtual experience to a user via system 100, according to an embodiment of the disclosure. As previously mentioned, various image display devices for providing a virtual experience to a user may be used in the embodiments of the present disclosure. In the embodiments described below, an HMD is used as an example of an image display device.

  The computer 200 mainly executes various processes based on inputs from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, and the controller 160.

  The computer 200 includes a processor 202, a memory 204, and a communication control unit 205. The processor 202 includes a virtual space specifying unit 902, an HMD motion detecting unit 904, a line-of-sight detecting unit 906, a reference line-of-sight determining unit 908, a view area determining unit 910, a virtual viewpoint specifying unit 912, and a view image generating unit 914. A virtual camera control unit 916, a game departure control unit 918, and a view image output unit 924. The memory 204 may be configured to store various information. In one example, the memory 204 may include virtual space data 926, object data 928, application data 930, removal determination data 932, alert data 934, other data 938, and the like. The memory 204 also includes various data necessary for calculation for providing output information corresponding to inputs from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, the controller 160, and the like to the display unit 112 associated with the HMD 110. But it's okay. The object data 928 may include data regarding various objects arranged in the virtual space. The removal determination data 932 may include various data used to determine the removal operation of the image display device (HMD or the like) by the user. The display unit 112 may be built in the HMD 110 or a display of another device (for example, a smartphone) that can be attached to the HMD 110.

  The components included in the processor 202 in FIG. 9 are merely one example of expressing the functions executed by the processor 202 as concrete modules. The functionality of multiple components may be implemented by a single component. Processor 202 may be configured to perform the functions of all components.

  FIG. 10 is a flowchart of a general process for displaying an image of a virtual space in which a user is immersed in the display unit 112.

With reference to FIGS. 9 and 10, general processing of the system 100 for providing an image in the virtual space will be described. The virtual space 400 may be provided by the interaction of the HMD sensor 120, the gaze sensor 140, the computer 200, and the like.

  The process starts at step 1002. As an example, the game application included in the application data 930 may be executed by the computer 200. In step 1004, the processor 202 (or the virtual space identifying unit 902) generates the celestial spherical virtual space image 402 that constitutes the virtual space 400 in which the user is immersed, by referring to the virtual space data 926 and the like. The HMD sensor 120 detects the position and inclination of the HMD 110. The information detected by the HMD sensor 120 is transmitted to the computer 200. In step 1006, the HMD operation detection unit 904 acquires the position information, the tilt information, and the like of the HMD 110. In step 1008, the view direction is determined based on the acquired position information and tilt information.

  When the gaze sensor 140 detects the movement of the eyeballs of the left and right eyes of the user, the information is transmitted to the computer 200. In step 1010, the line-of-sight detection unit 906 identifies the directions in which the lines of sight of the right and left eyes are directed, and determines the line-of-sight direction N0. In step 1012, the reference line-of-sight determination unit 908 determines the visual field direction or the user's line-of-sight direction N0 determined by the inclination of the HMD 110 as the reference line of sight 408. The reference line-of-sight 408 may also be determined based on the position and tilt of the virtual camera 404 that follows the position and tilt of the HMD 110.

  In step 1014, the view area determination unit 910 determines the view area 410 of the virtual camera 404 in the virtual space 400. As shown in FIG. 4, the visual field 410 is a portion of the virtual space image 402 that constitutes the visual field of the user. The visual field 410 is determined based on the reference line of sight 408. The yz sectional view of the view field 410 viewed from the x direction and the xz sectional view of the view field 410 viewed from the y direction are shown in FIGS. 6 and 7 described above, respectively.

  In step 1016, the visual field image generation unit 914 generates a visual field image based on the visual field region 410. The view image includes two two-dimensional images for the right eye and the left eye. By superimposing these two-dimensional images on the display unit 112 (more specifically, the image for the right eye is output to the display unit for the right eye and the image for the left eye is output to the display unit for the left eye), the three-dimensional image is displayed. The virtual space 400 as an image is provided to the user. In step 1018, the visual field image output unit 924 outputs information regarding the visual field image to the display unit 112. The display unit 112 displays the visual field image based on the received information on the visual field image. The process ends at step 1020.

  FIG. 11 is a flowchart of a game departure control method 1100 according to one embodiment of the present disclosure. In one embodiment of the present disclosure, a computer program may cause the processor 202 (or the computer 200) to execute the steps illustrated in FIG. 11. Further, another embodiment of the present disclosure may be implemented as a computer including at least a processor and executing the method 1100 under the control of the processor.

  Hereinafter, embodiments of the present disclosure will be specifically described. Here, as a specific example to which the embodiment of the present disclosure can be applied, it is assumed that the user can enjoy the game by immersing himself in a virtual space in which an avatar, another object, etc. associated with the user are arranged. . However, the embodiments of the present disclosure are not necessarily limited to such an aspect. It will be apparent to those skilled in the art that the embodiments of the present disclosure can take various aspects within the scope defined by the claims.

FIG. 12 is a schematic diagram showing a situation in which a plurality of users, for example, two users, user A and user B, are performing match play of an online game. The number of users who participate in the game is not limited to two. Embodiments of the present disclosure can also be applied to games in which a larger number of users can participate. The system 100 shown in FIG. 9 is installed in each place of the user A and the user B, and these systems are mutually connected via the network 192 and the server 150. In order to simplify the description, in FIG. 12, an HMD 1202 such as the above HMD 110 worn by the user A, a controller 1204 such as the above controller (movable device related to the game) held in the right hand, and these HMD 1202. And only the personal computer (PC) 1208 on the desk 1206 to which the controller 1204 is connected is shown. However, it will be apparent that the other elements shown in FIG. 1 or 9 may be provided. User B has a similar device, but the device is omitted for simplification. Hereinafter, the embodiment of the present disclosure will be described on the assumption that the users A and B shown in FIG. 12 are performing online match play.

  Returning to FIG. 11, the process first proceeds to step 1104. In step 104, the game leaving control unit 918 determines whether the user A is playing an online match in which other users are also participating based on the information regarding the currently played game obtained from the application data 930 and the like. judge. If it is determined in step 1104 that the user A is not playing (“N” in step 1104), the process proceeds to step 1122 and ends. For example, this process is executed when the user A is not in the online match play, or when the user A or the opponent user B normally ends the game being played.

  On the other hand, if the user A is currently playing (“Y” in step 1104), the process proceeds to step 1106. In step 1106, the game leaving control unit 918 determines whether or not the removal operation of the HMD 1202 or the like by the user A has been detected. Details of detection of the removal operation of the HMD 1202 and the like will be described later with reference to FIGS. 12 to 19B. If no removal operation was detected (“N” in step 1106), the process returns to before step 1104. On the other hand, when the removal operation is detected (“Y” in step 1106), the process proceeds to step 1108. In step 1108, the game leaving control unit 918 generates an alert for the user A as an output for prompting an input regarding return to the game by referring to the alert data 934 or the like. Generating this alert may include generating a user interface. The alert data 934 may store data such as the content, form, and location of the generated alert. The alert can be generated in the form of display, sound, vibration, light, etc., can be generated in a plurality of forms, or can be generated in duplicate. A specific example of the alert will be described later with reference to FIGS. 20 and 21.

  The process proceeds to step 1110, and the game leaving control unit 918 determines whether or not the user A has selected to return to the play in response to the output prompting the input regarding the return to the game. When it is determined that the return to the play is selected (“Y” in step 1110), the process returns to step 1104. If the return to play is not selected (“N” in step 1110), the process proceeds to step 1112.

  In step 1112, the game leaving control unit 918 determines whether or not the non-return to the play, that is, the normal end is selected. If normal termination is selected (“Y” in step 1112), the process proceeds to step 1114. In step 1114, normal termination processing is performed. In one example, as a result of the user A selecting to leave match play, the loss of the user A and the win of the opponent user B may be determined and the game may be ended. In a game in which a plurality of users participate, how to determine the outcome of the game when some users leave may be arbitrarily determined. For example, even if user A has a higher score than user B, user A may decide to lose when user A leaves.

  If no return to normal play (normal end) is not selected (“N” in step 1112), the process proceeds to step 1116. In step 1116, the game departure control unit 918 determines whether the communication connection necessary for playing the game has been disconnected. For example, when the communication cable for connecting to the network is pulled out, it is determined that the communication connection is disconnected. If it is determined that the communication connection has been disconnected (“Y” in step 1116), the process proceeds to step 1118. In step 1118, the game leave control unit 918 ends the process after performing the illegal leave process. In an example of the improper exit processing, in addition to the disadvantage imposed on the user A in the above normal end processing for the user A who is determined not to end the match play normally, restrictions on future game participation, Penalties and other penalties may be imposed. What kind of disadvantage is imposed on the user A in the illegal withdrawal process can be arbitrarily determined according to the type of game, the influence on the opponent, and the like.

  On the other hand, if it is determined that the communication connection has not been disconnected (“N” in step 1116), the process proceeds to step 1120. In step 1120, the game departure control unit 918 determines whether a predetermined time has passed. The predetermined time may be set, for example, for the time that has elapsed since the removal operation was detected in step 1106. Alternatively, the predetermined time may be set for the time that has elapsed since the alert was generated in step 1108. Alternatively, the predetermined time may be set for the time elapsed from the time point associated with steps 1110, 1112 or 1116. These predetermined times may correspond to the time when the opponent user B waits for some response from the user A. Therefore, it is not preferable to set the predetermined time too long. As an example, the predetermined time may be set to 30 seconds. Depending on the type of game, the predetermined time may be a longer time or a shorter time. By providing such a time limit, it is possible to prevent the opponent player from being annoyed by waiting indefinitely.

  If the predetermined time has not elapsed (“N” in step 1120), the process returns to step 1110, and whether or not return to play is selected, whether or not not return to play is selected, communication is performed. The determination as to whether or not the connection has been disconnected and whether or not a predetermined time has elapsed is repeatedly executed. The processing regarding each determination is as described above. When it is determined in step 1120 that the predetermined time has elapsed (“Y” in step 1120), the process proceeds to step 1118, the above-mentioned illegal withdrawal process is executed, and this flow ends.

  Next, with reference to FIGS. 13 to 19, a specific example of detection of the removal operation of the HMD or the like in step 1106 of FIG. 11 will be described.

First, an example of detection of the HMD removal operation based on the removal of the HMD or the release of the controller will be described with reference to FIG. As shown in the figure, consider a situation in which the user A removes the worn HMD 1202 from his / her head and places it on a nearby desk. Here, the HMD placed on the desk is shown as an HMD 1302. In this case, the movement of the HMD 1302 is almost or completely stationary. Therefore, by determining whether or not the moving distance of the HMD has become equal to or less than the minimum distance, it is possible to know whether or not the HMD is in the completely stationary state or in a state close thereto. This makes it possible to detect whether the user A has removed the HMD 1202. The above minimum distance may be stored in advance in the removal determination data 932 for the HMD and the controller. While the user wears the HMD 1202 and plays the game, some movement of the HMD occurs. Therefore, by using the above-mentioned minimum distance as a threshold, it is possible to determine whether or not the HMD is in a completely stationary state or a state close to it. The position and movement of the HMD 1202 can be detected using the position tracking function of the HMD sensor 120 or the position and tilt detection function of the HMD by the sensor 114 described above. In the case of an HMD that does not have such a function, a sensor such as a gyro sensor may be added. Although the desk is shown as the place where the HMD is placed in FIG. 13, the embodiment of the present disclosure can be similarly applied to the case where the HMD is placed on any other place such as a floor or a chair.

  FIG. 13 also shows a situation in which the controller 1204 held by the user A is placed on the desk, and the placed controller is shown as the controller 1304. Also in this case, the controller 1304 is in a completely stationary state or a state close thereto. While the user holds the controller in his hand and plays the game, some movement occurs. Therefore, when it is determined that the controller is in the completely stationary state or a state close to the completely stationary state, the user has let go of the controller and placed it somewhere (that is, the removal operation has been performed). The position, movement, etc. of the controller can be detected by, for example, the motion sensor 130 described above. Some users put the controller first before removing the HMD. In such a case, by performing the processing based on the stationary state of the controller, it is possible to detect the removal operation by the user earlier than the processing based on the movement of the HMD.

  Next, with reference to FIG. 14, as another example of the processing based on the movement of the HMD, detection of the HMD removal operation using the locus of the movement of the HMD during the removal operation will be described. As shown in the figure, when the user A tries to remove the HMD 1404 mounted on the head 1402, the user A first lifts the HMD 1404 upward along the trajectory 1406, and then moves the lifted position somewhere else. HMD 1404 is lowered down along track 1408 for placement on (eg, on a desk). In such a case, the locus 1406 and the locus 1408 are parabolic curves. Therefore, when the movement of the HMD along such a parabolic trajectory is detected, it can be determined that the HMD removal operation is being performed. Considering a situation in which the user A simply looks up for comparison with such a case, the HMD follows a locus 1410, for example. It can be seen that the movement distance of the trajectory 1410 is short and the amount of vertical movement is small, so that it is considerably different from the parabolic movement of the trajectory 1406 or the trajectory 1408. Therefore, it is possible to distinguish the operation of removing the HMD by the user and the operation of the user simply pointing up. According to this detection method, the removal operation can be detected at a stage before the HMD is placed somewhere and completely stopped.

  As another example, the HMD removal operation may be detected by using an unnatural HMD movement in game play that is unlikely to occur during game play. For example, such unnatural movement may be stored in the removal determination data 932 as data in advance. The removal operation can be detected by determining whether the detected HMD movement is the same as or similar to the previously stored movement.

  Next, with reference to FIG. 15, detection of the HMD removal operation using the attachment / detachment sensor will be described. In this example, the HMD 1504 attached to the head 1502 of the user A is provided with the attachment / detachment sensor 1506 schematically indicated by the dotted line. By using the attachment / detachment sensor 1506, it is possible to detect the removal operation by determining that the HMD 1504 has been removed from the head when the forehead of the user A is no longer detected. The attachment / detachment sensor 1506 may be, for example, a sensor that uses infrared rays, a mechanical switch type sensor, or any other type sensor.

  Next, with reference to FIG. 16, an example in which the HMD removal operation is detected based on the position of the HMD or the like will be described. Here, the position also includes the orientation of the HMD and the like. As shown, the HMD 1602 is assumed to be in the downward position. In this case, the occurrence of the removal operation can be detected by determining that the HMD 1602 is in the removed state when the downward state of the HMD 1602 continues for a certain period of time or longer. The same processing can be performed when the HMD 1602 is in the upward state or the state in which the HMD 1602 is in the other direction continues for a predetermined time or more. It is almost impossible in game play that the orientation of the HMD remains the same for a certain time or longer. Therefore, when such a situation occurs, it can be determined that the HMD is placed on the desk, floor, or the like. Similarly, it is determined that the controller 1604 has been released from the user's hand and placed on a desk or the like when a certain time or more has passed while the controller 1604 is in a state of a specific orientation (for example, sideways orientation). May be.

  The example in which the HMD removal operation is detected based on one of the movement and the position of the HMD has been described above. However, the embodiment of the present disclosure is not limited to such an aspect. The removal operation may be detected based on both the movement and the position of the HMD or the like. For example, the removal operation is performed based on a movement from a predetermined position due to a predetermined movement, a movement to a predetermined position due to a predetermined movement, a predetermined movement at a predetermined position, or the like. May be detected.

  An example in which the HMD removal operation detection is performed based on both the movement and the position of the HMD and the like will be described with reference to FIGS. 17A and 17B. In FIGS. 17A and 17B, the direction of the HMD is indicated by a thick white arrow for simplification. As shown, the HMD 1702 of FIG. 17A is moved upward a certain distance in a forward facing condition. This motion can be interpreted as a motion of moving the HMD 1702 overhead. On the other hand, the HMD 1704 in FIG. 17B is moved downward for a certain distance or more. This motion can be interpreted as a motion of placing the HMD 1704 on the floor or the like. In this way, the removal operation of the HMD may be detected based on the interpretation of the movement of the HMD.

  In the above description, an example in which the removal operation of the HMD or the like is detected on the assumption that the HMD or the like is detached from the head or placed on a desk in advance. As another example, if the HMD movement tracking is continued by performing the HMD position tracking, data regarding the HMD movement is accumulated. Then, teacher data indicating what kind of movement the user is performing the detaching operation can be acquired and used for machine learning using artificial intelligence (AI). In that case, it may also be possible to detect the removal operation based on movements and positions that were not assumed in advance.

  Next, with reference to FIGS. 18A and 18B, an example in which the HMD removal operation is detected based on at least one of the position and the movement of at least one of the head and eyes of the user will be described. Note that the movement and position of the user's head can be specified as the position and movement of the HMD when the HMD is worn, so here, in particular, processing based on the position and movement of the eyes will be described. As shown in FIGS. 18A and 18B, the HMD 1802 includes a gaze sensor 1804 (schematically illustrated) such as the gaze sensor 140 described above, and performs eye tracking of the eyes of the user. In FIG. 18A, it is assumed that the user wears the HMD 1802 on his / her head. At this time, the gaze sensor 1804 can track the eyes of the user. On the other hand, in FIG. 18B, when the user lifts the HMD overhead to remove it from the head, the gaze sensor 1804 is unable to track the user's eyes. In this way, it may be determined that the removal operation has been performed when the position and / or movement of the eyes cannot be detected. Furthermore, it may be determined that the removal operation has been performed when an unnatural eye movement that normally cannot occur during game play is detected. The data used for such a determination may be stored in the removal determination data 932.

Next, with reference to FIG. 19A and FIG. 19B, an example in which the HMD removal operation is detected based on the relative positional relationship between the HMD and the controller will be described. FIG. 19A shows HM
A state in which the user A wearing the D1902 holds the controller 1904 in his hand is shown. The maximum distance D1 between the HMD 1902 and the controller 1904 at this time is substantially determined by the length of the hand of the user A. In this situation, when the user A places the controller 1904 on a desk a little away in preparation for removing the HMD 1902, the distance D2 between the controller 1906 placed on the desk and the HMD 1902 is larger than the maximum distance D1. growing. Therefore, when the positional relationship between the controller and the HMD satisfies a predetermined condition (in this example, the distance D2 is larger than the maximum value of the distance D1), the HMD removal operation is performed (or is performed). I am trying to)). The same process can be performed when the controller is placed at the position indicated by reference numeral 1908 instead of on the desk and the distance D3 between the controller 1908 and the HMD 1902 becomes equal to or larger than the maximum distance D1.

  On the other hand, as shown in FIG. 19B, when user A moves controller 1904 to the position indicated by reference numeral 1910 and approaches HMD 1902, the distance between controller 1910 and HMD 1902 becomes a certain distance or less. This state can be interpreted as a state in which the user A is trying to remove the HMD 1902 and the hand holding the controller is brought close to the HMD 1902. Therefore, it may be determined that the removal operation of the HMD is performed when a predetermined condition such as a distance between the controller and the HMD being equal to or less than a predetermined distance is satisfied. Alternatively, when it is not necessary to bring the controller closer to the HMD due to the nature of the game, it may be determined that the HMD removal operation has occurred when the controller has been brought closer to the HMD for a certain period of time or longer.

  19A and 19B, the determination based on whether the positional relationship between the controller and the HMD satisfies a predetermined condition has been described. As another example, the same determination can be performed based on whether the positional relationship between the controller and the user's head satisfies a predetermined condition.

  Next, details of the alert generation in step 1108 of FIG. 11 will be described with reference to FIGS. 20 and 21.

  FIG. 20 shows an example of the alert displayed on the display unit 112 (see FIG. 9) of the HMD 110. The alert 2002 is displayed with the detection of the removal operation of the HMD or the like as a trigger, and provides a user interface (UI) for receiving an input from the user. In this example, the alert 2002 includes a text display 2004 “Would you like to return to play?” And a “Yes” button 2006 for selecting a return. When selecting the return to the play in step 1110 of FIG. 11, the user may operate the “Yes” button 2006. Further, the alert 2002 includes a text display 2008 “Do you want to select not to return to play (normal end)?” And a “Yes” button 2010 for selecting normal end, that is, normal departure. When selecting normal termination in step 1112 of FIG. 11, the user may operate the “Yes” button 2010. Further, the alert 2002 may include a warning sentence 2012 for the user, "Please note that if you do not end normally, you will be illegally leaving and you will be disadvantaged for future game play." The warning message can prompt the user to return to the game or to normally end the game. As another example, the alert 2002 may include a warning statement such as "Please recover within 30 seconds. If not recovered, it will be considered as a retirement."

The removal operation detection method described with reference to FIGS. 13 to 19 is a method that can detect the removal operation before the HMD is removed or during the removal of the HMD (in terms of timing, an alert 2002 displayed on the display unit 112 of the HMD 110 is displayed. A case where the user can visually recognize) and a method in which the removal operation cannot be detected unless the removal of the HMD is completed (timingally, the HMD 11
The case where the user cannot visually recognize the alert 2002 displayed on the display unit 112 of 0).

  FIG. 21 illustrates duplicate alerts according to embodiments of the present disclosure that can be applied to the techniques described in FIGS. 13-19. Since the alert 2002 shown in FIG. 20 is displayed on the display unit 112 of the HMD 110, the user cannot see the alert 2002 after removing the HMD. Therefore, during the removal operation of the HMD or the like, an additional alert may be generated in addition to the above-mentioned alert 2002 in order to generate an alert before the removal operation is completed to attract the user's attention more reliably. For example, as shown, text or some other type of alert 2106 may be displayed on the display of a PC 2104 running a VR game placed on user A's desk 2102. This additional alert 2106 may include a warning statement, such as warning statement 2012 shown in FIG. In another example, the additional alert may be to vibrate the controller 2108 located on the desk 2102. Alternatively, the additional alert may be to turn on or blink a lamp 2112 provided on the removed HMD 2110. Further, the additional alert may be to generate an alarm sound and emit the alarm sound via the headphones 2114 used by the user. When a speaker is used instead of headphones, an alarm sound may be emitted through the speaker.

  By performing the duplicate alert of FIG. 21, even if the removal operation cannot be detected unless the removal of the HMD is completed, the user notices the alert. Therefore, it is recommended that the user attach the HMD again and end it normally. Can be urged. In addition, even if the removal operation can be detected before or during removal of the HMD, the user does not always notice the alert, so it is useful to issue the duplicate alert of FIG.

  Although the embodiments of the present disclosure have been described above, it should be understood that these are merely examples and do not limit the scope of the present disclosure. It should be understood that changes, additions, improvements, etc. of the embodiments can be appropriately made without departing from the spirit and scope of the present disclosure. The scope of the present disclosure should not be limited by any of the above-described embodiments, but should be defined by the claims and their equivalents.

  100 ... HMD system, 112 ... Display part, 114 ... Sensor, 120 ... HMD sensor, 130 ... Motion sensor, 140 ... Gaze sensor, 150 ... Server, 160A, 160B ... Controller, 190 ... Head, 192 ... Network, 200 ... Computer, 202 ... Processor, 204 ... Memory, 205 ... Communication control unit, 206 ... Storage, 208 ... Input / output interface, 210 ... Communication interface, 212 ... Bus, 400 ... Virtual space, 402 ... Virtual space image, 404 ... Virtual camera , 406 ... Center, 408 ... Reference line of sight, 410 ... View area, 800 ... Right controller, 802 ... Grip, 804 ... Frame, 806 ... Top surface, 808, 810, 814, 816 ... Button, 812 ... Infrared LED, 818 ... Analog stick, 820, 22 ... Button, 824R, 824L ... Analog stick, 902 ... Virtual space specifying unit, 904 ... HMD motion detecting unit, 906 ... Line-of-sight detecting unit, 908 ... Standard line-of-sight determining unit, 910 ... View area determining unit, 912 ... Virtual viewpoint specifying 914 ... View image generation unit, 916 ... Virtual camera control unit, 918 ... Game departure control unit, 924 ... View image output unit, 926 ... Virtual space data, 928 ... Object data, 930 ... Application data, 932 ... Detachment determination Data, 934 ... Alert data

Claims (11)

A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
Output a UI (User Interface) to the image display device to prompt input of whether or not to end the virtual experience when the removal operation is detected,
A program to execute. The step of detecting the removal operation includes
Detecting a first part included in the head of the first user;
The program according to claim 1, further comprising a step of determining that the removal operation has been performed when the first portion cannot be detected. The step of detecting the removal operation includes
Tracking the image display device,
The program according to claim 1, further comprising a step of determining that the removal operation is performed when the image display device is stationary or when the image display device moves along a predetermined trajectory. The step of detecting the removal operation includes
Tracking an operating device used by the first user;
Determining that the removal operation has been performed when the operating device is stationary or when the positional relationship between the operating device and the image display device satisfies a predetermined condition. The listed program. On the computer,
Accepting an input to end the virtual experience,
Gracefully leaving the first user from the virtual experience in response to receiving an input to end the virtual experience;
The program according to any one of claims 1 to 4, which executes the program. On the computer,
Accepting an input that does not end the virtual experience,
Returning the first user to the virtual experience in response to receiving an input that does not end the virtual experience;
The program according to claim 1, wherein the program is executed. On the computer,
The step of determining that the first user has illegally withdrawn from the virtual experience in response to neither input of ending the virtual experience nor input of not ending the virtual experience is executed. The program according to any one of 1 to 6. The UI includes a warning to the effect that the first user is considered to have been illegally withdrawn from the virtual experience if the first user does not return to the virtual experience or normally exits. 7. The program according to any one of 7. A step of causing at least one of an audio output device and an image output device associated with the image display device to perform an output for urging a normal departure from the virtual experience when the removal operation is detected. The program according to any one of items 1 to 8. A computer for providing the first user with a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user,
By the control of the processor included in the computer,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
And a step of causing the image display device to output a UI (User Interface) prompting an input as to whether or not to end the virtual experience when the removal operation is detected. Computer-implemented for providing to the first user a virtual experience in a virtual space in which a plurality of users including the first user participate via an image display device associated with the head of the first user. Method,
Defining the virtual space,
Providing the virtual space to the first user via the image display device;
Detecting a removal operation of removing the image display device by the first user,
Outputting a UI (User Interface) to the image display device for prompting an input as to whether or not to end the virtual experience when the removal operation is detected.