JP7474822B2 - COMPUTER-IMPLEMENTED METHOD, PROGRAM, AND COMPUTER FOR PROVIDING A VIRTUAL EXPERIENCE - Patent application - Google Patents

Description

Patent Law Article 30, Paragraph 2 applies. April 14-15, 2017 VRLA Expo 2017 COLOPL, Inc. exhibited a trial version of the VR action game "TITAN SLAYER."

The present disclosure relates to a method, program, and computer executed on a computer for providing a virtual experience.

The following Patent Document 1 discloses a technology that allows a user to manipulate a virtual hand and have it handle an item, such as a sword object, used in a virtual space, providing the user with a virtual experience as if the user were fighting an enemy object in a virtual space.

Japanese Patent No. 5996138

However, the technology disclosed in Patent Document 1 does not disclose any idea of how to update the state of items used in a virtual space. Therefore, the technology disclosed in Patent Document 1 leaves room for improvement in the virtual experience that users have.

The present disclosure aims to provide a method, a program, and a computer that can be executed on a computer to provide a virtual experience that can improve a user's virtual experience.

According to one aspect of the present disclosure, there is provided a computer-executed method for providing a user with a virtual experience, the method including the steps of: defining a virtual space for providing the virtual experience; moving a virtual viewpoint within the virtual space in response to a movement of the user's head; moving a control object within the virtual space in response to a movement of a part of the user's body; and updating a state of an item used within the virtual space, at least on condition that the control object has been moved out of the range of view from the virtual viewpoint.

According to the present disclosure, a method, program, and computer executed on a computer can be provided for providing a virtual experience that can improve the virtual experience of a user.

FIG. 1 is a diagram illustrating an outline of the configuration of an HMD system according to an embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer according to an aspect. FIG. 3 is a diagram conceptually showing a uvw field of view coordinate system set in an HMD device according to an embodiment. FIG. 4 is a diagram conceptually illustrating one mode of expressing a virtual space according to an embodiment. FIG. 5 is a top view of the head of a user wearing an HMD device according to an embodiment. FIG. 6 is a diagram showing a YZ cross section of the visual field in the virtual space as viewed from the X direction. FIG. 7 is a diagram showing an XZ cross section of the visual field in the virtual space as viewed from the Y direction. FIG. 8 is a diagram illustrating a schematic configuration of a controller according to an embodiment. FIG. 9 is a block diagram illustrating a modular configuration of a computer according to one embodiment. FIG. 10 is a flowchart showing a process executed by the HMD system used by a user to provide the user with a virtual space. Fig. 11(A) is a diagram showing an example of a user wearing an HMD device and holding a controller, Fig. 11(B) is a diagram showing an example of a virtual camera, a left hand object, and a right hand object arranged in a virtual space in the state shown in Fig. 11(A). FIG. 12 is a diagram showing an example of a field of view image that represents the virtual space shown in FIG. 11B in the field of view area of a virtual camera. Fig. 13(A) is a diagram showing an example of a state in which a user has moved their right hand holding a controller to near their right shoulder which is outside the user's field of vision, Fig. 13(B) is a diagram showing an example of a virtual camera, a left hand object, and a right hand object arranged in a virtual space in the state shown in Fig. 13(A). FIG. 14 is a diagram showing an example of a field of view image that represents the virtual space shown in FIG. 13B in the field of view area of a virtual camera. Fig. 15(A) is a diagram showing an example of a state in which a user moves their right hand holding a controller into the user's field of view while performing a selection operation, Fig. 15(B) is a diagram showing an example of a virtual camera, a left hand object, and a right hand object arranged in a virtual space in the state shown in Fig. 15(A). FIG. 16 is a diagram showing an example of a field of view image that represents the virtual space shown in FIG. 15B in the field of view area of a virtual camera. FIG. 17 is a flowchart showing an example of an update process for updating the state of an item in this embodiment. FIG. 18 is a flowchart showing an example of the item effect activation process in this embodiment. Fig. 19(A) is a diagram showing an example of a state in which a user has moved their right hand holding a controller out of the user's field of vision, and Fig. 19(B) is a diagram showing an example of a virtual camera, a left hand object, and a right hand object arranged in a virtual space in the state shown in Fig. 19(A). FIG. 20 is a diagram showing an example of a field of view image that represents the virtual space shown in FIG. 19(B) in the field of view area of a virtual camera. FIG. 21 is a flowchart showing an example of an update process for updating the state of an item in this embodiment. FIG. 22 is a diagram showing an example of a UI board, a left hand object, and a right hand object that are arranged in a virtual space and are used to associate an item with a first area. FIG. 23 is a diagram showing an example of a state in which a specific object is selected by a right hand object. FIG. 24 is a diagram showing an example of a state in which a specific object is placed in the second area by a right hand object. FIG. 25 is a diagram showing an example of a state in which a specific object is placed in the second area. FIG. 26 is a flowchart showing an example of an update process for updating the state of an item in this embodiment.

<Details of the embodiment of the present disclosure>
Hereinafter, the details of the embodiment of the present disclosure will be described with reference to the drawings. In the following description, the same components are basically given the same reference numerals. Therefore, as for components that have already been described (components that have already been described and are given reference numerals), the description will not be repeated in principle unless necessary.

[Configuration of HMD system]
The configuration of an HMD (Head Mount Device) system 100 will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an outline of the configuration of an HMD system 100 according to an embodiment. In one aspect, the HMD system 100 is provided as a system for home use or a system for commercial use.

The HMD system 100 includes an HMD device 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD device 110 includes a display 112, a camera 116, a microphone 118, and a gaze sensor 140. The controller 160 may include a motion sensor 130.

In one aspect, the computer 200 can be connected to the Internet or other network 19 and can communicate with a server 150 or other computers connected to the network 19. In another aspect, the HMD device 110 can include a sensor 114 instead of the HMD sensor 120.

The HMD device 110 is worn on the user's head and can provide the user with a virtual space during operation. More specifically, the HMD device 110 displays an image for the right eye and an image for the left eye on the display 112. When each eye of the user views the respective image, the user can recognize the image as a three-dimensional image based on the parallax between the two eyes. The display 112 can be configured integrally with the HMD device 110 or can be a separate entity.

The display 112 is realized, for example, as a non-transparent display device. In one aspect, the display 112 is disposed on the main body of the HMD device 110 so as to be located in front of both eyes of the user. Thus, when the user visually recognizes the three-dimensional image displayed on the display 112, the user can be immersed in the virtual space. In one embodiment, the virtual space includes, for example, images of a background, objects, and a menu that the user can select. In one embodiment, the display 112 can be realized as a liquid crystal display or an organic EL (Electro Luminescence) display provided in a so-called smartphone or other information display terminal.

In one aspect, the display 112 may include a sub-display for displaying an image for the right eye and a sub-display for displaying an image for the left eye. In another aspect, the display 112 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the display 112 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is perceived by only one of the eyes.

The camera 116 acquires a facial image of the user wearing the HMD device 110. The facial image acquired by the camera 116 can be used to detect the user's facial expression by image analysis processing. The camera 116 may be, for example, an infrared camera built into the main body of the HMD device 110 to detect pupil movement, opening and closing of eyelids, eyebrow movement, etc. Alternatively, the camera 116 may be an external camera disposed outside the HMD device 110 as shown in FIG. 1 to detect movement of the user's mouth, cheeks, chin, etc. Also, the camera 116 may be composed of both the infrared camera and the external camera described above.

The microphone 118 captures the voice uttered by the user. The voice captured by the microphone 118 can be used to detect the user's emotions through voice analysis processing. The voice can also be used to give voice instructions to the virtual space. The voice can also be sent to an HMD system used by another user via the network 19 and the server 150, etc., and output from a speaker or the like connected to the HMD system. This allows conversation (chat) between users sharing the virtual space.

The HMD sensor 120 includes multiple light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared light. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD device 110. The HMD sensor 120 uses this function to detect the position and inclination of the HMD device 110 in real space.

In another aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and inclination of the HMD device 110 by performing image analysis processing using image information of the HMD device 110 output from the camera.

In another aspect, the HMD device 110 may include a sensor 114 as a position detector instead of the HMD sensor 120. The HMD device 110 may detect the position and inclination of the HMD device 110 itself using the sensor 114. For example, if the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, or a gyro sensor, the HMD device 110 may detect its own position and inclination using any of these sensors instead of the HMD sensor 120. As an example, if the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD device 110 around the three axes in the real space over time. The HMD device 110 calculates the change in the angle of the HMD device 110 around the three axes over time based on each angular velocity, and further calculates the inclination of the HMD device 110 based on the change in the angle over time. The HMD device 110 may also include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance. Also, an element presenting real space may be included in the field of view image displayed on the display 112. For example, an image captured by a camera mounted on the HMD device 110 may be superimposed on a part of the field of view image, or the transmittance of a part of the transmissive display device may be set high to make it possible to view real space from a part of the field of view image.

The gaze sensor 140 detects the direction in which the gaze of the right and left eyes of the user 190 is directed (gaze direction). The detection of the direction is realized, for example, by a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right and left eyes of the user 190 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. The gaze sensor 140 can detect the gaze direction of the user 190 based on each detected rotation angle.

The server 150 can send a program to the computer 200 to provide a virtual space to the user.

In another aspect, the server 150 may communicate with other computers 200 to provide virtual reality to HMD devices used by other users. For example, in an amusement facility, when multiple users play a participatory game, each computer 200 communicates signals based on the actions of each user with the other computers 200, allowing multiple users to enjoy a common game in the same virtual space.

The controller 160 accepts input of commands from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be held by the user 190. In another aspect, the controller 160 is configured to be wearable on the body or a part of the clothing of the user 190. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal sent from the computer 200. In another aspect, the controller 160 accepts operations given by the user 190 to control the position, movement, etc. of an object placed in a space that provides virtual reality.

In one aspect, the motion sensor 130 is attached to the user's hand and detects the movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, number of rotations, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided, for example, in a glove-type controller 160. In one embodiment, for safety in the real space, it is desirable that the controller 160 is attached to something that is not easily blown away by being worn on the hand of the user 190, such as a glove-type controller. In another aspect, a sensor that is not attached to the user 190 may detect the movement of the user 190's hand. For example, a signal from a camera that captures the user 190 may be input to the computer 200 as a signal representing the movement of the user 190. The motion sensor 130 and the computer 200 are connected to each other by wire or wirelessly. In the case of wireless, the communication form is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

[Hardware configuration]
A computer 200 according to the present embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of a hardware configuration of computer 200 according to one aspect. Computer 200 includes, as main components, a processor 10, a memory 11, a storage 12, an input/output interface 13, and a communication interface 14. Each component is connected to a bus 15.

The processor 10 executes a series of instructions contained in a program stored in the memory 11 or storage 12 based on a signal provided to the computer 200 or based on the satisfaction of a predetermined condition. In one aspect, the processor 10 is realized as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array), or other device.

Memory 11 temporarily stores programs and data. At least one of the programs and data is loaded from storage 12, for example. Data stored in memory 11 includes data input to computer 200 and data generated by processor 10. In one aspect, memory 11 is realized as RAM (Random Access Memory) or other volatile memory.

The storage 12 permanently holds programs and data. The storage 12 is realized, for example, as a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The programs stored in the storage 12 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with other computers 200. The data stored in the storage 12 includes data and objects for defining the virtual space.

In another aspect, the storage 12 may be realized as a removable storage device such as a memory card. In yet another aspect, a configuration may be used in which programs and data stored in an external storage device are used instead of the storage 12 built into the computer 200. With such a configuration, for example, in a situation where multiple HMD systems 100 are used, such as an amusement facility, it becomes possible to collectively update programs, data, etc.

In one embodiment, the input/output interface 13 communicates signals between the HMD device 110, the HMD sensor 120, or the motion sensor 130. In one aspect, the input/output interface 13 is realized using a terminal such as a Universal Serial Bus (USB) interface, a Digital Visual Interface (DVI), a High-Definition Multimedia Interface (HDMI (registered trademark)), or other terminal. Note that the input/output interface 13 is not limited to those mentioned above.

In one embodiment, the input/output interface 13 may further communicate with the controller 160. For example, the input/output interface 13 receives an input of a signal output from the motion sensor 130. In another aspect, the input/output interface 13 sends an instruction output from the processor 10 to the controller 160. The instruction instructs the controller 160 to vibrate, output sound, emit light, etc. When the controller 160 receives the instruction, it executes either vibration, sound output, or light emission in accordance with the instruction.

The communication interface 14 is connected to the network 19 and communicates with other computers (e.g., server 150) connected to the network 19. In one aspect, the communication interface 14 is realized as, for example, a LAN (Local Area Network) or other wired communication interface, or WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication), or other wireless communication interface. Note that the communication interface 14 is not limited to the above.

In one aspect, the processor 10 accesses the storage 12, loads one or more programs stored in the storage 12 into the memory 11, and executes a series of instructions contained in the programs. The one or more programs may include an operating system for the computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space using the controller 160, and the like. The processor 10 sends a signal for providing the virtual space to the HMD device 110 via the input/output interface 13. The HMD device 110 displays an image on the display 112 based on the signal.

The server 150 is connected to the control devices of each of the multiple HMD systems 100 via the network 19.

2 shows a configuration in which the computer 200 is provided outside the HMD device 110, but in another aspect, the computer 200 may be built into the HMD device 110. As an example, a portable information and communication terminal (e.g., a smartphone) including a display 112 may function as the computer 200.

The computer 200 may also be configured to be shared by multiple HMD devices 110. With such a configuration, for example, the same virtual space can be provided to multiple users, allowing each user to enjoy the same application as other users in the same virtual space. In such a case, the multiple HMD systems 100 in this embodiment may be directly connected to the computer 200 via the input/output interface 13. Furthermore, each function of the server 150 in this embodiment may be implemented in the computer 200.

In one embodiment, a global coordinate system is preset in the HMD system 100. The global coordinate system has three reference directions (axes) that are parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-to-back direction perpendicular to both the vertical and horizontal directions. In this embodiment, the global coordinate system is one of the viewpoint coordinate systems. Thus, the horizontal direction, the vertical direction (up-down direction), and the front-to-back direction in the global coordinate system are defined as the x-axis, y-axis, and z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction in real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-to-back direction in real space.

In one aspect, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD device 110, it detects the presence of the HMD device 110. The HMD sensor 120 further detects the position and inclination of the HMD device 110 in real space according to the movement of the user 190 wearing the HMD device 110 based on the values of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect temporal changes in the position and inclination of the HMD device 110 using each value detected over time.

The global coordinate system is parallel to the coordinate system of the real space. Therefore, each tilt of the HMD device 110 detected by the HMD sensor 120 corresponds to each tilt around the three axes of the HMD device 110 in the global coordinate system. The HMD sensor 120 sets a uvw field of view coordinate system for the HMD device 110 based on the tilt of the HMD device 110 in the global coordinate system. The uvw field of view coordinate system set for the HMD device 110 corresponds to the viewpoint coordinate system when a user 190 wearing the HMD device 110 views an object in a virtual space.

[uvw visual coordinate system]
The uvw field of view coordinate system will be described with reference to Fig. 3. Fig. 3 is a diagram conceptually showing the uvw field of view coordinate system set in the HMD device 110 according to an embodiment. The HMD sensor 120 detects the position and inclination of the HMD device 110 in the global coordinate system when the HMD device 110 is started up. The processor 10 sets the uvw field of view coordinate system in the HMD device 110 based on the detected values.

As shown in FIG. 3, the HMD device 110 sets a three-dimensional uvw field of view coordinate system with the head of the user wearing the HMD device 110 as the center (origin). More specifically, the HMD device 110 tilts the horizontal, vertical, and front-back directions (x-axis, y-axis, z-axis) that define the global coordinate system around each axis by the tilt of the HMD device 110 around each axis in the global coordinate system, and sets the three newly obtained directions as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw field of view coordinate system in the HMD device 110.

In a certain situation, when a user 190 wearing the HMD device 110 stands upright and looks straight ahead, the processor 10 sets a uvw field of view coordinate system parallel to the global coordinate system in the HMD device 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-to-back direction (z-axis) in the global coordinate system coincide with the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw field of view coordinate system in the HMD device 110.

After the uvw field of view coordinate system is set in the HMD device 110, the HMD sensor 120 can detect the tilt (amount of change in tilt) of the HMD device 110 in the set uvw field of view coordinate system based on the movement of the HMD device 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD device 110 in the uvw field of view coordinate system as the tilt of the HMD device 110. The pitch angle (θu) represents the tilt angle of the HMD device 110 around the pitch direction in the uvw field of view coordinate system. The yaw angle (θv) represents the tilt angle of the HMD device 110 around the yaw direction in the uvw field of view coordinate system. The roll angle (θw) represents the tilt angle of the HMD device 110 around the roll direction in the uvw field of view coordinate system.

The HMD sensor 120 sets the uvw field of view coordinate system of the HMD device 110 after the HMD device 110 moves in the HMD device 110 based on the detected tilt angle of the HMD device 110. The relationship between the HMD device 110 and the uvw field of view coordinate system of the HMD device 110 is always constant regardless of the position and tilt of the HMD device 110. When the position and tilt of the HMD device 110 change, the position and tilt of the uvw field of view coordinate system of the HMD device 110 in the global coordinate system change in conjunction with the change in the position and tilt.

In one aspect, the HMD sensor 120 may identify the position of the HMD device 110 in real space as a relative position with respect to the HMD sensor 120 based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between multiple points (e.g., the distance between each point). In addition, the processor 10 may determine the origin of the uvw field of view coordinate system of the HMD device 110 in real space (global coordinate system) based on the identified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. 4. FIG. 4 is a diagram conceptually showing one aspect of expressing the virtual space 2 according to an embodiment. The virtual space 2 has a spherical structure covering the entire 360-degree direction of the center 21. In FIG. 4, in order to avoid complicating the description, the upper half of the celestial sphere in the virtual space 2 is illustrated. Meshes are defined in the virtual space 2. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system defined in the virtual space 2. The computer 200 provides the user with a virtual space 2 in which virtual space elements visible by the user are deployed by associating each element (e.g., object, still image, video, etc.) that can be deployed (placed) in the virtual space 2 with a mesh corresponding to the deployment position (placement position) in the virtual space 2. The virtual space elements are elements deployed in the virtual space 2, and include, for example, various objects (e.g., objects that can be operated by the user, objects that operate based on a predetermined algorithm, and objects that do not operate, etc.) and various images (background images, menu images selected by the user, photos, video images, etc.).

In a certain aspect, an XYZ coordinate system with the center 21 as the origin is defined in the virtual space 2. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a type of viewpoint coordinate system, the horizontal direction, vertical direction (up and down direction), and front and back directions in the XYZ coordinate system are defined as the X axis, Y axis, and Z axis, respectively. Therefore, the X axis (horizontal direction) of the XYZ coordinate system is parallel to the x axis of the global coordinate system, the Y axis (vertical direction) of the XYZ coordinate system is parallel to the y axis of the global coordinate system, and the Z axis (front and back direction) of the XYZ coordinate system is parallel to the z axis of the global coordinate system.

When the HMD device 110 is started up, i.e., in the initial state of the HMD device 110, the virtual camera 1 is positioned at the center 21 of the virtual space 2. The virtual camera 1 moves in the virtual space 2 in conjunction with the movement of the HMD device 110 in the real space. As a result, the changes in the position and orientation of the HMD device 110 in the real space are reproduced in the virtual space 2 in the same manner.

A uvw field of view coordinate system is defined for the virtual camera 1, as in the case of the HMD device 110. The uvw field of view coordinate system of the virtual camera 1 in the virtual space 2 is defined so as to be linked to the uvw field of view coordinate system of the HMD device 110 in the real space (global coordinate system). Therefore, when the tilt of the HMD device 110 changes, the tilt of the virtual camera 1 also changes accordingly. In addition, the virtual camera 1 may move in the virtual space 2 in conjunction with the movement in the real space of the user wearing the HMD device 110, or may move in response to the operation of the user 190 accepted by the controller 160.

The orientation of the virtual camera 1 is determined according to the position and inclination of the virtual camera 1, and therefore the line of sight (reference line of sight 5) that serves as a reference when the user views the virtual space elements is determined according to the orientation of the virtual camera 1. However, the orientation of the virtual camera 1 may also be determined according to the line of sight direction of the user 190 detected by the gaze sensor 140. The processor 10 of the computer 200 defines the field of view 23 in the virtual space 2 based on the reference line of sight 5. The field of view 23 corresponds to the field of view of the user wearing the HMD device 110 in the virtual space 2.

The gaze direction of the user 190 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 190 views an object. The uvw field of view coordinate system of the HMD device 110 is equal to the viewpoint coordinate system when the user 190 views the display 112. In addition, the uvw field of view coordinate system of the virtual camera 1 is linked to the uvw field of view coordinate system of the HMD device 110. Therefore, in a certain aspect, the HMD system 100 can consider the gaze direction of the user 190 detected by the gaze sensor 140 to be the user's gaze direction in the uvw field of view coordinate system of the virtual camera 1.

[User's gaze]
Determining the user's line of sight direction will be described with reference to Fig. 5. Fig. 5 is a top view of the head of a user 190 wearing an HMD device 110 according to an embodiment.

In one aspect, the gaze sensor 140 detects the gaze of each of the right and left eyes of the user 190. In one aspect, when the user 190 is looking at something close, the gaze sensor 140 detects gazes R1 and L1. In another aspect, when the user 190 is looking at something far away, the gaze sensor 140 detects gazes R2 and L2. In this case, the angle that the gazes R2 and L2 make with respect to the roll direction w is smaller than the angle that the gazes R1 and L1 make with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the gaze detection result, the computer 200 identifies the gaze point N1, which is the intersection of the lines of sight R1 and L1, based on the detection values. On the other hand, when the computer 200 receives the detection values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 identifies the gaze direction N0 of the user 190 based on the position of the identified gaze point N1. For example, the computer 200 detects the direction of the line that passes through the midpoint of the line connecting the right eye R and the left eye L of the user 190 and the gaze point N1 as the gaze direction N0. The gaze direction N0 is the direction in which the user 190 is actually looking with both eyes. The gaze direction N0 also corresponds to the direction in which the user 190 is actually looking with respect to the field of view 23.

In another aspect, the HMD system 100 may include a television broadcast receiving tuner. With such a configuration, the HMD system 100 can display television programs in the virtual space 2.

In yet another aspect, the HMD system 100 may be equipped with a communication circuit for connecting to the Internet or a calling function for connecting to a telephone line.

[View Area]
The field of view 23 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing a YZ cross section of the field of view 23 seen from the X direction in the virtual space 2. Fig. 7 is a diagram showing an XZ cross section of the field of view 23 seen from the Y direction in the virtual space 2.

As shown in FIG. 6, the field of view 23 in the YZ cross section includes an area 24. The area 24 is defined by the reference line of sight 5 of the virtual camera 1 and the YZ cross section of the virtual space 2. The processor 10 defines the range including the polar angle α centered on the reference line of sight 5 in the virtual space 2 as the area 24.

As shown in FIG. 7, the field of view 23 in the XZ cross section includes an area 25. The area 25 is defined by the reference line of sight 5 and the XZ cross section of the virtual space 2. The processor 10 defines the range including the azimuth angle β centered on the reference line of sight 5 in the virtual space 2 as the area 25.

In a certain aspect, the HMD system 100 provides a virtual space to the user 190 by displaying a field of view image on the display 112 based on a signal from the computer 200. The field of view image is an image representing virtual space elements in a space corresponding to the field of view 23 among the virtual space elements deployed in the virtual space 2. When the user 190 moves the HMD device 110 worn on the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the field of view 23 in the virtual space 2 changes. As a result, the field of view image displayed on the display 112 is updated from an image representing the virtual space elements in the space corresponding to the field of view 23 before the user 190 moves the HMD device 110 to an image representing the virtual space elements in the space corresponding to the field of view 23 after the user 190 moves the HMD device 110. The user can visually recognize a desired direction in the virtual space 2.

While wearing the HMD device 110, the user 190 can see the virtual space elements deployed in the virtual space 2 without seeing the real world. Therefore, the HMD system 100 can give the user a high sense of immersion in the virtual space 2.

In one aspect, the processor 10 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement in real space of the user 190 wearing the HMD device 110. In this case, the processor 10 specifies the image area to be projected onto the display 112 of the HMD device 110 (i.e., the field of view 23 in the virtual space 2) based on the position and orientation of the virtual camera 1 in the virtual space 2. In other words, the field of view of the user 190 in the virtual space 2 is defined by the virtual camera 1.

According to one embodiment, it is preferable that the virtual camera 1 includes two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. It is also preferable that an appropriate parallax is set for the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In this embodiment, the technical idea of the present disclosure is illustrated by assuming that the virtual camera 1 includes two virtual cameras and is configured so that the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD device 110.

[controller]
An example of the controller 160 will be described with reference to Fig. 8. Fig. 8 is a diagram showing a schematic configuration of the controller 160 according to an embodiment.

As shown in state (A) of FIG. 8, in one aspect, the controller 160 may include a right controller 160R and a left controller. The right controller 160R is operated by the right hand of the user 190. The left controller is operated by the left hand of the user 190. In one aspect, the right controller 160R and the left controller are configured symmetrically as separate devices. Thus, the user 190 can freely move both the right hand holding the right controller 160R and the left hand holding the left controller. In another aspect, the controller 160 may be an integrated controller that accepts operation from both hands. The right controller 160R will be described below.

The right controller 160R includes a grip 30, a frame 31, and a top surface 32. The grip 30 is configured to be held by the right hand of the user 190. For example, the grip 30 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the user 190's right hand.

The grip 30 includes buttons 33, 34 and a motion sensor 130. The button 33 is located on the side of the grip 30 and is operated by the middle finger of the right hand. The button 34 is located on the front of the grip 30 and is operated by the index finger of the right hand. In one aspect, the buttons 33, 34 are configured as trigger-type buttons. The motion sensor 130 is built into the housing of the grip 30. Note that if the movements of the user 190 can be detected from around the user 190 by a camera or other device, the grip 30 does not need to include the motion sensor 130.

The frame 31 includes a plurality of infrared LEDs 35 arranged along its circumferential direction. The infrared LEDs 35 emit infrared light in accordance with the progress of a program using the controller 160 during execution of the program. The infrared light emitted from the infrared LEDs 35 can be used to detect the positions and attitudes (tilt, direction) of the right controller 160R and the left controller. In the example shown in FIG. 8, the infrared LEDs 35 are arranged in two rows, but the number of rows is not limited to that shown in FIG. 8. An arrangement of one row or three or more rows may be used. In the example shown in FIG. 8, the controller 160 is a controller that the user holds in his/her hand, so that the movement of the user's hand can be detected by detecting the position and attitude (tilt, direction) of the controller 160. Note that, if the controller 160 can be attached to the body or a part of the clothing of the user 190, the movement of the body or a part of the clothing of the user 190 can be detected by detecting the position and attitude (tilt, direction) of the controller 160.

The top surface 32 includes buttons 36, 37 and an analog stick 38. The buttons 36, 37 are configured as push buttons. The buttons 36, 37 are operated by the thumb of the right hand of the user 190. In a certain situation, the analog stick 38 is operated in any direction within 360 degrees from the initial position (neutral position). Such operations include, for example, operations for moving an object placed in the virtual space 2.

In one aspect, the right controller 160R and the left controller include a battery for driving the infrared LED 35 and other components. The battery may be, but is not limited to, a rechargeable battery, a button type battery, a dry cell battery, or the like. In another aspect, the right controller 160R and the left controller may be connected to, for example, a USB interface of the computer 200. In this case, the right controller 160R and the left controller do not require a battery.

8, for example, the yaw, roll, and pitch directions are defined for the right hand 810 of the user 190. When the user 190 extends his/her thumb and index finger, the extending direction of the thumb is defined as the yaw direction, the extending direction of the index finger is defined as the roll direction, and the direction perpendicular to the plane defined by the axis of the yaw direction and the axis of the roll direction is defined as the pitch direction.
[Control device of HMD device]

The control device of the HMD device 110 will be described with reference to FIG. 9. In one embodiment, the control device is realized by a computer 200 having a known configuration. FIG. 9 is a block diagram showing the computer 200 according to one embodiment as a modular configuration.

As shown in FIG. 9, the computer 200 includes a main control module 220, a memory module 240, and a communication control module 250. The main control module 220 includes, as sub-modules, a virtual space control module 221, a virtual camera control module 222, an object control module 223, a collision determination module 224, a parameter control module 225, a field of view image generation module 226, and a display control module 227. However, the main control module 220 does not need to include all of the above-mentioned modules, and may not include some of them.

In one embodiment, the main control module 220 is implemented by the processor 10. In another embodiment, multiple processors 10 may operate as the main control module 220. The memory module 240 is implemented by the memory 11 or the storage 12. The communication control module 250 is implemented by the communication interface 14.

The virtual space control module 221 controls the virtual space 2 provided to the user 190. In this embodiment, the virtual space control module 221 specifies the virtual space 2 in the HMD system 100 by identifying virtual space data representing the virtual space 2. Note that in this embodiment, an example is described in which the placement of virtual space elements such as the virtual camera 1, various objects, and background images in the virtual space 2 is performed by a module other than the virtual space control module 221, but this is not limited to this, and the virtual space control module 221 may perform the placement. In addition, the placement of the virtual space elements in the virtual space 2 may be performed for the entire virtual space 2, or may be limited to a viewing area 23 determined by the virtual camera control module 222 described later.

The virtual camera control module 222 places the virtual camera 1 in the virtual space 2 and controls the operation of the virtual camera 1 in the virtual space 2. In this embodiment, the virtual camera control module 222 can control the behavior, orientation, etc. of the virtual camera 1 in the virtual space 2 based on the head orientation of the user wearing the HMD device 110, i.e., the movement of the HMD device 110. In other words, the field of view 23 of the virtual camera 1 is determined based on the head orientation of the user wearing the HMD device 110. However, the operation control of the virtual camera 1 is not limited to this. For example, it may be determined based on key operations of the controller 160 by the user.

The object control module 223 places objects in the virtual space 2 and controls the actions of the objects in the virtual space 2. The objects may be any virtual object that can be placed in the virtual space 2. Examples of objects include, but are not limited to, control objects, target objects such as items handled by the control objects, enemy characters, and background objects.

The operation object is an object associated with a user wearing the HMD device 110 and whose operation is controlled by the user. The operation object may be, for example, a player character itself associated with the user wearing the HMD device 110, or an object constituting a part of the body of the player character. The player character is an alter ego in the virtual space 2 of the user wearing the HMD device 110, and may also be called an avatar. In this embodiment, the operation object is a virtual hand associated with the hand of the user wearing the HMD device 110, that is, a hand object constituting the hand of the player character, as an example. In this case, the object control module 223 operates the hand object based on the movement of the controller 160 held by the user in the hand or the key operation of the controller 160 by the user. However, the operation object is not limited to a hand object. For example, the operation object may be a finger object (virtual finger), a foot object, or a stick object corresponding to a stick used by the user, instead of a hand object. When the operation object is a finger object, in particular, the operation object corresponds to the axis part of the direction (axial direction) in which the finger points. In addition, if the control object is an object that constitutes part of the player character's body, it does not matter whether or not the various objects that make up the player character's body other than the control object are placed.

The target object may be any object that is directly or indirectly operated (handled) by the operation object. Examples of the target object include items used in a virtual space. Examples of the items include virtual objects used in the progress of the game. However, it does not matter whether the object is a tangible or intangible object. For example, in an action game in which an enemy character is attacked or attacked by an enemy character, items include, but are not limited to, weapon objects for attacking enemy characters and armor objects for preventing attacks from enemy characters. Examples of weapon objects include, but are not limited to, objects that imitate guns, swords, spears, axes, bows, and grenades. Examples of armor objects include, but are not limited to, objects that imitate shields and armor. Note that the weapon object and armor object may be objects that activate their effect regardless of parameters, such as contact with other objects, or objects that activate their effect by consuming associated parameters. For example, if the weapon object is a gun object that imitates a gun, the parameter associated with the gun object is the remaining number of bullet objects fired from the gun object. In this case, if there are any remaining bullet objects, the gun object will consume the remaining bullet objects and fire them.

An enemy character is an object that influences the player character. In an action game such as that described above, an enemy character may be, for example, an object that attacks the player character or is attacked by the player character. An enemy character may be an NPC (Non Player Character) whose actions are controlled by a specific program, or an object controlled by another user. Background objects may include, for example, landscapes including forests, mountains, and other objects that are placed according to the progress of the game story, animals, and the like. Note that the object control module 223 may control non-moving objects to remain placed in fixed positions.

The collision determination module 224 determines a collision between objects by determining a collision (contact) in a collision area between the objects. The collision determination module 224 can, for example, detect the timing when one object touches another object, or detect the timing when the objects stop touching. In this embodiment, the collision determination module 224 determines a collision between the control object and various areas. Specifically, the collision determination module 224 determines a collision between the control object and various areas by determining a collision (contact) between a collision area set for the control object and various areas formed by the collision area.

The parameter control module 225 controls various parameters. As described above, if an item becomes effective by consuming an associated parameter, the parameter control module 225 controls the remaining amount of the parameter when the item becomes effective.

The field of view image generation module 226 generates field of view image data to be displayed on the display 112 based on the field of view area 23 determined by the virtual camera control module 222. Specifically, the field of view image generation module 226 generates field of view image data based on the field of view of virtual camera 1 defined based on the movement of virtual camera 1, virtual space data specified by the virtual space control module 221, etc.

The display control module 227 displays the field of view image on the display 112 of the HMD device 110 based on the field of view image data generated by the field of view image generation module 226. For example, the display control module 227 displays the field of view image on the display 112 by outputting the field of view image data generated by the field of view image generation module 226 to the HMD device 110.

The memory module 240 holds data used by the computer 200 to provide the virtual space 2 to the user 190. In one aspect, the memory module 240 holds space information 241, object information 242, and user information 243. The space information 241 includes, for example, one or more templates defined for providing the virtual space 2. The object information 242 includes, for example, content to be played in the virtual space 2, information for arranging objects used in the content, and other attribute information such as drawing data of the player character and its size information. The content may include, for example, a game, content showing a landscape similar to that of the real world, and the like. The user information 243 includes, for example, a program for making the computer 200 function as a control device for the HMD system 100, an application program that uses each content held in the object information 242, and the like.

The data and programs stored in the memory module 240 are input by the user of the HMD device 110. Alternatively, the processor 10 downloads the programs or data from a computer (e.g., a server 150) operated by the operator providing the content, and stores the downloaded programs or data in the memory module 240.

The communication control module 250 can communicate with the server 150 and other information and communication devices via the network 19.

In one aspect, the main control module 220 can be realized using, for example, Unity (registered trademark) provided by Unity Technologies, Inc. In another aspect, the main control module 220 can be realized as a combination of circuit elements that realize each process.

Processing in computer 200 is realized by hardware and software executed by processor 10. Such software may be stored in advance on a hard disk or other memory module 240. In addition, the software may be stored on a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other network. Such software is read from the data recording medium by an optical disk drive or other data reading device, or downloaded from server 150 or other computer via communication control module 250, and then temporarily stored in memory module 240. The software is read from memory module 240 by processor 10 and stored in RAM in the form of an executable program. Processor 10 executes the program.

The hardware constituting the computer 200 shown in FIG. 9 is general. The operation of the hardware of the computer 200 is well known, so a detailed description will not be repeated.

The data recording medium is not limited to CD-ROM, FD (Flexible Disk), and hard disk, but may also be a non-volatile data recording medium that holds a program in a fixed manner, such as magnetic tape, cassette tape, optical disk (MO (Magnetic Optical Disc)/MD (Mini Disc)/DVD (Digital Versatile Disc)), IC (Integrated Circuit) card (including memory card), optical card, mask ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash ROM, and other semiconductor memory.

The program referred to here may include not only programs that can be executed directly by the processor 10, but also programs in source program format, compressed programs, encrypted programs, etc.

[Control Structure]
A control structure of computer 200 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing a process executed by HMD system 100 used by user 190 to provide virtual space 2 to user 190.

In step S1, the processor 10 of the computer 200, as the virtual space control module 221, identifies virtual space data and defines the virtual space 2. The processor 10, as the object control module 223, also places various objects in this virtual space 2. The virtual space control module 221 defines the operations of the objects so that they can be controlled within the virtual space 2.

In step S2, the processor 10 initializes the virtual camera 1 as the virtual camera control module 222. For example, the processor 10 places the virtual camera 1 in a work area of the memory at a predefined center point in the virtual space 2, and directs the line of sight of the virtual camera 1 in the direction in which the user 190 is facing.

In step S3, the processor 10, as the field of view image generation module 226, generates field of view image data for displaying the initial field of view image. The display control module 227 transmits (outputs) the generated field of view image data to the HMD device 110 via the communication control module 250.

In step S4, the display 112 of the HMD device 110 displays a field of view image based on the field of view image data received from the computer 200. The user 190 wearing the HMD device 110 can recognize the virtual space 2 by visually recognizing the field of view image.

In step S5, the HMD sensor 120 detects the position and inclination of the HMD device 110 based on the multiple infrared lights emitted from the HMD device 110. The detection results are sent to the computer 200 as motion detection data.

In step S6, the processor 10, as the virtual camera control module 222, determines the field of view direction of the user 190 wearing the HMD device 110 based on the position and inclination of the HMD device 110.

In step S7, the controller 160 detects an operation by the user 190 in real space. For example, in one aspect, the controller 160 detects that a button has been pressed by the user 190. In another aspect, the controller 160 detects a hand movement of the user 190. A signal indicating the detection content is sent to the computer 200.

In step S8, the processor 10, as the object control module 223, the collision determination module 224, and the parameter control module 225, reflects the motion detection data sent from the HMD sensor 120, the detection contents sent from the controller 160, and the control contents of various objects in the virtual space 2.

For example, the processor 10, as the object control module 223, moves an operation object in the virtual space 2 based on the motion detection data of the HMD device 110 and the detection contents of the controller 160. Also, for example, the processor 10, as the collision determination module 224, performs collision determination of the collision area of each object. Also, for example, the processor 10, as the parameter control module 225, controls parameters associated with items.

In step S9, the processor 10, as the field of view image generation module 226, generates field of view image data for displaying a field of view image based on the results of the processing in step S8. The display control module 227 outputs the generated field of view image data to the HMD device 110.

In step S10, the display 112 of the HMD device 110 updates the field of view image based on the received field of view image data and displays the updated field of view image.

[Item status update method]
Next, a method for updating the state of an item in this embodiment will be described. Fig. 11 (A) is a diagram showing an example of a user 190 wearing an HMD device 110 and holding controllers 160L and 160R. Fig. 11 (B) is a diagram showing an example of a virtual camera 1, a left hand object 400L, and a right hand object 400R arranged in a virtual space 2 in the state shown in Fig. 11 (A). Fig. 12 is a diagram showing an example of a field of view image M showing the virtual space 2 shown in Fig. 11 (B) by the field of view area 23 of the virtual camera 1.

In this embodiment, the position and movement of the HMD device 110 in real space is detected by position tracking and reflected in the position and movement of the virtual camera 1 in the virtual space 2. Similarly, the positions and movements of the controllers 160L and 160R in real space are detected by position tracking and reflected in the positions and movements of the left hand object 400L and the right hand object 400R in the virtual space 2. For this reason, in this embodiment, the positional relationship of the left hand (controller 160L) and right hand (controller 160R) relative to the head (HMD device 110) of the user 190 is reflected in the virtual space 2 as the positional relationship of the left hand object 400L and right hand object 400R relative to the virtual camera 1. Therefore, according to this embodiment, the user 190 wearing the HMD device 110 can not only move the left hand object 400L and the right hand object 400R in the virtual space 2 based on the movement of his/her own left hand and right hand, but can also move the left hand object 400L and the right hand object 400R in the virtual space 2 as if they were the user 190's own virtual hands. Thus, according to this embodiment, the user 190 wearing the HMD device 110 can experience the virtual space 2 from a subjective viewpoint (first-person viewpoint), so that the user 190 can enjoy a virtual experience as if he/she were the player character 490. In other words, the user 190 can enjoy a virtual experience as if he/she were the player character 490 and facing and fighting against the enemy character 500. In this embodiment, among the objects constituting the body of the player character 490, a left hand object 400L and a right hand object 400R shown by solid lines in FIG. 11(B) are placed in the virtual space 2, but objects constituting other parts shown by dashed lines in FIG. 11(B) are omitted from placement in the virtual space 2. In the following description, the left hand object 400L and the right hand object 400R may be collectively referred to simply as the "hand object 400."

In this embodiment, the first area is set in a specific part of the body of the player character 490. The first area is, for example, an area configured by a collision area, and is an area to which an item to be used by the hand object 400 can be associated. Examples of the items include, but are not limited to, a weapon object for attacking the enemy character 500 and an armor object for preventing attacks from the enemy character 500. The user 190 moves his/her hand (controller 160) to collide the collision area set in the hand object 400 with the first area, and performs a selection operation to select an item associated with the first area, thereby causing the hand object 400 to select an item associated with the first area. Examples of the selection operation include, but are not limited to, continuously pressing a trigger-type button such as button 33 or 34.

As described above, in this embodiment, the user 190 can enjoy a virtual experience as if he/she were the player character 490 facing and fighting the enemy character 500. Therefore, when attacking the enemy character 500, the user 190 brings the hand object 400 close to the first area associated with the weapon object and performs a selection operation to cause the hand object 400 to select a weapon object, and then operates the selected weapon object with the hand object 400 to attack the enemy character 500. Similarly, when defending against an attack from the enemy character 500, the user 190 brings the hand object 400 close to the first area associated with the armor object and performs a selection operation to cause the hand object 400 to select an armor object, and then operates the selected armor object with the hand object 400 to defend against the attack from the enemy character 500.

Here, in order to improve the virtual experience of the user 190, it is preferable to set the first area to a part of the body of the player character 490 where the player character 490 is expected to actually wear a weapon object or armor object. In addition, if the first area is located outside the range of the field of view 23 of the virtual camera 1 (player character 490), it is possible to omit drawing a graphic in which the player character 490 is wearing a weapon object or armor object, which is also preferable from the viewpoint of processing load. For this reason, in this embodiment, an example will be described in which the first area is set to the left shoulder, right shoulder, left hip, and right hip of the player character 490, but this is not limited to this. Thus, in this embodiment, the first area is an area whose relative position with respect to the virtual camera 1 is fixed and which is located outside the range of the field of view 23.

In the example shown in FIG. 11(B), a gun object is associated with a collision area CB, which is a first area set on the right shoulder of the player character 490. In the example shown in FIG. 11(B), a gun object is illustrated in the collision area CB to make it easier to understand that the gun object is associated with the collision area CB, but it is not necessary to visualize the gun object. Hereinafter, a method of selecting a gun object by the right hand object 400R will be specifically described with reference to FIG. 13 to FIG. 16. FIG. 13(A) is a diagram showing an example of a state in which the user 190 moves the right hand holding the controller 160R to the vicinity of the right shoulder located outside the field of view of the user 190. FIG. 13(B) is a diagram showing an example of the virtual camera 1, the left hand object 400L, and the right hand object 400R arranged in the virtual space 2 in the state shown in FIG. 13(A). FIG. 14 is a diagram showing an example of a field of view image M in which the virtual space 2 shown in FIG. 13(B) is represented by the field of view area 23 of the virtual camera 1. Fig. 15(A) is a diagram showing an example of a state in which a user 190 has moved their right hand holding a controller 160R into the field of view of the user 190 while performing a selection operation. Fig. 15(B) is a diagram showing an example of a virtual camera 1, a left hand object 400L, and a right hand object 400R arranged in a virtual space 2 in the state shown in Fig. 15(A). Fig. 16 is a diagram showing an example of a field of view image M that shows the virtual space 2 shown in Fig. 15(B) in the field of view area 23 of the virtual camera 1.

As shown in FIG. 13(A), the user 190 moves the right hand holding the controller 160R to the vicinity of his/her right shoulder. As a result, as shown in FIG. 13(B), the processor 10 moves the right hand object 400R to the vicinity of the right shoulder of the player character 490, which is located outside the range of the field of view 23 of the virtual camera 1. In this case, since the right hand object 400R is located outside the range of the field of view 23, the field of view image M does not include the right hand object 400R, as shown in FIG. 14. In addition, with the movement of the right hand object 400R, the processor 10 determines whether or not a first positional relationship is established between the right hand object 400R and a first area set on the right shoulder of the player character 490. Specifically, the processor 10 determines whether or not a collision area CA of the right hand object 400R collides with a collision area CB constituting the first area.

When the user 190 moves his/her right hand to the vicinity of his/her right shoulder, the user 190 performs the selection operation described above. As shown in FIG. 13(B), when the collision area CA and the collision area CB are colliding, the processor 10 causes the right hand object 400R to select the gun object while the selection operation continues. As a result, the processor 10, as the object control module 223, updates the state of the gun object from an unselected state in which the gun object is not selected by the right hand object 400R to a selected state in which the gun object is selected by the right hand object 400R.

As shown in FIG. 15(A), the user 190 continues the selection operation while moving their right hand into their field of view. This causes the processor 10 to move the right hand object 400R into the field of view 23, as shown in FIG. 15(B). At this time, the gun object 450 is in a selected state selected by the right hand object 400R, so the gun object 450 is being held by the right hand object 400R. In addition, because the right hand object 400R and the gun object 450 are located within the field of view 23, the right hand object 400R and the gun object 450 are included in the field of view image M, as shown in FIG. 16.

When the gun object 450 is in a selected state, the user 190 performs an effect activation operation to activate the gun object 450. An example of an effect activation operation is a firing operation to fire a bullet object from the gun object 450. By performing a firing operation, the bullet object can be fired from the gun object 450 to attack the enemy object 500. An example of an effect activation operation is, but is not limited to, pressing a push button such as button 36 or 37.

In addition, when the gun object 450 is in a selected state, if the selection operation by the user 190 is released, the selection of the gun object 450 by the right hand object 400R is released. For example, assume that the selection operation is the continuous pressing of a trigger-type button such as button 33 or 34. In this case, when the processor 10 detects that the pressing of the trigger-type button has been released, the processor 10, as the object control module 223, causes the right hand object 400R to release the gun object 450 and updates the state of the gun object from a selected state to a non-selected state.

Figure 17 is a flowchart showing an example of an update process for updating the state of an item in this embodiment, and in detail shows the update process between the selected state and the non-selected state of an item.

In step S11, the processor 10, as the object control module 223, detects the movement of the user 190's hand (controller 160) and moves the hand object 400 in the virtual space 2 in conjunction with the detected hand movement. At this time, the processor 10, as the collision determination module 224, determines whether or not the collision area CA of the hand object 400 after the movement collides with another collision area.

In step S12, the processor 10, as the object control module 223, determines whether or not a selection operation has been performed by the user 190. If a selection operation has not been performed (No in step S12), the process returns to step S11. If a selection operation has been performed (Yes in step S12), the process proceeds to step S13.

In step S13, the processor 10, as the object control module 223, checks whether the collision determination module 224 has determined that the collision area CA of the hand object 400 is colliding with the collision area CB that constitutes the first region. If the collision area CA is not colliding with the collision area CB (No in step S13), the process returns to step S11. If the collision area CA is colliding with the collision area CB (Yes in step S13), the process proceeds to step S14.

In step S14, the processor 10, functioning as the object control module 223, causes the hand object 400 to select an item associated with the first region. As a result, the item is held in the hand object 400. The processor 10, functioning as the object control module 223, also updates the state of the item from a non-selected state to a selected state. However, if no item is associated with the first region, the processor 10 does not cause the hand object 400 to select an item.

In step S15, the processor 10, as the object control module 223, detects the movement of the user 190's hand (controller 160) and moves the hand object 400 in the virtual space 2 in conjunction with the detected hand movement. At this time, the processor 10, as the collision determination module 224, determines whether or not the collision area CA of the hand object 400 after the movement collides with another collision area.

In step S16, the processor 10, as the object control module 223, determines whether the selection operation by the user 190 has been released. If the selection operation has not been released (No in step S16), the process returns to step S15. If the selection operation has been released (Yes in step S16), the process proceeds to step S17.

In step S17, the processor 10, as the object control module 223, cancels the selection of the item by the hand object 400. This causes the item to be released from the hand object 400. The processor 10, as the object control module 223, also updates the state of the item from a selected state to a non-selected state. After this, the process returns to step S11.

Figure 18 is a flowchart showing an example of an item effect activation process in this embodiment. In detail, the flowchart shown in Figure 18 is performed when an item is in a selected state and the item is an item that activates its effect by consuming an associated parameter.

In step S21, the processor 10, functioning as the parameter control module 225, determines whether or not an effect activation operation has been performed by the user 190. If an effect activation operation has not been performed (No in step S21), the process ends. If an effect activation operation has been performed (Yes in step S21), the process proceeds to step S22.

In step S22, the processor 10, as the parameter control module 225, checks whether the amount of parameters associated with the selected item required to activate the effect remains. If the amount of parameters required to activate the effect is not remaining (No in step S22), the process ends. If the amount of parameters required to activate the effect remains (Yes in step S22), the process proceeds to step S23. For example, the item is a gun object 450, the parameter associated with the gun object 450 is the remaining number of bullet objects to be fired from the gun object 450, and the gun object 450 fires a bullet object by consuming one bullet object. In this case, if the remaining number of bullet objects is 1 or more, the amount of parameters required to activate the effect remains, and if the remaining number of bullet objects is 0, the amount of parameters required to activate the effect does not remain.

In step S23, the processor 10, as the object control module 223, activates the effect of the selected item, and as the parameter control module 225, consumes the amount of parameters required to activate the effect. Specifically, the processor 10 subtracts the amount required to activate the effect from the remaining amount of parameters. For example, as described above, if the item is a gun object 450, one bullet object can be fired from the gun object 450 by consuming one bullet object.

Note that the flowchart shown in FIG. 18 is assumed to be executed when the item selected by the hand object 400 is located within the field of view 23, but is not limited to this.

Next, a method of recovering parameters associated with an item will be described with reference to Figs. 19 and 20. In this embodiment, a method of recovering the remaining amount of parameters associated with an item by moving the item selected by the hand object 400 outside the range of the field of view 23 of the virtual camera 1 is adopted. Here, a method of recovering the remaining number of bullet objects associated with the gun object 450 will be described as an example, but the present invention is not limited to this. Fig. 19(A) is a diagram showing an example of a state in which the user 190 has moved the right hand holding the controller 160R outside the field of view of the user 190. Fig. 19(B) is a diagram showing an example of the virtual camera 1, the left hand object 400L, and the right hand object 400R arranged in the virtual space 2 in the state shown in Fig. 19(A). Fig. 20 is a diagram showing an example of a field of view image M showing the virtual space 2 shown in Fig. 19(B) in the field of view 23 of the virtual camera 1.

As shown in FIG. 19(A), the user 190 moves the right hand holding the controller 160R out of the field of view in order to restore the remaining number of bullet objects associated with the gun object 450. As a result, as shown in FIG. 19(B), the processor 10 moves the right hand object 400R and the gun object 450 held by the right hand object 400R out of the field of view 23 of the virtual camera 1. In this case, since the right hand object 400R and the gun object 450 are located outside the field of view 23, as shown in FIG. 20, the right hand object 400R and the gun object 450 are no longer included in the field of view image M. Here, the processor 10, as the parameter control module 225, restores the remaining number of bullet objects associated with the gun object 450 when the gun object 450 moves out of the field of view 23. As a result, the processor 10, as the parameter control module 225, updates the state of the gun object 450 from a first state in which at least a part of the bullet object has been consumed to a second state in which the remaining number of bullet objects is greater than that in the first state. In this embodiment, the processor 10 will be described by taking as an example a case where the remaining number of bullet objects is restored to an upper limit value, but is not limited thereto. The extent to which the remaining number of bullet objects is restored can be set arbitrarily, and it is sufficient that the remaining number of bullet objects is at least greater than the remaining number of bullet objects at the time when the gun object 450 moves out of the range of the field of view 23. In this manner, in this embodiment, when the gun object 450 is moved out of the range of the field of view 23, the remaining number of bullet objects associated with the gun object 450 is restored, so that it is possible to omit drawing a graphic such as reloading the bullet objects into the gun object 450, and to reduce the processing load. Note that the processor 10 may output a sound effect from a speaker or the like connected to the HMD system to inform the user 190 that the bullet objects have been reloaded into the gun object 450 at the timing of restoring the remaining number of bullet objects associated with the gun object 450.

Figure 21 is a flowchart showing an example of an update process for updating the state of an item in this embodiment, and in detail shows an update process from a first state in which a parameter associated with an item has been consumed to a second state in which the consumed parameter has been restored. Note that the flowchart shown in Figure 21 is performed when the item is in a selected state and the item is an item that becomes effective by consuming an associated parameter.

In step S31, the processor 10, as the object control module 223, detects the movement of the user's 190 hand (controller 160) and moves the hand object 400 in the virtual space 2 in conjunction with the detected hand movement.

In step S32, the processor 10, as the parameter control module 225, determines whether or not the item selected by the hand object 400 is located within the field of view 23. If the item is not located within the field of view 23 (No in step S32), the process returns to step S31. If the item is located within the field of view 23 (Yes in step S32), the process proceeds to step S33.

In step S33, the processor 10, as the parameter control module 225, determines whether the remaining amount of the parameter associated with the item selected by the hand object 400 is full. If the remaining amount of the parameter is full (Yes in step S33), the process returns to step S31. If the remaining amount of the parameter is not full (No in step S33), the process proceeds to step S34.

In step S33, the processor 10, as the parameter control module 225, fills up the remaining amount of the parameter associated with the item.

Next, a method for associating an item with a first area will be described with reference to Figures 22 to 25. In this embodiment, a method for associating an item with a first area will be described using a specific object for identifying the item and a character model representing the player character 490 as an example, but the method for associating an item with a first area is not limited to this.

22 is a diagram showing an example of a UI board 300, a left hand object 400L, and a right hand object 400R arranged in the virtual space 2 and used to associate an item with a first area. A character model 301 representing a player character 490 is drawn on the UI board 300. A second area 305 associated with the first area set on the right shoulder of the player character 490 is set on the right shoulder of the character model 301. A second area 306 associated with the first area set on the left shoulder of the player character 490 is set on the left shoulder of the character model 301. A second area 307 associated with the first area set on the right hip of the player character 490 is set on the right waist of the character model 301. A second area 308 associated with the first area set on the left hip of the player character 490 is set on the left waist of the character model 301. Additionally, specific objects 311 for identifying a shield object, specific objects 312 for identifying a sword object, and specific objects 313 for identifying a gun object are provided so as to be attachable to the UI board 300. The shield object, sword object, and gun object are all examples of items. The specific objects may be any object that can identify an item.

In this embodiment, a specific object is selected by the hand object 400, and the selected specific object is placed in an arbitrary second region, whereby an item indicated by the specific object placed in the second region is associated with the first region associated with the second region. Hereinafter, with reference to FIG. 23 to FIG. 25, a method of associating a gun object indicated by the specific object 313 with a first region set on the right shoulder of the player character 490 associated with the second region 305 by placing the specific object 313 in the second region 305 will be specifically described. FIG. 23 is a diagram showing an example of a state in which the specific object 313 is selected by the right hand object 400R. FIG. 24 is a diagram showing an example of a state in which the specific object 313 is placed in the second region 305 by the right hand object 400R. FIG. 25 is a diagram showing an example of a state in which the specific object 313 is placed in the second region 305.

When the user 190 moves the right hand holding the controller 160R, the processor 10 moves the right hand object 400R near the specific object 313 as shown in FIG. 23. As the right hand object 400R moves, the processor 10 determines whether or not the collision area CA of the right hand object 400R is colliding with the collision area CC of the specific object 313. When the right hand object 400R moves near the specific object 313, the user 190 performs the selection operation described above. As shown in FIG. 23, if the collision area CA is colliding with the collision area CC, the processor 10 causes the right hand object 400R to select the specific object 313 while the selection operation is continuing.

When the user 190 continues the selection operation and moves the right hand, the processor 10 moves the right hand object 400R and the specific object 313 selected by the right hand object 400R to the vicinity of the second region 305 as shown in FIG. 24. With the movement of the right hand object 400R, the processor 10 determines whether or not a second positional relationship is established between the specific object 313 and the second region 305. Specifically, the processor 10 determines whether or not the collision area CC of the specific object 313 and the collision area CD of the second region 305 collide. When the specific object 313 moves to the vicinity of the second region 305, the user 190 cancels the selection operation. When the collision area CC and the collision area CD collide as shown in FIG. 24, the processor 10 places the specific object 313 in the second region 305 as shown in FIG. 25 when the selection operation is canceled. As a result, the processor 10 associates the gun object indicated by the specific object 313 with the first region set on the right shoulder of the player character 490 associated with the second region 305, and updates the state of the gun object from an unassociated state in which it is not associated with the first region to an associated state in which it is associated with the first region.

Figure 26 is a flowchart showing an example of an update process for updating the state of an item in this embodiment, and in detail shows the update process from an unassociated state, in which the item is not associated with a first area, to an associated state, in which the item is associated with the first area.

In step S41, the processor 10, as the object control module 223, detects the movement of the user 190's hand (controller 160) and moves the hand object 400 in the virtual space 2 in conjunction with the detected hand movement. At this time, the processor 10, as the collision determination module 224, determines whether or not the collision area CA of the hand object 400 after the movement collides with another collision area.

In step S42, the processor 10, as the object control module 223, determines whether or not a selection operation has been performed by the user 190. If a selection operation has not been performed (No in step S42), the process returns to step S11. If a selection operation has been performed (Yes in step S42), the process proceeds to step S43.

In step S43, the processor 10, as the object control module 223, checks whether the collision determination module 224 has determined that the collision area CA of the hand object 400 and the collision area CC of the specific object are colliding. If the collision area CA and the collision area CC are not colliding (No in step S43), the process returns to step S41. If the collision area CA and the collision area CC are colliding (Yes in step S43), the process proceeds to step S44.

In step S44, the processor 10, as the object control module 223, causes the hand object 400 to select a specific object. As a result, the specific object is held by the hand object 400.

In step S45, the processor 10, as the object control module 223, detects the movement of the user 190's hand (controller 160) and moves the hand object 400 in the virtual space 2 in conjunction with the detected hand movement. At this time, the processor 10, as the collision determination module 224, determines whether the collision area CA of the moved hand object 400 and the collision area CC of the specific object selected by the hand object 400 collide with other collision areas.

In step S46, the processor 10, as the object control module 223, determines whether the selection operation by the user 190 has been released. If the selection operation has not been released (No in step S46), the process returns to step S45. If the selection operation has been released (Yes in step S46), the process proceeds to step S47.

In step S47, the processor 10, functioning as the object control module 223, checks whether the collision determination module 224 has determined that the collision area CC of the specific object and the collision area CD of the second region are colliding. If the collision area CC and the collision area CD are not colliding (No in step S47), the process proceeds to step S48. If the collision area CC and the collision area CD are colliding (Yes in step S47), the process proceeds to step S49.

In step S48, the processor 10, as the object control module 223, cancels the selection of the specific object by the hand object 400. This causes the specific object to be released from the hand object 400. Then, the process returns to step S41.

In step S49, the processor 10, as the object control module 223, places the specific object selected by the hand object 400 in the second region. This causes the specific object to be moved away from the hand object 400. This causes the processor 10 to associate the item indicated by the specific object with the first region associated with the second region, and update the state of the item from an unassociated state in which it is not associated with the first region to an associated state in which it is associated with the first region.

In this embodiment, it is assumed that the process shown in FIG. 26, the process shown in FIG. 17, the process shown in FIG. 18, and the process shown in FIG. 21 are in different modes. The process of updating the state of an item shown in FIG. 26 from an unassociated state to an associated state is assumed to be executed in a mode other than gameplay mode, such as lobby mode. The update process between the selected state and unselected state of an item shown in FIG. 17, the effect activation process of an item shown in FIG. 18, and the update process of parameters associated with an item shown in FIG. 21 are assumed to be executed in gameplay mode, for example. However, this is not limited to this.

In addition, in this embodiment, the item selection process described in FIG. 17 is described using an example in which the condition is that the hand object 400 is in a first positional relationship with the first area, but this is not limited thereto, and the condition may be that the hand object 400 is located outside the range of the field of view 23. In this way, by positioning the hand object 400 outside the range of the field of view 23, it is possible to cause the hand object 400 to select an item.

In addition, in the description of each of the above-mentioned embodiments, a virtual space (VR space) in which the user 190 is immersed by the HMD device 110 has been exemplified, but a see-through HMD device may be adopted as the HMD device 110. In this case, a virtual experience in an augmented reality (AR) space or a mixed reality (MR) space may be provided to the user 190 by outputting a field of view image so that a part of the image constituting the virtual space is superimposed as a field of view image on the real space visually recognized by the user 190 through the see-through HMD device. In this case, an action on a target object such as an item in the virtual space 2 may be caused based on the movement of the controller instead of the hand object of the player character. Specifically, the processor 10 may specify coordinate information of the position of the controller in the real space and define the position of the target object in the virtual space 2 in relation to the coordinate information in the real space. As a result, the processor 10 is able to grasp the positional relationship between the controller in the real space and the target object in the virtual space 2, and to execute processing corresponding to the above-mentioned collision control between the controller and the target object. As a result, it becomes possible to affect the target object based on the movement of the controller.

Although the embodiment of the present disclosure has been described above, the technical scope of the present invention should not be interpreted as being limited by the description of this embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications of the embodiment are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents.

The subject matter disclosed in this specification is presented, for example, in the following items:
(Item 1)
1. A method implemented on a computer (e.g., computer 200) for providing a virtual experience to a user (e.g., user 190), comprising:
A step of defining a virtual space (e.g., virtual space 2) for providing the virtual experience (e.g., step S1 of FIG. 10);
A step of moving a virtual viewpoint (e.g., virtual camera 1) in the virtual space in response to a movement of the user's head (e.g., step S6 in FIG. 10 );
A step of moving an operation object (e.g., a hand object 400) in the virtual space in response to a movement of a part of the user's body (e.g., step S8 in FIG. 10 );
updating a state of an item used in the virtual space when the operation object is moved out of a range of a field of view (e.g., a field of view area 23) from the virtual viewpoint (e.g., step S14 in FIG. 17 and step S34 in FIG. 21);
The method includes:
(Item 2)
The method according to item 1, in which, in the updating step (e.g., FIG. 17), when the state of the item is an unselected state in which the item is not selected by the control object, the state of the item is updated to a selected state in which the item is selected by the control object, on the condition that at least a first positional relationship is established between the control object and a first area in which a relative position with respect to the virtual viewpoint is fixed and which is located outside the field of view.
(Item 3)
The method described in item 2, in which the updating step (e.g., FIG. 17) updates the state of the item to the selected state, at least on the condition that the first positional relationship is established between the first area and the operation object, when the item is associated with the first area and the state of the item is the unselected state.
(Item 4)
A method according to item 2 or 3, in which, in the updating step (e.g., FIG. 17), when the first positional relationship is established between the first area and the operation object and a selection operation for selecting the item is performed, the state of the item is updated to the selected state, and when the selection operation is released, the state of the item is updated to the unselected state.
(Item 5)
A method according to item 3 or 4, in which, in the updating step (e.g., FIG. 26), when the state of the item is an unassociated state not associated with the first area and a specific object for identifying the item is selected with the operation object, the state of the item is updated to an associated state associated with the first area, at least on the condition that a second positional relationship is established between a second area associated with the first area and the operation object.
(Item 6)
6. The method of claim 5, wherein the mode of updating the state of the item from the unassociated state to the associated state and the mode of updating the state of the item from the unselected state to the selected state are different modes.
(Item 7)
The item (e.g., gun object 450) takes effect by consuming a parameter associated with the item;
The method according to item 1, wherein in the updating step (e.g., FIG. 21), when the item is selected with the control object and the state of the item is a first state in which at least a portion of a parameter associated with the item has been consumed, when the control object moves out of the field of view, the state of the item is updated to a second state in which a greater amount of the parameter remains than in the first state.
(Item 8)
A program for causing a computer to execute the method according to any one of items 1 to 7.
(Item 9)
1. A computer for providing a virtual experience to a user, comprising:
Under the control of the processor of the computer,
defining a virtual space for providing said virtual experience;
moving a virtual viewpoint within the virtual space in response to a movement of the user's head;
moving an operation object in the virtual space in response to a movement of a part of the user's body;
updating a state of an item used in the virtual space on condition that the operation object is moved out of a range of view from the virtual viewpoint;
The computer on which it runs.

1...Virtual camera, 2...Virtual space, 5...Reference line of sight, 10...Processor, 11...Memory, 12...Storage, 13...Input/output interface, 14...Communication interface, 15...Bus, 19...Network, 21...Center, 23...Viewing area, 24, 25...Area, 31...Frame, 32...Top surface, 33, 34, 36, 37...Button, 35...Infrared LED, 38...Analog stick, 100...HMD system, 110...HMD device, 112...Display, 114...Sensor, 116...Camera, 118...Microphone, 120...HMD sensor, 130...Motion sensor, 140 ...Gaze sensor, 150...Server, 160...Controller, 160R...Right controller, 190...User, 200...Computer, 220...Main control module, 221...Virtual space control module, 222...Virtual camera control module, 223...Object control module, 224...Collision determination module, 225...Parameter control module, 226...View image generation module, 227...Display control module, 240...Memory module, 241...Space information, 242...Object information, 243...User information, 250...Communication control module, M...View image

Claims (7)

On the computer,
A definition means for defining a virtual space for providing a virtual experience;
an operating means for moving an operation object in the virtual space in response to a movement of a part of the user 's body;
a first update means for updating an item from an unassociated state, in which the item is not associated with a first area, to an associated state, in which the item is associated with the first area;
a second update means for updating an item associated with the first area to a selected state in which the item is selected by the operation object on condition that a first positional relationship is established between the first area and the operation object;
and a usage means for using the item in the selected state,
the first region is set to a portion of a player character associated with the user;
A program that makes it work. The program according to claim 1 , wherein the first area is an area set for each of a plurality of locations of the player character associated with the user. 3. The program according to claim 2, wherein the item is a weapon object, and the effect of the weapon object is activated by consuming a parameter associated with the weapon object. The program according to any one of claims 1 to 3, further comprising a recovery means for recovering parameters associated with the item by moving the item in a selected state selected by the operation object outside the range of the user's field of view. The program according to claim 1 , further comprising a means for moving a virtual viewpoint within the virtual space in response to a movement of the user 's head, so that the first area is positioned outside a range of the user's field of view. The program according to claim 1, which omits drawing processing of the item associated with the first area. On the computer,
A definition means for defining a virtual space for providing a virtual experience;
an operating means for moving an operation object in the virtual space in response to a movement of a part of the user 's body;
a first update means for updating an item from an unassociated state, in which the item is not associated with a first area, to an associated state, in which the item is associated with the first area;
a second update means for updating an item associated with the first area to a selected state in which the item is selected by the operation object on condition that a first positional relationship is established between the first area and the operation object;
and a usage means for using the item in the selected state,
The first area is set to a portion of a player character associated with the user.

Images