JP2019207714A - Information processing method, computer and program - Google Patents

Abstract

To provide a user with virtual experience in a virtual space with high entertainment.SOLUTION: A method in a system including an HMD and a sensor for sensing a position of a part of a body of a user includes the steps of identifying virtual space data defining a virtual space, identifying a virtual viewpoint, generating a view image according to the virtual space data, the virtual viewpoint and an orientation of the HMD, moving an operation object in a normal mode according to movement of the part of the body, switching the operation object to a select mode in a case where the part of the body meets a predetermined condition, moving the virtual view point on the basis of the movement of the part of the body or the movement of the operation object during continuation of the selection mode in a case where a subject object is not selected by the operation object in the selection mode, and outputting the view image generated based on the virtual viewpoint to a display unit associated with the HMD.SELECTED DRAWING: Figure 11

Description

  The present disclosure relates to an information processing method, a computer, and a program.

  Japanese Patent Application Laid-Open No. 2004-133867 discloses a technique that enables a virtual experience in a virtual space to be enjoyed from various viewpoints by moving the viewpoint in the virtual space.

  In the technology disclosed in Patent Document 1, the user can only observe one object from various angles. With regard to such technology, there is room for improvement in order to enhance the entertainment characteristics of virtual experiences.

Japanese Patent No. 5869177

  An object of the present disclosure is to provide a user with a virtual experience in a virtual space with high entertainment properties.

  According to an embodiment of the present disclosure, there is provided an information processing method in a system including a head mounted device and a sensor configured to detect the position of a part of a body other than a user's head, Identifying virtual space data defining a virtual space including an operation object associated with a body part and at least one target object, wherein the operation object includes at least two operation modes including a normal mode and a selection mode A step of identifying a virtual viewpoint in the virtual space, a step of generating a visual field image according to the virtual space data, the virtual viewpoint, and an orientation of the head mounted device, The operation object is passed through the virtual space according to the movement of the body part. A step of moving in the mode, a step of switching the operation mode of the operation object to the selection mode when the state of the part of the body satisfies a predetermined condition, and the operation object when the operation mode is switched to the selection mode. When none of the at least one target object is selected by the step, the virtual viewpoint is moved based on the movement of the body part or the operation object during the selection mode, and the virtual Outputting a view image generated based on a viewpoint to a display unit associated with the head mounted device.

According to another embodiment of the present disclosure, a computer for use in a system comprising a head mounted device and a sensor configured to sense a position of a body part other than a user's head. The computer includes a processor, the processor identifying virtual space data defining a virtual space including an operation object associated with the body part and at least one target object, wherein the operation object has a normal mode and It can be moved in at least two operation modes including a selection mode, identifies a virtual viewpoint in the virtual space, and displays a view image according to the virtual space data, the virtual viewpoint, and the orientation of the head mounted device. Generating and manipulating the manipulation object in the virtual space according to the movement of the body part Is moved in the normal mode, and when the state of the part of the body satisfies a predetermined condition, the operation mode of the operation object is switched to the selection mode, and when the operation mode is switched to the selection mode, the operation object When none of the at least one target object is selected, the virtual viewpoint is moved based on the movement of the body part or the movement of the operation object during the selection mode, and based on the virtual viewpoint. A computer configured to output the generated field-of-view image to a display unit associated with the head mounted device is provided.

  According to another embodiment of the present disclosure, a program for causing a processor to execute the above method is provided.

  According to the embodiment of the present disclosure, it is possible to provide a user with a virtual experience in a virtual space with high entertainment properties.

  Other features and advantages of the present disclosure will become apparent from the following description of embodiments, the accompanying drawings, and the appended claims.

1 schematically shows the configuration of an HMD system 100. FIG. 2 is a block diagram illustrating an example of a basic hardware configuration of a computer according to an embodiment. It is a figure which represents notionally the uvw visual field coordinate system set to HMD by one Embodiment. FIG. 3 is a diagram conceptually illustrating an aspect of expressing a virtual space according to an embodiment. It is a figure showing the head of the user who wears HMD110 from the top by one embodiment. It is a figure showing the yz cross section which looked at the visual recognition area from the x direction in virtual space. It is a figure showing the xz cross section which looked at the visual recognition area from the y direction in virtual space. It is a figure showing the schematic structure of the controller by one Embodiment. It is a block diagram which shows the function of the computer for implement | achieving the display process of the virtual space, etc. in the HMD system by one Embodiment. It is a flowchart of the general process for displaying the image of the virtual space where a user immerses on a display part. 3 is a flowchart of a method according to an embodiment of the present disclosure. It is a figure which illustrates schematically the mode of the game assumed in one embodiment of this indication. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. It is a figure showing an example of processing of one embodiment of this indication typically. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. It is a figure showing an example of processing of one embodiment of this indication typically. 6 illustrates an example view image with reduced visual information according to an embodiment of the present disclosure. It is a graph which shows the example of the relationship between the rotation angle of a virtual viewpoint, and time. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. An example of a field-of-view image generated by processing of one embodiment of this indication is shown. It is a figure showing an example of processing of one embodiment of this indication typically. An example of a field-of-view image generated by processing of one embodiment of this indication is shown.

[Description of Embodiment of Present Disclosure]
First, configurations of exemplary embodiments of the present disclosure will be listed and described. A method, a computer, and a program according to an embodiment of the present disclosure may include the following configurations.

(Item 1)
An information processing method in a system comprising a head mounted device and a sensor configured to detect a position of a part of a body other than a user's head,
Identifying virtual space data defining a virtual space including an operation object associated with the body part and at least one target object, wherein the operation object includes at least two operations including a normal mode and a selection mode Steps that can be moved in mode,
Identifying a virtual viewpoint in the virtual space;
Generating a field-of-view image according to the virtual space data, the virtual viewpoint, and the orientation of the head mounted device;
Moving the operation object in the normal mode in the virtual space in response to movement of the body part;
When the state of the part of the body satisfies a predetermined condition, switching the operation mode of the operation object to the selection mode;
When none of the at least one target object is selected by the operation object when switched to the selection mode, the movement of the body part or the operation object is continued during the selection mode. Moving the virtual viewpoint based on:
Outputting a field-of-view image generated based on the virtual viewpoint to a display unit associated with the head-mounted device.

(Item 2)
The method of claim 1, wherein the movement of the body part includes a position of the body part.

(Item 3)
If the at least one target object is selected by the operation object when the selection mode is switched, based on the movement of the body part or the operation object during the selection mode, 3. The method according to item 1 or 2, further comprising moving the at least one target object.

(Item 4)
The body part is a right hand and a left hand, and the operation object includes a right hand object and a left hand object,
4. The method according to any of items 1 to 3, further comprising moving the virtual viewpoint based on the movement of the right hand and / or the left hand, or the right hand object and / or the left hand object.

(Item 5)
When the movement mode of both the right hand object and the left hand object is the selection mode, if movement of the right hand and the left hand is detected, the movement of the right hand and the left hand that moves first or the front 5. The method according to item 4, further comprising the step of moving the virtual viewpoint based on the movement of the operation object corresponding to the hand that has moved.

(Item 6)
The item according to any one of items 1 to 5, further comprising a step of gradually increasing the moving speed of the virtual viewpoint when a continuous movement of a part of the body or the operation object is detected in the selection mode. the method of.

(Item 7)
Increasing the moving speed of the virtual viewpoint according to one movement among the continuous movements as compared to the moving speed of the virtual viewpoint according to the movement performed before the one movement; The method according to item 6, comprising:

(Item 8)
When the movement included in the continuous movement is a movement at a certain speed or higher, and / or when a plurality of movements included in the continuous movement are performed with a time interval smaller than the predetermined time interval, Item 8. The method according to Item 6 or 7, further comprising increasing the moving speed of the virtual viewpoint.

(Item 9)
When one movement included in the continuous movement is a movement of a certain speed or more and / or a movement of a certain distance or more, the movement speed of the virtual viewpoint is gradually increased while the one movement is performed. The method according to any of items 6 to 8, further comprising a step of increasing.

(Item 10)
The method further includes the step of zeroing or reducing the moving speed of the virtual viewpoint when the angle of the movement of the body part or the operation object relative to the roll axis of the visual field is larger than half of the visual field angle. 10. The method according to any one of items 1 to 9.

(Item 11)
A component in the direction of the moving speed of the virtual viewpoint based on at least one component in the direction perpendicular to the roll axis of the field of view of the movement of the body part or the operation object is set to zero, or 11. A method according to any of items 1 to 10, further comprising a reducing step.

(Item 12)
When the operation mode of both the right hand object and the left hand object is the selection mode, when none of the at least one target object is selected, the right hand and the left hand are similar in the same direction. 6. The method according to item 4 or 5, further comprising the step of rotating the virtual viewpoint based on the movement when a movement surrounding the shape area is detected.

(Item 13)
13. The method of item 12, further comprising the step of rotating the virtual viewpoint based on the movement projected onto the horizontal plane when the movement surrounding the similarly shaped region is tilted with respect to the horizontal plane.

(Item 14)
14. The method according to item 12 or 13, further comprising reducing visual information of the view field image when rotating the virtual viewpoint.

(Item 15)
15. The method according to any of items 12 to 14, wherein the rotation of the virtual viewpoint is a combination of a rotation at a first speed and a rotation at a second speed that is slower or zero than the first speed.

(Item 16)
When the movement mode of both the right hand object and the left hand object is the selection mode, and when any of the at least one target object is not selected, a movement approaching the right hand and the left hand is detected. The method according to any one of items 4, 5 and 12 to 15, further comprising the step of reducing or increasing the range of the virtual space included in the view image.

(Item 17)
When the movement mode of both the right hand object and the left hand object is the selection mode, and when any of the at least one target object is not selected, a movement in which the right hand and the left hand move away is detected. The method according to any one of items 4, 5 and 12 to 15, further comprising the step of reducing or increasing the range of the virtual space included in the view image.

(Item 18)
The step of reducing or increasing the range of the virtual space included in the view image includes a step of changing the size of the virtual camera in the virtual space without changing the position of the virtual viewpoint. The method according to 16 or 17.

(Item 19)
A program for causing a processor to execute the method according to any one of items 1 to 18.

(Item 20)
A computer comprising a processor and a memory, wherein the method according to any one of items 1 to 18 is executed under the control of the processor.

(Item 21)
A computer for use in a system comprising a head mounted device and a sensor configured to sense a position of a body part other than a user's head, the computer comprising a processor, the processor comprising: ,
Identifying virtual space data defining a virtual space including an operation object associated with the body part and at least one target object, and the operation object is moved in at least two operation modes including a normal mode and a selection mode Can
Identify a virtual viewpoint in the virtual space;
According to the virtual space data, the virtual viewpoint, and the direction of the head mounted device, a view image is generated,
In response to the movement of the body part, the operation object is moved in the normal mode in the virtual space,
When the state of the part of the body satisfies a predetermined condition, the operation mode of the operation object is switched to the selection mode,
When none of the at least one target object is selected by the operation object when switched to the selection mode, the movement of the body part or the operation object is continued during the selection mode. Based on the virtual viewpoint,
A computer configured to output a view field image generated based on the virtual viewpoint to a display unit associated with the head mounted device.

[Details of Embodiment of the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, similar elements are denoted by the same reference numerals. The names and functions are also the same. A duplicate description of such elements is omitted.

  A configuration of a head-mounted device (HMD) system 100 will be described with reference to FIG. FIG. 1 schematically shows the configuration of the HMD system 100. In one example, the HMD system 100 is provided as a home system or a business system. The HMD may be a so-called head mounted display including a display unit, or may be a head mounted device on which a terminal such as a smartphone having the display unit can be attached.

  The HMD system 100 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a display unit 112 and a gaze sensor 140. The controller 160 may include a motion sensor 130.

  In one example, the computer 200 may be connectable to a network 192 such as the Internet, and may be communicable with a computer such as a server 150 connected to the network 192. In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120.

  The HMD 110 may be worn on the head of the user 190 and provide a virtual space to the user during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the display unit 112, respectively. When each eye of the user visually recognizes each image, the user can recognize the image as a three-dimensional image based on the parallax of both eyes.

  The display unit 112 is realized as, for example, a non-transmissive display device. In one example, the display unit 112 is disposed on the main body of the HMD 110 so as to be positioned in front of both eyes of the user. Therefore, when the user visually recognizes the three-dimensional image displayed on the display unit 112, the user can be immersed in the virtual space. In an embodiment, the virtual space includes, for example, a background, an object that can be operated by the user, an image of a menu that can be selected by the user, and the like. In an embodiment, the display unit 112 may be realized as a liquid crystal display unit or an organic EL (Electro Luminescence) display unit included in an information display terminal such as a smartphone.

  In one example, the display unit 112 may include a sub display unit for displaying an image for the right eye and a sub display unit for displaying an image for the left eye. In another aspect, the display unit 112 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the display unit 112 includes a high-speed shutter. The high-speed shutter operates so that an image for the right eye and an image for the left eye can be displayed alternately so that the image is recognized only by one of the eyes.

  In one example, the HMD 110 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. More specifically, the HMD sensor 120 may read a plurality of infrared rays emitted from the HMD 110 and detect the position and inclination of the HMD 110 in the real space.

  In an aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and inclination of the HMD 110 by executing image analysis processing using image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120 as a position detector. The HMD 110 can detect the position and inclination of the HMD 110 itself using the sensor 114. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 detects its own position and inclination using any one of these sensors instead of the HMD sensor 120. obtain. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects angular velocities around the three axes of the HMD 110 in real space over time. The HMD 110 calculates a temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates a tilt of the HMD 110 based on the temporal change of the angle. The HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance. Moreover, the view field image may include a configuration for presenting the real space in a part of the image configuring the virtual space. For example, an image captured by a camera mounted on the HMD 110 may be displayed so as to be superimposed on a part of the field-of-view image, or a part of the field-of-view image is set by setting a high transmittance of a part of the transmissive display device. Real space may be visible from a part.

  The gaze sensor 140 detects the direction (line of sight) in which the line of sight of the user 190's right eye and left eye is directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, the gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris with respect to the irradiated light. . The gaze sensor 140 can detect the line of sight of the user 190 based on each detected rotation angle.

  Server 150 may send a program to computer 200. In another aspect, the server 150 may communicate with other computers 200 for providing virtual reality to HMDs used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's operation with another computer 200, and a plurality of users are common in the same virtual space. Allows you to enjoy the game.

  The controller 160 is connected to the computer 200 by wire or wireless. The controller 160 receives input of commands from the user 190 to the computer 200. In an aspect, the controller 160 is configured to be gripped by the user 190. In another aspect, the controller 160 is configured to be wearable on the body of the user 190 or a portion of clothing. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the computer 200. In another aspect, the controller 160 receives an operation from the user 190 for controlling the position and movement of an object arranged in the virtual space.

  In an aspect, the motion sensor 130 is attached to the user's hand to detect movement of the user's hand. For example, the motion sensor 130 detects the rotation speed, rotation speed, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided in the glove-type controller 160, for example. In some embodiments, for safety in real space, it is desirable that the controller 160 be worn on something that does not fly easily by being worn on the hand of the user 190, such as a glove shape. In another aspect, a sensor that is not worn by the user 190 may detect the movement of the user's 190 hand. Optical sensors that detect the position, orientation, direction of movement, distance of movement, etc. of various parts of the body of the user 190 may be used. For example, a signal from a camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190. For example, the motion sensor 130 and the computer 200 are connected to each other wirelessly. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

  A computer 200 according to an embodiment of the present disclosure will be described with reference to FIG. FIG. 2 is a block diagram illustrating an example of a basic hardware configuration of the computer 200 according to an embodiment of the present disclosure. The computer 200 includes a processor 202, a memory 204, a storage 206, an input / output interface 208, and a communication interface 210 as main components. Each component is connected to the bus 212.

  The processor 202 executes a series of instructions included in a program stored in the memory 204 or the storage 206 based on a signal given to the computer 200 or when a predetermined condition is satisfied. In one aspect, the processor 202 is realized as a device such as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or an FPGA (Field-Programmable Gate Array).

  The memory 204 temporarily stores programs and data. The program is loaded from the storage 206, for example. The data includes data input to the computer 200 and data generated by the processor 202. In one aspect, the memory 204 is implemented as a volatile memory such as a RAM (Random Access Memory).

  The storage 206 permanently stores programs and data. The storage 206 is realized, for example, as a nonvolatile storage device such as a ROM (Read-Only Memory), a hard disk device, or a flash memory. Programs stored in the storage 206 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, a program for realizing communication with another computer 200, and the like. Data stored in the storage 206 includes data and objects for defining the virtual space.

  In another aspect, the storage 206 may be realized as a removable storage device such as a memory card. In still another aspect, a configuration using a program and data stored in an external storage device may be used instead of the storage 206 built in the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used as in an amusement facility, it is possible to update programs and data collectively.

  In some embodiments, the input / output interface 208 communicates signals between the HMD 110, the HMD sensor 120, and the motion sensor 130. In one aspect, the input / output interface 208 is realized using a terminal such as USB (Universal Serial Bus, USB), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or the like. The input / output interface 208 is not limited to the above.

  In certain embodiments, the input / output interface 208 may further communicate with the controller 160. For example, the input / output interface 208 receives signals output from the controller 160 and the motion sensor 130. In another aspect, the input / output interface 208 sends instructions output from the processor 202 to the controller 160. The command instructs the controller 160 to vibrate, output sound, emit light, and the like. When the controller 160 receives the command, the controller 160 executes vibration, sound output, light emission, and the like according to the command.

  The communication interface 210 is connected to the network 192 and communicates with other computers (for example, the server 150) connected to the network 192. In one aspect, the communication interface 210 is realized as a wired communication interface such as a LAN (Local Area Network), or a wireless communication interface such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), or NFC (Near Field Communication). Is done. The communication interface 210 is not limited to the above.

  In an aspect, the processor 202 accesses the storage 206, loads one or more programs stored in the storage 206 into the memory 204, and executes a series of instructions included in the program. The one or more programs may include an operating system of the computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space, and the like. The processor 202 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 208. The HMD 110 displays an image on the display unit 112 based on the signal.

  In the example shown in FIG. 2, the computer 200 is provided outside the HMD 110. However, in another aspect, the computer 200 may be embedded in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including the display unit 112 may function as the computer 200.

  Further, the computer 200 may be configured to be used in common for a plurality of HMDs 110. According to such a configuration, for example, since the same virtual space can be provided to a plurality of users, each user can enjoy the same application as other users in the same virtual space.

  In an embodiment, in the HMD system 100, a global coordinate system is preset. The global coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-rear direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, vertical direction (up-down direction), and front-rear direction in the global coordinate system are defined as an x-axis, a y-axis, and a z-axis, respectively. More specifically, in the global coordinate system, the x axis is parallel to the horizontal direction of the real space. The y axis is parallel to the vertical direction of the real space. The z axis is parallel to the front-rear direction of the real space.

  In certain aspects, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects the infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect temporal changes in the position and inclination of the HMD 110 using each value detected over time.

The global coordinate system is parallel to the real space coordinate system. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination around the three axes of the HMD 110 in the global coordinate system. The HMD sensor 120 is an HMD in the global coordinate system.
Based on the inclination of 110, the uvw visual field coordinate system is set to HMD110. The uvw visual field coordinate system set in the HMD 110 corresponds to a viewpoint coordinate system when the user 190 wearing the HMD 110 views an object in the virtual space.

  The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually showing the uvw visual field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and inclination of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 202 sets the uvw visual field coordinate system to the HMD 110 based on the detected value.

  As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system with the head (origin) of the user wearing the HMD 110 as the center (origin). More specifically, the HMD 110 includes a horizontal direction, a vertical direction, and a front-rear direction (x-axis, y-axis, and z-axis) that define the global coordinate system by an inclination around each axis of the HMD 110 in the global coordinate system. Three directions newly obtained by tilting around the axis are set as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw visual field coordinate system in the HMD 110.

  In an aspect, when the user 190 wearing the HMD 110 stands upright and is viewing the front, the processor 202 sets the uvw visual field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch direction (u-axis) and yaw direction (v-axis) of the uvw visual field coordinate system in the HMD 110. And the roll direction (w-axis).

  After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (the amount of change in inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system, respectively, as the inclination of the HMD 110. The pitch angle (θu) represents the inclination angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the inclination angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the inclination angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

  The HMD sensor 120 sets the uvw visual field coordinate system in the HMD 110 after the HMD 110 has moved to the HMD 110 based on the detected tilt angle of the HMD 110. The relationship between the HMD 110 and the uvw visual field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

  In one aspect, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between the points (for example, the distance between the points). The position in the real space may be specified as a relative position to the HMD sensor 120. Further, the processor 202 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually illustrating one aspect of expressing a virtual space 400 according to an embodiment. The virtual space 400 has a spherical shape that covers the entire 360 ° direction of the center 406. In FIG. 4, the upper half of the celestial sphere in the virtual space 400 is illustrated in order not to complicate the description. In the virtual space 400, each mesh is defined. The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system defined in the virtual space 400. The computer 200 associates each partial image constituting the content (still image, moving image, etc.) that can be developed in the virtual space 400 with each corresponding mesh in the virtual space 400 so that the virtual space image that can be visually recognized by the user A virtual space 400 to be developed is provided to the user.

  In one aspect, the virtual space 400 defines an xyz coordinate system with the center 406 as the origin. The xyz coordinate system is, for example, parallel to the global coordinate system. Since the xyz coordinate system is a kind of viewpoint coordinate system, the horizontal direction, vertical direction (vertical direction), and front-rear direction in the xyz coordinate system are defined as the x-axis, y-axis, and z-axis, respectively. Therefore, the x axis (horizontal direction) of the xyz coordinate system is parallel to the x axis of the global coordinate system, the y axis (vertical direction) of the xyz coordinate system is parallel to the y axis of the global coordinate system, and the xyz coordinate system The z-axis (front-rear direction) is parallel to the global coordinate system z-axis.

  When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 404 is disposed at the center 406 of the virtual space 400. In an aspect, the processor 202 displays an image captured by the virtual camera 404 on the display unit 112 of the HMD 110. The virtual camera 404 similarly moves in the virtual space 400 in conjunction with the movement of the HMD 110 in the real space. Thereby, changes in the position and orientation of the HMD 110 in the real space can be similarly reproduced in the virtual space 400.

  As in the case of the HMD 110, the virtual camera 404 has a uvw visual field coordinate system. The uvw visual field coordinate system of the virtual camera 404 in the virtual space 400 is defined so as to be linked to the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 404 changes accordingly. The virtual camera 404 can also move in the virtual space 400 in conjunction with the movement of the user wearing the HMD 110 in the real space.

  The processor 202 of the computer 200 defines the viewing area 410 in the virtual space 400 based on the arrangement position of the virtual camera 404 and the reference line of sight 408. The visual recognition area 410 corresponds to an area of the virtual space 400 that is visually recognized by the user wearing the HMD 110.

  The line of sight of the user 190 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 190 visually recognizes the object. The uvw visual field coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the display unit 112. Further, the uvw visual field coordinate system of the virtual camera 404 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, the HMD system 100 according to an aspect can regard the line of sight of the user 190 detected by the gaze sensor 140 as the line of sight of the user in the uvw visual field coordinate system of the virtual camera 404.

  The determination of the user's line of sight will be described with reference to FIG. FIG. 5 is a diagram showing the head of user 190 wearing HMD 110 according to an embodiment from above.

  In one aspect, the gaze sensor 140 detects each line of sight of the user's 190 right eye and left eye. In an aspect, when the user 190 is looking nearby, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

  When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 specifies the gazing point N1 that is the intersection of the lines of sight R1 and L1. On the other hand, when the detected values of the lines of sight R2 and L2 are received from the gaze sensor 140, the computer 200 specifies the intersection of the lines of sight R2 and L2 as the point of sight. The computer 200 specifies the line of sight N0 of the user 190 based on the specified position of the gazing point N1. For example, the computer 200 detects, as the line of sight N0, the extending direction of the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1. The line of sight N0 is a direction in which the user 190 is actually pointing the line of sight with both eyes. The line of sight N0 corresponds to the direction in which the user 190 is actually pointing the line of sight with respect to the visual recognition area 410.

  In another aspect, the HMD system 100 may include a microphone and a speaker in any part of the HMD system 100. The user can give a voice instruction to the virtual space 400 by speaking to the microphone.

  In another aspect, the HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 400.

  In still another aspect, the HMD system 100 may include a communication circuit for connecting to the Internet or a call function for connecting to a telephone line.

  The visual recognition area 410 will be described with reference to FIGS. FIG. 6 is a diagram illustrating a yz section when the visual recognition area 410 is viewed from the x direction in the virtual space 400. FIG. 7 is a diagram illustrating an xz cross section of the visual recognition area 410 viewed from the y direction in the virtual space 400.

  As shown in FIG. 6, the visible region 410 in the yz section includes a region 602. The area 602 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the yz section of the virtual space 400. The processor 202 defines a range including the polar angle α around the reference line of sight 408 in the virtual space as a region 602.

  As shown in FIG. 7, the visible region 410 in the xz cross section includes a region 702. The area 702 is defined by the arrangement position of the virtual camera 404, the reference line of sight 408, and the xz cross section of the virtual space 400. The processor 202 defines a range including the azimuth angle β around the reference line of sight 408 in the virtual space 400 as a region 702. The polar angles α and β are determined according to the arrangement position of the virtual camera 404 and the orientation of the virtual camera 404.

  In one aspect, the HMD system 100 provides the user 190 with a field of view in the virtual space by causing the display unit 112 to display a field of view image based on a signal from the computer 200. The view field image corresponds to a portion of the virtual space image 402 that is superimposed on the viewing area 410. When the user 190 moves the HMD 110 worn on the head, the virtual camera 404 moves in conjunction with the movement. As a result, the position of the visual recognition area 410 in the virtual space 400 changes. Thereby, the visual field image displayed on the display unit 112 is updated to an image that is superimposed on the visual recognition area 410 in the virtual space 400 in the direction in which the user faces in the virtual space image 402. The user can visually recognize a desired direction in the virtual space 400.

Thus, the orientation (tilt) of the virtual camera 404 corresponds to the user's line of sight (reference line of sight 408) in the virtual space 400, and the position where the virtual camera 404 is arranged corresponds to the user's viewpoint in the virtual space 400. Therefore, by moving the virtual camera 404 (including the operation of changing the arrangement position and the operation of changing the orientation), the image displayed on the display unit 112 is updated, and the field of view (including the viewpoint and the line of sight) of the user 190 is moved. Is done.

  While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 402 developed in the virtual space 400 without visually recognizing the real world. Therefore, the HMD system 100 can give the user a high sense of immersion in the virtual space 400.

  In an aspect, the processor 202 may move the virtual camera 404 in the virtual space 400 in conjunction with movement of the user 190 wearing the HMD 110 in real space. In this case, the processor 202 specifies an image region (that is, the visual recognition region 410 in the virtual space 400) projected on the display unit 112 of the HMD 110 based on the position and orientation of the virtual camera 404 in the virtual space 400.

  According to an embodiment, the virtual camera 404 may include two virtual cameras: a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Also, appropriate parallax may be set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 400.

  An example of the controller 160 will be described with reference to FIG. FIG. 8 is a diagram illustrating a schematic configuration of the controller 160 according to an embodiment.

  In certain aspects, the controller 160 may include a right controller and a left controller. For simplicity of explanation, only the right controller 800 is shown in FIG. The right controller 800 is operated with the right hand of the user 190. The left controller is operated with the left hand of the user 190. In an aspect, the right controller 800 and the left controller are configured symmetrically as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller 800 and the left hand holding the left controller. In another aspect, the controller 160 may be an integrated controller that accepts operations of both hands. Hereinafter, the right controller 800 will be described.

  The right controller 800 includes a grip 802, a frame 804, and a top surface 806. The grip 802 is configured to be gripped by the right hand of the user 190. For example, the grip 802 can be held by the palm of the right hand of the user 190 and three fingers (middle finger, ring finger, little finger).

  The grip 802 includes buttons 808 and 810 and a motion sensor 130. The button 808 is disposed on the side surface of the grip 802 and receives an operation with the middle finger of the right hand. The button 810 is disposed on the front surface of the grip 802 and receives an operation with the index finger of the right hand. In some embodiments, the buttons 808, 810 are configured as trigger buttons. The motion sensor 130 is built in the housing of the grip 802. Note that the grip 802 may not include the motion sensor 130 when the operation of the user 190 can be detected from around the user 190 by a camera or other device.

  The frame 804 includes a plurality of infrared LEDs 812 arranged along the circumferential direction thereof. The infrared LED 812 emits infrared light in accordance with the progress of the program while the program using the controller 160 is being executed. The infrared rays emitted from the infrared LED 812 can be used to detect the positions and postures (tilt and orientation) of the right controller 800 and the left controller (not shown). In the example shown in FIG. 8, infrared LEDs 812 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. 8. An array of one column or three or more columns may be used.

  The top surface 806 includes buttons 814 and 816 and an analog stick 818. Buttons 814 and 816 are configured as push buttons. Buttons 814 and 816 receive an operation with the thumb of the right hand of user 190. In a certain aspect, the analog stick 818 accepts an operation in an arbitrary direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object arranged in the virtual space 400.

  In an aspect, the right controller 800 and the left controller include a battery for driving a member such as the infrared LED 812. The battery may be either a primary battery or a secondary battery, and the shape thereof may be arbitrary, such as a button type or a dry battery type. In another aspect, the right controller 800 and the left controller may be connected to a USB interface of the computer 200, for example. In this case, the right controller 800 and the left controller may be supplied with power via the USB interface.

  FIG. 9 is a block diagram illustrating functions of the computer 200 for realizing display processing or the like of the virtual space 400 in the HMD system 100 according to an embodiment of the present disclosure. The computer 200 controls image output to the display unit 112 based mainly on inputs from the HMD sensor 120, the motion sensor 130, the gaze sensor 140, and the controller 160.

  The computer 200 includes a processor 202, a memory 204, and a communication control unit 205. The processor 202 includes a virtual space specifying unit 902, an HMD motion detection unit 903, a line-of-sight detection unit 904, a reference line-of-sight determination unit 906, a visual field region determination unit 908, a virtual viewpoint specification unit 910, and a visual field image generation unit 912. An object control unit 914, an operation mode switching unit 916, a selection determination unit 918, a virtual viewpoint control unit 920, a motion determination unit 922, a virtual camera control unit 924, and a view image output unit 926. . Memory 204 may be configured to store various information. In one example, the memory 204 may include virtual space data 928, object data 930, application data 932, and other data 934. The memory 204 also includes various data necessary for computation to provide output information corresponding to inputs from the HMD sensor 120, motion sensor 130, gaze sensor 140, controller 160, etc., to the display unit 112 associated with the HMD 110. But you can. The object data 930 may include data related to operation objects, target objects, and the like arranged in the virtual space. The display unit 112 may be built in the HMD 110 or may be a display of another device (for example, a smartphone) that can be attached to the HMD 110.

  In FIG. 9, the components included in the processor 202 are just one example of expressing the function executed by the processor 202 as a specific module. The functions of multiple components may be realized by a single component. The processor 202 may be configured to perform the functions of all components.

  FIG. 10 is a flowchart of a general process for displaying an image of the virtual space in which the user is immersed on the display unit 112.

  A general process of the HMD system 100 for providing a virtual space image will be described with reference to FIGS. 9 and 10. The virtual space 400 may be provided by the interaction of the HMD sensor 120, the gaze sensor 140, the computer 200, and the like.

Processing begins at step 1002. As an example, a game application included in the application data may be executed by the computer 200. In step 1004, the processor 202 (virtual space specifying unit 902)
28, the celestial spherical virtual space image 402 constituting the virtual space 400 into which the user is immersed is generated. The position and inclination of the HMD 110 are detected by the HMD sensor 120. Information detected by the HMD sensor 120 is transmitted to the computer 200. In step 1006, the HMD motion detection unit 903 acquires position information and tilt information of the HMD 110. In step 1008, the viewing direction is determined based on the acquired position information and tilt information.

  When the gaze sensor 140 detects the movements of the left and right eyeballs of the user, the information is transmitted to the computer 200. In step 1010, the line-of-sight detection unit 904 identifies the direction in which the right eye and the left eye are directed, and determines the line-of-sight direction N0. In step 1012, the reference visual line determination unit 906 determines the visual field direction determined by the inclination of the HMD 110 or the user's visual line direction N0 as the reference visual line 408. The reference line of sight 408 may also be determined based on the position and tilt of the virtual camera 404 that follows the position and tilt of the HMD 110.

  In step 1014, the view area determination unit 908 determines the view area 410 of the virtual camera 404 in the virtual space 400. As shown in FIG. 4, the visual field area 410 is a part of the virtual space image 402 that constitutes the visual field of the user. The visual field area 410 is determined based on the reference line of sight 408. A yz sectional view of the visual field region 410 viewed from the x direction and an xz sectional view of the visual field region 410 viewed from the y direction are shown in FIGS. 6 and 7 described above, respectively.

  In step 1016, the visual field image generation unit 912 generates a visual field image based on the visual field region 410. The field-of-view image includes two two-dimensional images for the right eye and the left eye. These two-dimensional images are superimposed on the display unit 112 (more specifically, the right-eye image is output to the right-eye display unit, and the left-eye image is output to the left-eye display unit). A virtual space 400 as an image is provided to the user. In step 1018, the view image output unit 926 outputs information related to the view image to the display unit 112. The display unit 112 displays the view image based on the received view image information. The process ends at step 1020.

  FIG. 11 is a flowchart of a method 1100 according to one embodiment of the present disclosure. In an embodiment of the present disclosure, a computer program may cause the processor 202 (or the computer 200) to execute the steps illustrated in FIG. Also, another embodiment of the disclosure may be implemented as a processor 202 (or computer 200) that performs the method 1100.

  Hereinafter, embodiments of the present disclosure will be specifically described. Here, as a specific example to which the embodiment of the present disclosure can be applied, a plurality of users are in a virtual space in which each user's avatar, game field, each user's unit acting in the game field, and the like are arranged. Imagine a game that you can immerse and enjoy. However, embodiments of the present disclosure are not necessarily limited to such an aspect. It will be apparent to those skilled in the art that the embodiments of the present disclosure can take various forms that fall within the scope of the claims.

  FIG. 12 is a diagram schematically illustrating a game mode assumed in the present embodiment. In this example, two users 1212A and 1212B (hereinafter collectively referred to as “user 1212”) play the game. The users 1212A and 1212B wear HMDs 110A and 110B (hereinafter collectively referred to as “HMD110”) on the head, and further wear controllers 160A and 160B (hereinafter collectively referred to as “controller 160”). In one example, the controller 160 has the configuration described above with respect to FIG. In this case, each user wears the controller in both hands. However, this is only an example of the controller 160. Various aspects of the controller that can be worn on other parts of the user's body can be applied to embodiments of the present disclosure.

  In the virtual space 1200, avatars 1214A and 1214B (hereinafter also collectively referred to as “avatar 1214”) and a game field 1222 operated by the users 1212A and 1212B, respectively, are arranged. On the game field 1222, there are units 1216A and 1216B (hereinafter collectively referred to as “unit 1216”), bases 1224A and 1224B (hereinafter collectively referred to as “base 1224”), and the like, which each of the users 1212A and 1212B has. Be placed. During the game play, the user 1212 performs various operations so that his unit 1216 can reach the opponent user's base 1224.

  The user 1212 can enjoy a game played on the game field 1222 in the virtual space 1220 via the avatar 1214. The avatars 1214A and 1214B have operation objects 1220A and 1220B (hereinafter collectively referred to as “operation objects 1220”), respectively. Basically, the operation object 1220 is the body part of the avatar 1214 corresponding to the body part of the user 1212 to which the controller 160 is attached. For example, the operation object 1220 is the hand of the avatar 1214. This is only an example of the operation object 1220. The operation object 1220 may be another part of the body of the avatar 1214. For example, when the controller 160 is configured to be worn on the foot of the user 1212, the operation object 1220 may be the foot of the avatar 1214.

  As shown in FIG. 12, virtual cameras 1204A1 and 1204B1 (hereinafter collectively referred to as “virtual camera 1204-1”) may be arranged at the positions of avatars 1214A and 1214B, respectively. Further, virtual cameras 1204A2 and 1204B2 (hereinafter collectively referred to as “virtual camera 1204-2”) may be arranged at the positions of the units 1216A and 1216B, respectively. The user 1212 can not only view the video of the virtual space 1200 obtained from the virtual camera 1204-1 (from the viewpoint of the avatar 1214) but also the virtual obtained from the virtual camera 1204-2 (from the viewpoint of the unit 1216). A video of the space 1200 can also be seen.

  Returning to FIG. 11, the process starts at step 1102. The processor 202 reads and executes the game program included in the application data 932 stored in the memory 204.

  The process proceeds to step 1104, and the virtual space specifying unit 902 specifies virtual space data for the executed game based on the virtual space data 928, the object data 930, and the like. The virtual space data defines a virtual space including an operation object 1220 and at least one target object associated with a part of the user 1212's body (for example, a hand). The target objects may include various objects (not shown) such as chairs, shelves, and lamps that exist in the virtual space 1200, in addition to items related to the game such as cards (described later) used for the game. As will be described later, the operation object 1220 can be moved in at least two operation modes including a normal mode and a selection mode.

  The process proceeds to step 1106, and the virtual viewpoint specifying unit 910 specifies a virtual viewpoint in the virtual space 1200. In the example of FIG. 12, the virtual viewpoint may be the position of the virtual camera 1204-1 or 1204-2. The virtual viewpoint may be appropriately determined according to the progress of the game.

The process proceeds to step 1108, and the view image generation unit 912 generates a view image based on the virtual space data, the virtual viewpoint, and the orientation of the HMD. The field-of-view image is generated by, for example, the processing already described with reference to FIG. The generated view field image is output by the view field image output unit 926 to the display unit 112 associated with the HMD 110 and displayed by the display unit 112. The user 1212 wearing the HMD 110 can see the view field image displayed on the display unit 112.

  FIG. 13 shows an example of a field-of-view image generated by the processing in step 1108. In this example, the virtual camera 1204A1 associated with the avatar 1214A is set as the virtual viewpoint. As shown in the figure, the view image 1300 acquired by the virtual camera 1204A1 is an operation object (in this case, a hand) 1220A of the user 1212A, an avatar 1214B of the other user 1212B, a game field 1222, and a game field 1222. Units 1216A and 1216B, bases 1224A and 1224B, walls 1304, items 1302-1 and 1302-2 that can be used in the game such as cards owned by user 1212A, and the like are included. In this example, the operation object 1220A includes a left hand object 1220A1 and a right hand object 1220A2. The user 1212 </ b> A can obtain an immersive feeling as if he / she entered the virtual space 1200 and participated in a game performed on the game field 1222.

  The process proceeds to step 1110, and the motion determination unit 922 determines whether the state of a part of the user's body satisfies a predetermined condition. The predetermined condition can be set in various ways, and may be included in the application data 930 in the memory 204 or the like. For example, the predetermined condition may be that the user 1212A bends the finger of the user 1212A when the user 1212A presses a button (eg, the button 808 and / or 810) of the controller 160A (eg, the controller 800 shown in FIG. 8). . As another example, the predetermined condition may be that the user's movement detected by the motion sensor 130 includes a specific movement. In the following, the present embodiment will be described on the assumption that the predetermined condition is that the user 1212A bends the finger of the user 1212A by pressing the button of the controller 160A with the finger.

  In the present embodiment, the “movement” of a part of the user's body means various positions such as the position, orientation, direction of movement in the moving state, distance of movement, etc. of the part of the body. May include other concepts.

  If the predetermined condition is not satisfied (“N” in step 1110), the process proceeds to step 1112. In the present embodiment, the operation object 1220A can be moved in at least two modes including a normal mode and a selection mode. Here, the “normal mode” may simply mean a state in which the operation object 1220A can be moved. On the other hand, the “selection mode” can affect any target other than the operation object 1220A, such as an operation of grasping a target object such as the items 1302-1 and 1302-2 using the operation object 1220A. It may mean a state. In addition, the normal mode and the selection mode may be variously defined. Information regarding the mode of the operation mode may be included in the application data 932 or the like in the memory 204.

  In step 1112, the operation mode switching unit 916 sets the operation mode of the operation object 1220A to the normal mode. The process proceeds to step 1114, and the object control unit 914 moves the operation object 1220A according to the movement of a part of the body of the user 1212A. The process proceeds to step 1124, and the visual field image output unit 926 outputs the visual field image generated by the visual field image generation unit 912 to the display unit 112.

  If the predetermined condition is satisfied in step 1110 (“Y” in step 1110), the process proceeds to step 1116. In step 1116, the operation mode switching unit 916 sets the operation mode of the operation object 1220A to the selection mode.

  FIG. 14 shows an example of a field-of-view image displayed on the display unit 112 by the process of step 1116. When user 1212A presses the left controller button of controller 160A with the finger of the left hand, it is determined in step 1110 that a predetermined condition is satisfied, and in step 1116, the operation mode of operation object 1220A is set to the selection mode. In this case, as shown in FIG. 14, the object control unit 914 may change the form of the left hand object 1220A1 to a state in which the hand is held.

  The process proceeds to step 1118, and the selection determination unit 918 determines whether or not the target object has been selected by the operation object 1220A. As an example, when the operation object 1220A enters a range of a predetermined distance from the target object in a state where a predetermined condition is satisfied, it may be determined that the target object is selected. It will be apparent to those skilled in the art that the determination of whether a target object has been selected can be made in various ways.

  If it is determined that the target object is not selected (“N” in step 1118), the process proceeds to step 1120. In step 1120, the virtual viewpoint control unit 920 moves the virtual viewpoint (for example, the position of the virtual camera 1204A1) based on the movement of the body part of the user 1212A or the movement of the operation object 1220A during the selection mode. The processing proceeds to step 1124, and the view image output unit 926 outputs the view image generated based on the movement of the virtual viewpoint to the display unit 112.

  FIG. 15 shows an example of a view field image displayed on the display unit 112 by the processing of steps 1118 and 1120. If the user 1212A pulls the left hand back while the selection object is not selected during the selection mode, the motion sensor 130 detects this movement. Based on the detected movement, the object control unit 914 moves the left hand object 1220A1 backward as indicated by an arrow in FIG. The virtual viewpoint control unit 920 moves the virtual viewpoint in a predetermined method based on the left hand movement of the user 1212A or the left hand object 1220A1. For example, the virtual viewpoint control unit 920 may move the virtual viewpoint forward in response to the left hand or left hand object 1220A1 moving backward. In this case, a view field image 1500 shown in FIG. 15 is generated based on the virtual viewpoint moved forward. According to the present embodiment, it is possible to provide the user with a virtual experience that the user can freely move in the virtual space 1200 by the movement of a part of his / her body.

  Returning to FIG. 11, if it is determined in step 1118 that the target object has been selected (“Y” in step 1118), the process proceeds to step 1122. In step 1122, the virtual viewpoint control unit 920 moves the selected target object based on the movement of the part of the body of the user 1212A or the movement of the operation object 1220A while the selection mode is continued. The processing proceeds to step 1124, and the view image output unit 926 outputs the view image generated based on the movement of the virtual viewpoint to the display unit 112.

  FIG. 16 shows an example of a view field image displayed on the display unit 112 when it is determined that the target object has been selected. When the user moves his left hand so that the left hand object 1220A1 overlaps the card 1302-2 during the selection mode, the motion sensor 130 detects this movement. Based on the detected movement, the object control unit 914 causes the left hand object 1220A1 to hold the card 1302-2 as shown in FIG. As a result, the view field image 1600 is displayed on the display unit 112.

FIG. 17 shows an example of a field-of-view image displayed on the display unit 112 by the process of step 1122. When the user 1212A moves the left hand diagonally forward to the right, the motion sensor 130 detects this movement. Based on the detected movement, the object control unit 914 moves the left hand object 1220A1 and the target object 1302-2 as indicated by arrows in FIG. At this time, the virtual viewpoint control unit 920 may not move the virtual viewpoint at all or may move it somewhat. As a result, a view field image 1700 shown in FIG. 17 is generated and displayed on the display unit 112.

  FIG. 18 is a diagram schematically showing an example of processing in step 1120. In this example, the movement of the left hand of the user 1212A or the movement of the left hand object 1220A1 corresponding thereto is made across the outside and inside of the field of view as indicated by the arrows. As described above, when the angle of the movement of the part of the user's body or the movement of the operation object with respect to the roll axis of the visual field is larger than half of the visual field angle, the virtual viewpoint control unit 920 sets the moving speed of the virtual viewpoint to zero. Or may be less than when the movement is made within the field of view. Thereby, it is possible to avoid a viewpoint movement that is difficult to predict for the brain of the user 1212A wearing the HMD 110A, and it is possible to prevent the user 1212A from getting sick. For the same purpose, the virtual viewpoint control unit 920 determines the movement speed of the virtual viewpoint based on at least one of the components in the direction orthogonal to the roll axis of the visual field among the movement of the part of the user's body or the movement of the operation object. The component in that direction may be zeroed or reduced. For example, the processor 202 may decompose the magnitude of the movement of the left hand object 1220A1 based on the movement of the left hand of the user 1212A detected by the motion sensor 130 into a pitch direction component, a yaw direction component, and a roll direction component. The virtual viewpoint control unit 920 may set the components in these directions of the moving speed of the virtual viewpoint based on the movements of the pitch direction component and the yaw direction component to zero or reduce them from the actual size.

  FIG. 19 is a diagram schematically illustrating an example of processing in step 1120. In this example, when the user 1212A presses predetermined buttons of both the right controller and the left controller respectively attached to the right hand and the left hand, the operation modes of both the left hand object 1220A1 and the right hand object 1220A2 are set to the selection mode. ing. In this state, when the motion sensor 130 detects the movement of the right hand and the left hand, the virtual viewpoint control unit 920 moves the movement of the right hand or the left hand, or the operation object corresponding to the movement of the hand. The virtual viewpoint may be moved based on the movement of.

  In step 1120, the virtual viewpoint control unit 920 may move the virtual viewpoint by various methods other than the above example. For example, when a continuous movement of a part of the user's body or the operation object is detected in the selection mode, the virtual viewpoint control unit 920 may gradually increase the moving speed of the virtual viewpoint. In this example, the motion determination unit 922 may determine that a continuous motion has been made when a next motion is detected within a predetermined time after a certain motion is detected. Alternatively, the motion determination unit 922 may determine that a continuous motion has been made when a predetermined number of motions are detected within a predetermined time.

  In another example, the virtual viewpoint control unit 920 sets the moving speed of the virtual viewpoint corresponding to one of the continuous movements to the virtual viewpoint corresponding to the movement performed before the one movement. It may be increased as compared with the moving speed.

  In another example, the virtual viewpoint control unit 920 may be configured such that the motion included in the continuous motion is a motion at a certain speed or more, and / or the plurality of motions included in the continuous motion are smaller than the predetermined time interval. When it is performed with a time interval, the moving speed of the virtual viewpoint may be increased.

  In another example, the virtual viewpoint control unit 920 performs the one movement when the one movement included in the continuous movement is a movement of a certain speed or more and / or a movement of a certain distance or more. Meanwhile, the moving speed of the virtual viewpoint may be gradually increased.

  FIG. 20 is a diagram schematically illustrating an example of processing in step 1120. When the user 1212A presses predetermined buttons of both the right controller and the left controller, the operation modes of both the left hand object 1220A1 and the right hand object 1220A2 are set to the selection mode. If the movement surrounding the region of the same shape in the same direction by both the right hand and the left hand is detected as indicated by arrows 2002 and 2004 without selecting the target object, the virtual viewpoint control unit 920 The virtual viewpoint may be rotated based on the movement. Here, the “similar shape” may include various shapes other than a circle and an ellipse. As a result of the rotation of the virtual viewpoint, a view field image 2000 is generated and displayed on the display unit 112.

  In the example illustrated in FIG. 20, when the motion surrounding the region having the same shape is tilted with respect to the horizontal plane, the processor 202 may project the motion onto the horizontal plane. The virtual viewpoint control unit 920 may rotate the virtual viewpoint based on the movement projected onto the horizontal plane.

  In the example illustrated in FIG. 20, the visual field image generation unit 912 may reduce the visual information of the visual field image by blurring processing or the like when the virtual viewpoint is rotated. FIG. 21 shows an example of a field-of-view image 2100 with reduced visual information. Thereby, it is possible to reduce the influence that the user is likely to get drunk during the rotation of the virtual viewpoint.

  In the example illustrated in FIG. 20, the virtual viewpoint control unit 920 may not rotate the virtual viewpoint at a constant speed. For example, the rotation of the virtual viewpoint may be a combination of the rotation at the first speed and the rotation at the second speed that is slower or zero than the first speed. FIG. 22 is a graph showing an example of the relationship between the rotation angle of the virtual viewpoint and time. As shown in the figure, the virtual viewpoint control unit 920 rotates the virtual viewpoint greatly from the angle θ0 to θ1 during the time interval from T0 to T1, and changes the virtual viewpoint during the next time interval from T1 to T2. It is also possible to rotate the angle θ1 to θ2 small, and then repeat these rotations alternately. The virtual viewpoint may be rotated in various other modes. According to such an embodiment, it is possible to obtain an effect that makes the user less likely to get sick than when the virtual viewpoint is rotated at a constant speed.

  FIG. 23 is a diagram schematically illustrating an example of processing in step 1120. When the user 1212A presses predetermined buttons of both the right controller and the left controller, the operation modes of both the left hand object 1220A1 and the right hand object 1220A2 are set to the selection mode. When a movement approaching the right hand and the left hand is detected without selecting the target object, the virtual viewpoint control unit 920 may reduce or increase the range of the virtual space included in the view field image. In order to realize this, for example, as illustrated in FIG. 25, the virtual viewpoint control unit 920 may change the size of the virtual camera 404 in the virtual space 1200. For example, when a movement approaching the right hand and the left hand is detected, the virtual viewpoint control unit 920 may enlarge the size of the virtual camera 404 as illustrated in the upper part of FIG. By expanding the size of the virtual camera 404, the field of view that can be captured by the virtual camera 404 is expanded. A view field image 2300 in FIG. 23 is an example of a view field image generated in this case. It can be seen that the range of the virtual space included in the view image is larger than the view image 1900 shown in FIG. As another example, when a movement approaching the right hand and the left hand is detected, the virtual viewpoint control unit 920 may reduce the size of the virtual camera 404 as illustrated in the lower part of FIG. In this case, the field of view that can be taken by the virtual camera 404 is reduced, and the range of the virtual space included in the view field image is reduced.

FIG. 24 is a diagram schematically illustrating an example of processing in step 1120. Contrary to the example of FIG. 23, when a movement in which the right hand and the left hand move away is detected, the virtual viewpoint control unit 920 may reduce or increase the range of the virtual space included in the view field image. The specific processing is the same as in the case of FIG. As an example, the virtual viewpoint control unit 920 may reduce or enlarge the size of the virtual camera 404. The view image 2400 is an example of a view image generated when the size of the virtual camera 404 is reduced. It can be seen that the range of the virtual space included in the view image is smaller than the view image 1900 shown in FIG.

  In the example illustrated in FIGS. 23 and 24, the virtual viewpoint control unit 920 may change the range of the virtual space included in the view field image by a method different from the method for enlarging or reducing the size of the virtual camera 404. it can. For example, when the virtual camera 404 includes a right-eye virtual camera and a left-eye virtual camera, the virtual viewpoint control unit 920 may increase or decrease the distance between them. By increasing the distance between the right-eye virtual camera and the left-eye virtual camera, the range of the virtual space included in the view field image can be increased. Conversely, by reducing the distance between the right-eye virtual camera and the left-eye virtual camera, the range of the virtual space included in the view field image can be reduced.

  FIG. 26 shows an example of a field-of-view image generated by the processing in step 1108. In this example, the user 1212A sets the virtual camera 1204A2 associated with the unit 1216A as a virtual viewpoint by a predetermined operation in the game. As shown in the drawing, a view field image 2600 acquired by the virtual camera 1204A2 includes an operation object (here, both hands) 2620A1 and 2620A2 of the unit 1216A, an avatar 1214B of the opponent user, a unit 1216B of the opponent user, a game field 1222, an opponent Includes user base 1224B, wall 1304, and the like. Thus, in this embodiment, the user 1212A can not only enjoy the virtual experience in the virtual space from the viewpoint of the avatar 1212A but also the viewpoint of the unit 1216A that proceeds toward the opponent's base 1224B on the game field 1222. Can also enjoy a virtual experience. Also in the case of FIG. 26, the processing of the embodiment of the present disclosure described with reference to FIGS. 11 to 25 can be applied. That is, the user 1212A can operate the left hand object 2620A1 and the right hand object 2620A2 of the unit 1216A by movement of a part of his / her body. When the virtual viewpoint moves in the process of step 1120, the user 1212A moves to the opponent base 1224B as the unit 1216A on the game field 1222 (for example, by moving forward). You can get an experience. Therefore, according to the embodiment of the present disclosure, it is possible to provide a user with a virtual experience in a virtual space with high entertainment properties.

  Embodiments of the present disclosure have been described primarily as being implemented as processor 202 (or computer 200) or method 1100. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be implemented as a computer program that causes processor 202 to perform method 1100.

  While embodiments of the present disclosure have been described, it should be understood that these are only examples and are not intended to limit the scope of the present disclosure. It should be understood that changes, additions, improvements, and the like of the embodiments can be made as appropriate without departing from the spirit and scope of the present disclosure. The scope of the present disclosure should not be limited by any of the above-described embodiments, but should be defined only by the claims and their equivalents.

DESCRIPTION OF SYMBOLS 100 ... HMD system, 110 ... HMD, 112 ... Display part, 114 ... Sensor, 120 ... HMD sensor, 130 ... Motion sensor, 140 ... Gaze sensor, 150 ... Server, 160 ... Controller, 190 ... User, 192 ... Network, 200 ... Computer, 202 ... Processor, 204 ... Memory, 205 ... Communication controller, 206 ... Storage, 208 ... I / O interface, 210 ... Communication interface, 212 ... Bus, 400 ... Virtual space, 402 ... Virtual space image, 404 ... Virtual Camera, 406 ... Center, 408 ... Reference line of sight, 410 ... Viewing area, 602, 702 ... Area, 800 ... Controller, 802 ... Grip, 804 ... Frame, 806 ... Top, 808, 810, 814, 816 ... Button, 818 ... Analog stick, 90 ... virtual space specifying unit, 903 ... motion detecting unit, 904 ... gaze detecting unit, 906 ... reference gaze determining unit, 908 ... visual field region determining unit, 910 ... virtual viewpoint specifying unit, 912 ... visual field image generating unit, 914 ... object control Unit, 916 ... operation mode switching unit, 918 ... selection determination unit, 920 ... virtual viewpoint control unit, 922 ... determination unit, 924 ... virtual camera control unit, 926 ... visual field image output unit, 928 ... virtual space data, 930 ... object Data, 932 ... Application data, 934 ... Other data, 1200 ... Virtual space, 1204A1, A2, B1, B2 ... Virtual camera, 1212A, B ... User, 1214A, B ... Avatars, 1216A, B ... Unit, 1220A, B ... Operation object, 1222 ... Game field, 1224A, B ... Base, 1302- , 2 ... object, 1304 ... wall, 160A, B ... controller, 1220A1,2620A1 ... the left hand object, 1220A2,2620A ... the right hand object

Claims (1)

An information processing method in a system comprising a head mounted device and a sensor configured to detect a position of a part of a body other than a user's head,
Identifying virtual space data defining a virtual space including an operation object associated with the body part and at least one target object;
Identifying a virtual viewpoint in the virtual space;
Moving the operation object in the first mode in the virtual space in response to movement of the body part;
When the state of the part of the body satisfies a predetermined condition, switching the operation mode of the operation object to a second mode;
When none of the at least one target object is selected by the operation object when the mode is switched to the second mode, the movement of the body part or the operation object is continued during the second mode. Moving the virtual viewpoint based on movement.