JP2018195172A - Information processing method, information processing program, and information processing device - Google Patents

Abstract

To improve the convenience of a user in a virtual space.SOLUTION: An information processing method executed by a processor includes the steps of: a) generating virtual space data for defining a virtual space including a left hand object 5L, a right hand object 5R, a first instruction object 3, and a second instruction object 4; b) generating visual field image data indicating a visual field image displayed on an HMD on the basis of the movement of the HMD and the virtual space data; and c) moving the left hand object 5L and the right hand object 5R according to the movement of left hand and right hand of a user U wearing the HMD. The second instruction object 4 guides a finger of the user U to take a second motion, and is visually associated with the left hand object 5L and the right hand object 5R. The first instruction object 3 guides a hand of the user U to take the second motion.SELECTED DRAWING: Figure 8

Description

  The present disclosure relates to an information processing method, an information processing program, and an information processing apparatus.

  Patent Document 1 discloses a technique for controlling the movement of a virtual hand in a virtual space according to the movement of a user's hand in a real space. Thus, by moving the virtual hand according to the movement of the user's hand, it is possible to provide the user with an experience in a virtual space with a high degree of freedom (hereinafter referred to as a virtual experience).

Japanese Patent No. 5996138

  By the way, when providing various virtual experiences to the user, it is conceivable to set the movement of the user's hand as a trigger for various actions. However, if the user's hand movement is set as a trigger for various actions, the user needs to learn the various hand movements associated with the various actions. Therefore, there is room for improvement in improving user convenience in the virtual space.

  It is an object of the present disclosure to provide an information processing method, an information processing program, and an information processing apparatus that can improve user convenience in a virtual space.

According to one aspect of the present disclosure, an information processing method that is processed by a processor is:
a) generating virtual space data defining a virtual space including an operation object and an instruction object;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device and the virtual space data;
c) moving the operation object in response to movement of a part of the body other than the head of the user wearing the head-mounted device;
including.
The pointing object is visually associated with the operation object and guides the part of the user's body to perform a guiding action.

  According to the present disclosure, it is possible to provide an information processing method, an information processing program, and an information processing apparatus that can improve user convenience in a virtual space.

It is the schematic which shows a user terminal. It is a figure which shows the head of the user with which HMD was mounted | worn. It is a figure which shows the hardware constitutions of a control apparatus. It is a flowchart which shows the process which displays a visual field image on HMD. It is xyz space figure which shows an example of virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. It is the figure which showed typically the virtual space which concerns on 1st Embodiment. It is the schematic diagram which showed a mode that the 2nd instruction | indication object moved to the predetermined position in the visual field of a virtual camera. It is a flowchart for demonstrating the process which moves a 2nd instruction | indication object in the visual field of a virtual camera. It is the figure which showed typically the virtual space which concerns on 2nd Embodiment. It is a figure for demonstrating a motion of an avatar.

[Description of Embodiments Presented by the Present Disclosure]
An overview of an embodiment indicated by the present disclosure will be described.
(1) a) generating virtual space data defining a virtual space including an operation object and an instruction object;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device and the virtual space data;
c) moving the operation object in response to movement of a part of the body other than the head of the user wearing the head-mounted device;
Including
The pointing object is
Visually associated with the manipulation object and guiding the part of the user's body to perform a guiding action;
An information processing method executed by a processor.

  According to the above method, the pointing object guides a part of the user's body (for example, a hand or a finger) to perform a guiding operation. In this way, the user can move the body part so that the body part performs a guiding operation by visually recognizing the pointing object. For example, the user can relatively easily move the user's hand or finger (an example of a part of the user's body) associated with a predetermined action in the virtual space by visually recognizing the pointing object in a tutorial or the like. Can be learned. In this manner, an information processing method that can improve the convenience of the user in the virtual space can be provided.

  (2) The information processing method according to item (1), wherein the virtual space further includes a first connection object that connects the operation object and the instruction object so as to be visually associated with each other.

  According to the above method, the user's line of sight can be guided to the instruction object by the first connection object that connects the operation object and the instruction object so as to be visually associated with each other.

  (3) d) updating the visual aspect of the first connected object so as to keep the operation object and the pointing object visually associated with each other according to the movement of the operation object; The information processing method according to 2).

  According to the above method, the visual aspect of the first connected object is updated so that the operation object and the instruction object are continuously visually associated with the movement of the operation object. Thus, even when the operation object is moved, the user's line of sight can be guided to the instruction object by the first connected object.

(4) The virtual space is
A virtual controller having a virtual button and associated with the operation object;
A second connected object for connecting the pointing object and the virtual button so as to be visually associated;
The information processing method according to item (2) or (3).

  According to the above method, the user's line of sight can be guided to the virtual button and the pointing object by the second linked object that links the pointing object and the virtual button of the virtual controller so as to be visually associated with each other. For this reason, in a tutorial or the like, the user learns the association between the movement of the user's finger (an example of a part of the user's body) and the action in the virtual space by visually recognizing both the pointing object and the virtual button. Can do.

  (5) Items (1) to (4), further comprising: e) moving the pointing object so that the pointing object continues to be arranged at a predetermined position in the visual field in response to movement of the visual field of the user. The information processing method as described in any one of them.

  According to the above method, according to the movement of the user's visual field, the pointing object moves so as to continue to be arranged at a predetermined position in the user's visual field. Thus, since the pointing object follows the user's field of view, the convenience of the user in the virtual space can be improved.

  (6) The information processing method according to any one of items (1) to (4), wherein the instruction object is fixedly arranged in the virtual space.

  According to the above method, the pointing object is fixedly arranged in the virtual space. For this reason, even when the pointing object is located outside the user's field of view, the user's line of sight can be guided to the pointing object by the first linked object that links the operation object and the pointing object.

(7) The instruction object is an avatar arranged in the virtual space,
The pointing object is visually associated with the operation object by including a part of the virtual body of the avatar corresponding to the part of the user's body that is to perform the guidance operation,
When the movement of the part performs a guiding operation, the pointing object guides the part of the user's body to perform the guiding operation.
The information processing method according to item (6).

  According to the above method, when the movement of a part of the virtual body of the avatar performs the guiding operation, the pointing object guides the part of the user's body (for example, a hand or a finger) to perform the guiding operation. To do. Thus, the user can move a part of the body so that the part of the body performs a guiding operation by visually recognizing the movement of the part of the virtual body of the avatar. For example, in a tutorial or the like, a user visually recognizes a movement of a part of a virtual body of an avatar, thereby allowing a user's hand or finger associated with a predetermined action in the virtual space (an example of a part of the user's body). ) Can be learned relatively easily.

  (8) The information according to item (7), wherein the orientation of the avatar is set so that the back surface of the avatar is displayed on the visual field image when the avatar exists in the visual field of the user. Processing method.

  According to the above method, when the avatar exists in the user's field of view, the avatar's orientation is set so that the back surface of the avatar is displayed on the field of view image. Thus, since the avatar turns away from the user, the user can imitate the movement of a part of the virtual body of the avatar relatively easily.

  (9) The information processing method according to item (7) or (8), wherein the avatar is controlled so that only the portion of the virtual body moves.

  According to the above method, since the avatar is controlled so that only a part of the virtual body corresponding to a part of the user's body moves, the user's line of sight is guided to a part of the virtual body of the avatar (for example, a virtual hand). Can be made.

(10) The guidance operation includes a first movement and a second movement,
The part of the user's body includes a hand and a finger;
The pointing object includes a first pointing object that guides the hand to make the first movement, a second pointing object that guides the finger to make the second movement,
Have
f) further comprising generating a target object operated by the operation object in response to the second movement executed subsequent to the first movement.
The information processing method according to any one of items (1) to (3).

  According to the above method, the first pointing object guides the user's hand to make the first movement, and the second pointing object guides the user's finger to make the second movement. Furthermore, a target object that is operated by the operation object is generated in accordance with the second movement that is executed following the first movement. Thus, the user can move the user's hand so that the movement of the user's hand becomes the first movement by visually recognizing the first instruction object, and can also visually recognize the second instruction object. The user's finger can be moved so that the movement of the user's finger becomes the second movement. For example, in a tutorial or the like, the user visually recognizes the first instruction object and the second instruction object, so that the first movement of the user's hand associated with the action related to the appearance of the target object in the virtual space and the user's The second movement of the finger can be learned relatively easily.

  (11) An information processing program for causing a computer to execute the information processing method according to any one of items (1) to (10) is provided.

  According to the above, it is possible to provide an information processing program that can improve the convenience of the user in the virtual space.

(12) a processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
When the computer-readable instruction is executed by the processor, the information processing apparatus executes the information processing method according to any one of items (1) to (10).

  According to the above, it is possible to provide an information processing apparatus that can improve the convenience of the user in the virtual space.

[Details of Embodiments Presented by the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, about the member which has the same reference number as the member already demonstrated in description of this embodiment, the description is not repeated for convenience of explanation.

  First, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 1 is a schematic diagram showing a user terminal 1. As shown in FIG. 1, the user terminal 1 includes a head mounted device (HMD) 110, a headphone 116, a microphone 118, a position sensor 130, an external controller 320, and a control device mounted on the head of the user U. 120.

  The HMD 110 includes a display unit 112, an HMD sensor 114, and a gaze sensor 140. The display unit 112 includes a non-transmissive display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. Thereby, the user U can immerse in the virtual space by viewing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the user U's left eye and a right-eye display unit configured to provide an image to the user U's right eye. Also good. Further, the HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance.

  The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (such as an angular velocity sensor and a gyro sensor), and detects various movements (tilt and the like) of the HMD 110 mounted on the user U's head. be able to.

  The gaze sensor 140 has an eye tracking function that detects the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right eye gaze sensor irradiates, for example, infrared light to the right eye of the user U, and detects reflected light reflected from the right eye (particularly the cornea and iris), thereby acquiring information related to the rotation angle of the right eye's eyeball. May be. On the other hand, the left eye gaze sensor irradiates the left eye of the user U with, for example, infrared light, and detects reflected light reflected from the left eye (particularly the cornea and iris), thereby providing information on the rotation angle of the left eye's eyeball. May be obtained.

  Headphones 116 (audio output units) are attached to the left and right ears of the user U, respectively. The headphones 116 are configured to receive audio data (electrical signals) from the control device 120 and output audio based on the received audio data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Furthermore, the microphone 118 is configured to transmit audio data to the control device 120.

  The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 by wireless or wired communication, and is configured to detect information on the position, inclination, or light emission intensity of a plurality of detection points (not shown) provided in the HMD 110. . Further, the position sensor 130 is configured to detect information on the position, inclination and / or light emission intensity of a plurality of detection points (not shown) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared light or visible light. The position sensor 130 may include an infrared sensor and a plurality of optical cameras.

  The external controller 320 detects a movement of a part of the body of the user U (a part other than the head, and in this embodiment, the hand of the user U), thereby detecting a hand object (avatar) displayed in the virtual space. It is also used to control the movement of the virtual hand. The external controller 320 is an external controller 320R for right hand operated by the right hand of the user U (hereinafter simply referred to as controller 320R) and an external controller 320L for left hand operated by the left hand of the user U (hereinafter simply referred to as controller 320L). And).

  The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the finger of the right hand, and may include a plurality of operation buttons that are operated by the finger of the right hand. A right hand object (also referred to as an avatar's virtual right hand) that exists in the virtual space moves in accordance with the movement of the controller 320R. For example, when the user U presses a predetermined operation button of the controller 320R, the predetermined finger of the right hand object bends, while when the user U releases the predetermined operation button of the controller 320R, the predetermined finger of the right hand object extends. May be. As described above, the finger of the right-hand object may be controlled in accordance with an operation on the operation button of the controller 320R.

  The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the finger of the left hand, and may include a plurality of operation buttons operated by the finger of the left hand. A left hand object (also referred to as an avatar's virtual left hand) moves in the virtual space in accordance with the movement of the controller 320L. For example, when the user U presses a predetermined operation button of the controller 320L, the predetermined finger of the left hand object bends, while when the user U releases the predetermined operation button of the controller 320L, the predetermined finger of the left hand object extends. May be. As described above, the finger of the left hand object may be controlled in accordance with an operation on the operation button of the controller 320L.

  The control device 120 is a computer configured to control the HMD 110. The control device 120 identifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and installs the position of the virtual camera in the virtual space and the HMD 110 in the real space based on the identified position information. Thus, the position of the user U can be accurately associated. Furthermore, the control device 120 identifies the operation of the external controller 320 based on the information transmitted from the position sensor 130 and / or the external controller 320, and in the virtual space based on the identified operation of the external controller 320. The motion of the hand object displayed on the screen can be accurately associated with the motion of the external controller 320 in the real space. In particular, the control device 120 identifies the operation of the controller 320L based on information transmitted from the position sensor 130 and / or the controller 320L, and is displayed in the virtual space based on the identified operation of the controller 320L. The motion of the left hand object and the motion of the controller 320L in the real space (the motion of the left hand of the user U) can be accurately associated with each other. Similarly, the control device 120 identifies the operation of the controller 320R based on the information transmitted from the position sensor and / or the controller 320R, and is displayed in the virtual space based on the identified operation of the controller 320R. The movement of the right hand object and the movement of the controller 320R in the real space (the movement of the right hand of the user U) can be accurately associated.

  Further, the control device 120 specifies the gaze of the right eye and the gaze of the left eye of the user U based on the information transmitted from the gaze sensor 140 (the left eye gaze sensor and the right eye gaze sensor), and the right eye gaze. It is possible to specify the point of gaze that is the intersection of the left eye gaze. Furthermore, the control device 120 can specify the line of sight of both eyes of the user U (the line of sight of the user U) based on the specified gazing point. Here, the line of sight of the user U is the line of sight of both eyes of the user U, and coincides with the direction of a straight line passing through the middle point of the line connecting the right eye and the left eye of the user U and the gazing point.

  Next, with reference to FIG. 2, a method for acquiring information related to the position and inclination of the HMD 110 will be described. FIG. 2 is a diagram illustrating the head of the user U wearing the HMD 110. Information on the position and inclination of the HMD 110 that is linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands up is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the direction orthogonal to the v-axis and the w-axis is defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 is an angle around each uvw axis (that is, a yaw angle indicating rotation around the v axis, a pitch angle indicating rotation around the u axis, and a center around the w axis). The inclination determined by the roll angle indicating rotation) is detected. The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

  Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 3 is a diagram illustrating a hardware configuration of the control device 120. As shown in FIG. 3, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are connected to each other via a bus 126 so as to communicate with each other.

  The control device 120 may be configured as a personal computer, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. In addition, some functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

  The control unit 121 includes a memory and a processor. The memory includes, for example, a ROM (Read Only Memory) in which various programs are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and / or a GPU (Graphics Processing Unit), and a program specified from various programs embedded in the ROM is expanded on the RAM. It is comprised so that various processes may be performed in cooperation with.

  In particular, the control unit 121 may control various operations of the control device 120 by a processor developing a control program on a RAM and executing the control program in cooperation with the RAM. The control unit 121 displays a field image on the display unit 112 of the HMD 110 based on the field image data. Thereby, the user U can be immersed in the virtual space.

  The storage unit (storage) 123 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. The storage unit 123 may store data related to the user U authentication program, various images, and objects (such as instruction objects). Furthermore, a database including tables for managing various data may be constructed in the storage unit 123.

  The I / O interface 124 is configured to connect the position sensor 130, the HMD 110, the external controller 320, the headphones 116, and the microphone 118 to the control device 120 so that they can communicate with each other, for example, a USB (Universal) The terminal includes a serial bus (D) terminal, a digital visual interface (DVI) terminal, a high-definition multimedia interface (HDMI) (registered trademark) terminal, and the like. The control device 120 may be wirelessly connected to each of the position sensor 130, the HMD 110, the external controller 320, the headphones 116, and the microphone 118.

  The communication interface 125 is configured to connect the control device 120 to a communication network 30 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as a server via the communication network 30 and various processing circuits for wireless connection, for communicating via the communication network 30. It is configured to conform to the communication standard.

  Next, processing for displaying a field-of-view image on the HMD 110 will be described with reference to FIGS. 4 to 7. FIG. 4 is a flowchart showing a process for displaying the visual field image on the HMD 110. FIG. 5 is an xyz space diagram showing an example of the virtual space 200. The state (a) in FIG. 6 is a yx plan view of the virtual space 200 shown in FIG. The state (b) in FIG. 6 is a zx plan view of the virtual space 200 shown in FIG. FIG. 7 is a diagram illustrating an example of the visual field image V displayed on the HMD 110.

  As shown in FIG. 4, in step S1, the control unit 121 (see FIG. 3) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 5, the virtual space 200 is defined as an omnidirectional sphere centered on the center position 210 (in FIG. 5, only the upper half celestial sphere is shown). In the virtual space 200, an xyz coordinate system with the center position 210 as the origin is set. The virtual camera 300 defines a visual axis L for specifying the visual field image V (see FIG. 7) displayed on the HMD 110. The uvw coordinate system that defines the visual field of the virtual camera 300 is determined so as to be linked to the uvw coordinate system that is defined around the head of the user U in the real space. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

  Next, in step S2, the control unit 121 specifies the field of view CV (see FIG. 6) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 identifies the position and orientation of the virtual camera 300 in the virtual space 200 based on information regarding the position and tilt of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300 and specifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a part of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, a part of the virtual space 200 displayed on the HMD 110). Equivalent to Further, the visual field CV includes a first region CVa set as an angular range of the polar angle α around the visual axis L in the xy plane shown in the state (a) of FIG. 6, and the state (b) of FIG. 6. The xz plane shown has a second region CVb set as an angle range of the azimuth angle β with the visual axis L as the center. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the information regarding the identified line of sight of the user U and the position and inclination of the HMD 110. The orientation of the virtual camera 300 (virtual camera field axis L) may be determined.

  As described above, the control unit 121 can specify the visual field CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 updates the visual field CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. be able to. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the visual field CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change in the line of sight of the user U.

  Next, in step S <b> 3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

  Next, in step S4, the control unit 121 displays the field image V on the display unit 112 of the HMD 110 based on the field image data (see FIG. 7). As described above, the visual field CV of the virtual camera 300 is changed according to the movement of the user U wearing the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 is changed. I can immerse myself in 200.

  Note that the virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye view image data indicating the left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates right-eye view image data indicating a right-eye view image based on the virtual space data and the view of the right-eye virtual camera. Thereafter, the control unit 121 displays the left-eye field image on the left-eye display unit based on the left-eye field image data, and also displays the right-eye field image on the right-eye display unit based on the right-eye field image data. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally due to the parallax between the left-eye visual field image and the right-eye visual field image. As will be described later, the virtual camera may be arranged at the position of the avatar eye operated by the user. For example, the left-eye virtual camera may be placed in the left eye of the avatar, while the right-eye virtual camera may be placed in the right eye of the avatar.

  Further, the processing in steps S1 to S4 shown in FIG. 4 may be executed for each frame (a still image constituting a moving image). For example, when the frame rate of the moving image is 90 fps, the processes in steps S1 to S4 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). As described above, since the processes in steps S1 to S4 are repeatedly performed at predetermined intervals, the visual field of the virtual camera 300 is updated according to the operation of the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110. Is updated.

(First embodiment)
Next, a first embodiment of the present disclosure will be described below with reference to FIG. FIG. 8 is a diagram schematically illustrating the virtual space 200A according to the first embodiment. As shown in FIG. 8, the virtual space 200A includes a virtual camera 300, a left hand object 5L, a right hand object 5R, a left virtual controller 6L, and a right virtual controller 6R. Furthermore, the virtual space 200 </ b> A includes a first instruction object 3, a second instruction object 4, first connection objects 12 and 13, and second connection objects 14 and 16.

  The virtual space 200A shown in FIG. 8 is a tutorial for allowing the user U to learn a predetermined hand movement and finger movement associated with a predetermined action (for example, an action for taking out a virtual sword) before starting an action game or the like. Virtual space for The control unit 121 (see FIG. 3) generates virtual space data defining the virtual space 200A illustrated in FIG. 8, and then generates visual field image data based on the visual field CV of the virtual camera 300 and the virtual space data. Thereafter, the control unit 121 displays the field image V on the HMD 110 based on the field image data.

  The control unit 121 moves the left hand object 5L (an example of an operation object) in the virtual space 200A in accordance with the movement of the controller 320L (see FIG. 1) indicating the movement of the user U's left hand. In particular, the control unit 121 specifies the operation of the controller 320L based on the information transmitted from the position sensor 130 and / or the controller 320L, and determines the operation of the left hand object 5L based on the specified operation of the controller 320L. The operation of the controller 320L (the operation of the left hand of the user U) is accurately associated. For example, the control unit 121 may move the left hand object 5L according to the movement of the controller 320L and move the finger of the left hand object 5L according to the operation on the operation button of the controller 320L.

  Similarly, the control unit 121 moves the right hand object 5R (an example of an operation object) in the virtual space 200A according to the movement of the controller 320R (see FIG. 1) indicating the movement of the right hand of the user U. In particular, the control unit 121 identifies the operation of the controller 320R based on the information transmitted from the position sensor 130 and / or the controller 320R, and the operation of the right hand object 5R based on the identified operation of the controller 320R. The operation of the controller 320R (the operation of the right hand of the user U) is accurately associated. For example, the control unit 121 may move the right hand object 5R according to the movement of the controller 320R and move the finger of the right hand object 5R according to an operation on the operation button of the controller 320R.

  The left virtual controller 6L is an object corresponding to the controller 320L, and is visually associated with the left hand object 5L. The left virtual controller 6L includes virtual buttons 7L and 8L. The virtual button 7L is an object corresponding to the first button of the controller 320L, while the virtual button 8L is an object corresponding to the second button of the controller 320L. The left virtual controller 6L may move together with the left hand object 5L. The virtual button 7L of the left virtual controller 6L may be pressed by the finger of the left hand object 5L by pressing the first button of the controller 320L by the user U in the real space.

  The right virtual controller 6R is an object corresponding to the real space controller 320R, and is visually associated with the right hand object 5R. The right virtual controller 6R includes virtual buttons 7R and 8R. The virtual button 7R is an object corresponding to the first button of the controller 320R, while the virtual button 8R is an object corresponding to the second button of the controller 320R. The right virtual controller 6R may move together with the right hand object 5R. The virtual button 7R of the right virtual controller 6R may be pressed by the finger of the right hand object 5R by pressing the first button of the controller 320R by the user U in the real space.

  Further, the control unit 121 identifies a visual aspect such as the shape of the left virtual controller 6L (right virtual controller 6R) based on the information indicating the type of the controller 320L (320R) transmitted from the controller 320L (320R). May be.

  The first instruction object 3 is configured to guide the user U's hand (the entire hand) to perform a guiding operation. Here, the guiding operation refers to the operation of the user U guided by the first pointing object 3. In particular, the first indication object 3 is configured to guide the user U's hand (movement of the entire hand) to cause the first movement. In the present embodiment, an operation instruction “Please move your left hand from the bottom to the top” is displayed on the first instruction object 3. The user U is guided to move the left hand from the bottom to the top by visually recognizing the first indication object 3. The first instruction object 3 may be fixedly arranged in the virtual space 200A.

  The second instruction object 4 is configured to guide the user U's finger to perform a guiding operation. Here, the guiding operation refers to the operation of the user U guided by the second pointing object 4. In particular, the second pointing object 4 is configured to guide the user U's finger to make a second movement. In the present embodiment, an operation instruction “Please press the first button and the second button of the left controller” is displayed on the second instruction object 4. The user U is guided to press the first button and the second button of the controller 320L by visually recognizing the second pointing object 4.

  The first connected object 12 is configured to connect the left hand object 5L and the second pointing object 4 so as to be visually associated with each other. In this way, the second linked object 4 is visually associated with the left hand object 5L by the first connected object 12. In addition, the visual aspect (shape, etc.) of the first connected object 12 may be updated so that the left hand object 5L and the second pointing object 4 are continuously visually associated with the movement of the left hand object 5L. In this regard, the control unit 121 updates the visual aspect of the first connected object 12 after updating the position of the left hand object 5L. The longer the distance between the left hand object 5L and the second pointing object 4 is, the longer the length of the first connected object 12 is. On the contrary, the shorter the distance between the left hand object 5L and the second pointing object 4 is, the smaller the length of the first connected object 12 is. The display form of the first connected object 12 is not particularly limited as long as the user U can visually recognize that the second pointing object 4 is visually associated with the left hand object 5L by the first connected object 12. For example, a part of the first connected object 12 may not be visualized (it may be made transparent).

  The first connected object 13 is configured to connect the right hand object 5R and the second pointing object 4 so as to be visually associated with each other. Thus, the second linked object 4 is visually associated with the right hand object 5R by the first connected object 13. Further, the visual mode (shape, etc.) of the first connected object 13 may be updated so that the right hand object 5R and the second pointing object 4 are kept visually associated with each other in accordance with the movement of the right hand object 5R. The display form of the first connected object 13 is not particularly limited as long as the user U can visually recognize that the second pointing object 4 is visually associated with the right hand object 5R by the first connected object 13. For example, a part of the first linked object 13 may not be visualized.

  The second connection object 14 is configured to connect the virtual button 7L of the left virtual controller 6L and the second instruction object 4. By the second connection object 14, the second instruction object 4 is visually associated with the virtual button 7L. The display form of the second connected object 14 is not particularly limited. For example, a part of the second connected object 14 may not be visualized.

  The second connection object 16 is configured to connect the virtual button 8L of the left virtual controller 6L and the second instruction object 4. By the second connection object 16, the second instruction object 4 is visually associated with the virtual button 8L. Further, the visual mode (shape, etc.) of the second connected objects 14 and 16 may be updated according to the movement of the left hand object 5L. The display form of the second connected object 16 is not particularly limited. For example, a part of the second connected object 16 may not be visualized. In this embodiment, since the operation instruction “Please press the first button and the second button of the left controller” is displayed on the second instruction object 4, the two second linked objects 14 and 16 are displayed. Has been generated. On the other hand, when the operation instruction “Please press the first button of the left controller” is displayed on the second instruction object 4, only the second linked object 14 may be generated.

  In the present embodiment, when the left hand of the user U is moved from the bottom to the top and then the action of taking out the virtual sword (an example of the target object) is executed by pressing the first button and the second button of the controller 320L, The first instruction object 3 indicates a hand movement (first movement) associated with the action, and the second instruction object 4 indicates a finger movement (second movement) associated with the action. . In this way, the user U moves the hand so that the movement of the hand associated with the action of taking out the virtual sword (that is, moving the left hand from the bottom to the top) by visually recognizing the first pointing object 3. In addition, by visually recognizing the second indication object 4, the finger is moved so as to be a finger movement associated with the action (that is, pressing the first button and the second button of the controller 320 </ b> L). be able to. In this way, in the tutorial, the user U visually recognizes the first instruction object 3 and the second instruction object 4, so that the movement of the hand and the finger associated with the action related to the appearance of the virtual sword in the virtual space 200 </ b> A are displayed. You can learn movements relatively easily. Thus, the convenience of the user in the virtual space can be improved.

  In addition, according to the present embodiment, the first connected objects 12 and 13 can guide the user U's line of sight to the second pointing object 4. Further, the visual aspect of the first connected object 12 is updated according to the movement of the left hand object 5L, and the visual aspect of the first connected object 13 is updated according to the movement of the right hand object 5R. Thus, even when the left hand object 5L or the right hand object 5R is moved, the line of sight of the user U can be guided to the second pointing object 4 by the first connected object.

  Further, the user U's line of sight can be guided to the virtual buttons 7L and 8L and the second pointing object 4 by the second connected objects 14 and 16. For this reason, the user U can relatively easily move the finger associated with the predetermined action in the virtual space 200A by visually recognizing both the second pointing object 4 and the virtual buttons 7L and 8L in the tutorial. Can be learned.

  In the present embodiment, the action of taking out the virtual sword has been described as an example of the predetermined action. However, the present embodiment is applicable to various action tutorials. For example, by moving the user U's hand so that the movement of the user U's hand becomes the first movement, and moving the user U's finger so that the movement of the user U's finger becomes the second movement, When one action is executed, the first instruction object 3 guides the user U's hand to perform the first movement by displaying an operation instruction indicating the first movement. The second instruction object 4 guides the user U's finger to perform the second movement by displaying an operation instruction indicating the second movement.

  In the present embodiment, two first connected objects 12 and 13 are generated, but only one of the first connected objects 12 and 13 may be generated. The virtual space 200A is a connected display object that connects the first pointing object 3 and the left hand object 5L so as to be visually related and / or a connected display that connects the first pointing object 3 and the right hand object 5R so as to be visually related. An object may further be included. In this case, the first instruction object 3 is fixedly arranged in the virtual space 200A. For this reason, even when the first instruction object 3 is located outside the field of view CV of the virtual camera 300, the line of sight of the user U can be guided to the first instruction object 3 by the connected display object. In particular, since the connected display object is displayed on the visual field image, the user U's line of sight can be guided to the first indication object 3.

  The second instruction object 4 displays not only an operation instruction indicating the movement of the user U's finger related to the predetermined action but also an operation instruction indicating the movement of the user U's hand related to the predetermined action. Also good. In this case, the first instruction object 3 may not be arranged in the virtual space 200A.

  Next, processing for moving the second pointing object 4 will be described below with reference to FIGS. FIG. 9 is a schematic diagram illustrating a state in which the second pointing object 4 moves to a predetermined position within the field of view CV of the virtual camera 300. FIG. 10 is a flowchart for explaining processing for moving the second pointing object 4 to a predetermined position in the field of view CV (also referred to as the field of view of the user U) of the virtual camera 300.

  As shown in FIG. 9, as a precondition, it is assumed that the second pointing object 4 is located outside the field of view CV of the virtual camera 300 (the field of view of the user U) with the movement of the field of view CV of the virtual camera 300. At this time, as shown in FIG. 10, the control unit 121 determines whether or not the second pointing object 4 is located within the field of view CV (step S10). Next, when the control unit 121 determines that the second instruction object 4 is not located in the field of view CV (NO in step S10), the second instruction object 4 continues to be disposed at a predetermined position in the field of view CV. In this way, the second pointing object 4 is moved. On the other hand, when the control unit 121 determines that the second pointing object 4 is located in the visual field CV (YES in step S10), the process ends.

  In the process of step S11, the control unit 121 may gradually move the second pointing object 4 toward a predetermined position in the visual field CV after a predetermined period after the operation of the virtual camera 300 ends. Alternatively, the control unit 121 may start the movement process of the second instruction object 4 in the next frame after determining that the second instruction object 4 is located outside the field of view CV. Moreover, the predetermined position in the visual field CV is not particularly limited.

  According to the present embodiment, according to the movement of the visual field CV of the virtual camera 300, the second pointing object 4 moves so as to continue being arranged at a predetermined position in the visual field CV. Thus, since the 2nd instruction | indication object 4 follows the visual field CV, the convenience of the user U in virtual space can be improved. In other words, since the second pointing object 4 is displayed at a predetermined position on the visual field image, the convenience of the user U in the virtual space can be improved.

(Second Embodiment)
Next, a second embodiment of the present disclosure will be described below with reference to FIGS. 11 and 12. FIG. 11 is a diagram schematically illustrating a virtual space 200B according to the second embodiment. FIG. 12 is a diagram for explaining the movement of the avatar 9 (an example of the pointing object) arranged in the virtual space 200B. In particular, FIG. 12 shows the right side of the avatar 9 performing hand movements related to the action of taking out the virtual sword 80 (an example of a target object). The state (a) in FIG. 12 shows the movement of the left hand 93L of the avatar 9 in the first stage of the action of taking out the virtual sword. The state (b) in FIG. 12 shows the movement of the left hand 93L in the second stage of the action. The state (c) in FIG. 12 shows the movement of the left hand 93L in the third stage of the action. The state (d) of FIG. 12 shows the avatar 9 from which the virtual sword 80 has been taken out. The virtual space 200 </ b> B according to the second embodiment is different from the virtual space 200 </ b> A according to the first embodiment in that an avatar 9 is arranged on the virtual space instead of the first indication object 3.

  As shown in FIG. 12, the avatar 9 is configured to guide the user U's hand movement (movement of the entire hand) to cause the first movement. In particular, the avatar 9 includes a left hand 93L corresponding to the left hand of the user U who should execute the first movement. Thus, the avatar 9 is visually associated with the left hand object 5L. The operation of the avatar 9 is controlled by the control unit 121 (that is, the avatar 9 is an avatar controlled by a processor, not an avatar operated by another user). In the present embodiment, the avatar 9 shows a model of hand movement associated with the action of taking out the virtual sword 80. Specifically, the avatar 9 shows an example of the left hand movement associated with the action of taking out the virtual sword 80 by moving the left hand 93L (note that the avatar 9 does not move the right hand in this action). Is not moving his right hand). Thus, in the tutorial, the user U visually recognizes the movement of the left hand 93 </ b> L of the avatar 9 and the second pointing object 4, thereby moving the hand (first movement) associated with the action of taking out the virtual sword 80. The movement of the finger (second movement) can be learned relatively easily. Further, the control unit 121 generates the virtual sword 80 operated by the left hand object 5L in accordance with the finger movement executed following the hand movement associated with the action of taking out the virtual sword 80 (FIG. 12). State (d)).

  Further, since the avatar 9 shows a model of the left hand movement associated with the action of taking out the virtual sword 80, the avatar 9 is visually associated with the left hand object 5L. That is, since the user U moves his / her left hand by visually recognizing the left hand 93L of the avatar 9, the avatar 9 (more specifically, the left hand 93L of the avatar 9) is visually associated with the left hand object 5L. ing. Further, since the avatar 9 is controlled by the control unit 121 so that only the left hand 93L related to the action of taking out the virtual sword 80 is moved, the line of sight of the user U can be guided to the left hand 93L of the avatar 9. When the action to be learned is related to the movement of the right hand of the user U, the avatar 9 is controlled by the control unit 121 so that only the right hand of the avatar 9 moves. In this case, the right hand of the avatar 9 is associated with the right hand object 5 </ b> R, and the user U's line of sight can be guided to the right hand of the avatar 9.

  Further, as shown in FIG. 11, since the avatar 9 faces away from the virtual camera 300, the avatar 9 is displayed on the visual field image with the back facing away from the user U. In particular, when the avatar 9 is present in the visual field CV of the virtual camera 300, the control unit 121 sets the orientation of the avatar 9 so that the back surface of the avatar 9 is displayed on the visual field image. Since the avatar 9 faces away from the user U as described above, the direction of the movement of both the avatar 9 and the user U coincides. The movement can be imitated relatively easily.

  In the present embodiment, the virtual space 200B includes the connected display object and / or the avatar 9 (particularly the right hand of the avatar 9) that connects the avatar 9 (particularly, the left hand of the avatar 9) and the left hand object 5L so as to be visually associated. ) And the right hand object 5R may be further included.

  Further, in order to implement various processes executed by the control unit 121 of the user terminal 1 by software, a control program for causing the computer (processor) to execute various processes may be incorporated in the storage unit 123 or the memory in advance. Good. Alternatively, the control program can be a magnetic disk (HDD, floppy disk), optical disk (CD-ROM, DVD-ROM, Blu-ray (registered trademark) disk, etc.), magneto-optical disk (MO, etc.), flash memory (SD card, It may be stored in a computer-readable storage medium such as a USB memory or SSD. In this case, when the storage medium is connected to the control device 120, the control program stored in the storage medium is incorporated into the storage unit 123. The control unit 121 executes various processes by loading the control program incorporated in the storage unit 123 onto the RAM and executing the loaded program.

  The control program may be downloaded from a computer on the communication network 30 via the communication interface 125. Similarly in this case, the downloaded control program is incorporated into the storage unit 123.

  As mentioned above, although embodiment of this indication was described, the technical scope of this invention should not be limitedly interpreted by description of this embodiment. This embodiment is an example, and it is understood by those skilled in the art that various modifications can be made within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalents thereof.

  In the present embodiment, the movement of the avatar's hand is controlled according to the movement of the external controller 320 indicating the movement of the user U's hand, but in the virtual space according to the movement amount of the user U's hand itself. The movement of the hand object may be controlled. For example, instead of using an external controller, by using a glove-type device or a ring-type device worn on the user's finger, the position sensor 130 can detect the position and movement amount of the user U's hand, The movement and state of the user U's finger can be detected. Further, the position sensor 130 may be a camera configured to image the user U's hand (including a finger). In this case, the position and amount of movement of the user U's hand can be captured based on the image on which the user's hand is displayed without capturing any device directly on the user's finger by imaging the user's hand using the camera. And the movement and state of the finger of the user U can be detected.

  In the present embodiment, hand objects (left hand object and right hand object) are moved according to the movement of the hands (left hand and right hand) that are part of the user U's body. It is not limited. For example, the foot objects (left foot object and right foot object) may be moved according to the movement of the user U's foot (left foot, right foot) which is a part of the user U's body. When the predetermined action is related to the movement of the foot of the user U, an instruction object that is guided to cause the user U's foot to perform a guiding motion and is visually associated with the foot object may be generated.

  In this embodiment, the virtual space is used to provide a virtual experience such as VR (Virtual Reality), AR (Arranged Reality), and MR (Mixed Reality) to the user. When the virtual space provides VR, background data stored in the memory is used as the background of the virtual space. When the virtual space provides AR or MR, the real space is used as the background of the virtual space. In this case, the HMD 110 includes a transmissive display device (optical see-through or video see-through display device), so that the real space can be used as the background of the virtual space. If virtual space is applied to MR, the object may be influenced by real space. As described above, when the virtual space includes at least a part of a virtual scene such as a background or a virtual object, the user can be provided with a virtual experience capable of interacting with the virtual scene. When the virtual space is applied to AR or MR, the user's hand may be used instead of the hand object. In this case, the user's left hand is placed in the virtual space 200A instead of the left hand object 5L shown in FIG. 8, and the user's right hand is placed in the virtual space 200A instead of the right hand object 5R. The second pointing object 4 is visually associated with the left hand of the user by the first connected object 12 and visually associated with the right hand of the user by the first connected object 13.

1: User terminal 3: First indication object 4: Second indication object 5L: Left hand object (an example of an operation object)
5R: Right hand object (an example of an operation object)
6L: Left virtual controller 6R: Right virtual controller 7L: Virtual button 7R: Virtual button 8L: Virtual button 8R: Virtual button 9: Avatar 12, 13: First connected object 14, 16: Second connected object 30: Communication network 80 : Virtual sword 93L: Left hand 110: Head mounted device (HMD)
112: Display unit 114: Sensor 116: Headphone 118: Microphone 120: Control device 121: Control unit 123: Storage unit 124: I / O interface 125: Communication interface 126: Bus 130: Position sensor 140: Gaze sensor 200, 200A, 200B: Virtual space 210: Center position 300: Virtual camera 320: External controller 320L: Left hand external controller (controller)
320R: External controller for right hand (controller)

Claims (12)

a) generating virtual space data defining a virtual space including an operation object and an instruction object;
b) generating visual field image data indicating a visual field image displayed on the head mounted device based on the movement of the head mounted device and the virtual space data;
c) moving the operation object in response to movement of a part of the body other than the head of the user wearing the head-mounted device;
Including
The pointing object is
Visually associated with the manipulation object and guiding the part of the user's body to perform a guiding action;
An information processing method executed by a processor.   The information processing method according to claim 1, wherein the virtual space further includes a first connection object that connects the operation object and the instruction object so as to be visually associated with each other.   3. The method according to claim 2, further comprising: d) updating the visual aspect of the first connected object so as to keep the operation object and the pointing object visually associated with each other according to the movement of the operation object. Information processing method. The virtual space is
A virtual controller having a virtual button and associated with the operation object;
A second connected object for connecting the pointing object and the virtual button so as to be visually associated;
The information processing method according to claim 2 or 3.   5. The method according to claim 1, further comprising: e) moving the pointing object so that the pointing object continues to be disposed at a predetermined position in the visual field in accordance with the movement of the user's visual field. Information processing method described in 1.   The information processing method according to any one of claims 1 to 4, wherein the instruction object is fixedly arranged in the virtual space. The pointing object is an avatar arranged in the virtual space,
The pointing object is visually associated with the operation object by including a part of the virtual body of the avatar corresponding to the part of the user's body that is to perform the guidance operation,
When the movement of the part performs a guiding operation, the pointing object guides the part of the user's body to perform the guiding operation.
The information processing method according to claim 6.   The information processing method according to claim 7, wherein when the avatar exists in the user's field of view, the orientation of the avatar is set so that a back surface of the avatar is displayed on the field-of-view image.   The information processing method according to claim 7 or 8, wherein the avatar is controlled so that only the portion of the virtual body moves. The guiding operation includes a first movement and a second movement,
The part of the user's body includes a hand and a finger;
The pointing object includes a first pointing object that guides the hand to make the first movement, a second pointing object that guides the finger to make the second movement,
Have
f) further comprising generating a target object operated by the operation object in response to the second movement executed subsequent to the first movement.
The information processing method according to any one of claims 1 to 3.   An information processing program for causing a computer to execute the information processing method according to any one of claims 1 to 10. A processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
The information processing apparatus that executes the information processing method according to any one of claims 1 to 10, when the computer-readable instruction is executed by the processor.

Images