JP6580624B2 - Method for providing virtual space, program for causing computer to execute the method, and information processing apparatus for executing the program - Google Patents

Description

  This disclosure relates to a technique for providing a virtual space, and more particularly to a technique for displaying a user interface.

  A technique for providing a virtual space (virtual reality space) using a head-mounted device (HMD) device is known. For example, a technique is known in which an avatar that operates in conjunction with a user's movement is arranged on the virtual space to enhance the user's sense of immersion in the virtual space.

  In recent years, technology development for making user operations comfortable in a virtual space has been actively carried out. For example, Japanese Patent Laying-Open No. 2015-032085 (Patent Document 1) states that “a display control unit that displays a plurality of icons (menus) for calling each application on a display screen, and a display unit that displays a plurality of icons. The display control unit discloses a configuration in which a plurality of icons are displayed in a retreat area that avoids the view securing area on the display screen (see “Summary”).

Japanese Patent Laid-Open No. 2015-032085

  Since the technique described in Patent Literature 1 employs a configuration in which a plurality of icons (menus) are displayed in the retreat area at the end of the display unit, the substantial field of view of the user in the virtual space is the view securing area (of the display unit). Only in the center). In this case, since the field of view of the user in the virtual space is narrow, the user may feel stress.

  Moreover, since the method described in Patent Document 1 employs a configuration in which icons are constantly displayed on the end of the display unit, there is a possibility that the user's sense of immersion in the virtual space may be impaired. Therefore, there is a need for a technique that allows a user to operate a menu comfortably in a virtual space.

  The present disclosure has been made in order to solve the above-described problem, and an object in one aspect is to provide a user who can operate comfortably while suppressing a user's feeling of being immersed in a virtual space from being impaired. Is to provide an interface.

  In certain embodiments, a computer-implemented method is provided for providing virtual space to a head mounted device. The method includes the steps of defining a virtual space, displaying a field of view image on a display of the head mounted device to provide the virtual space to a user of the head mounted device, detecting movement of the head mounted device, and detecting A step of updating the view image in conjunction with the made movement, a step of arranging an avatar object corresponding to the user in the user's view in the virtual space, and a user interface for receiving an operation from the user on the avatar object Steps.

  The above and other objects, features, aspects and advantages of the disclosed technical features will become apparent from the following detailed description of the invention which is to be understood in connection with the accompanying drawings.

It is a figure showing the outline of a structure of a HMD system. It is a block diagram showing an example of the hardware constitutions of the computer according to a certain situation. It is a schematic diagram which represents notionally the uvw visual field coordinate system set to HMD according to an embodiment. It is a schematic diagram which represents notionally the one aspect | mode which represents the virtual space according to a certain embodiment. It is the schematic diagram showing the head of the user who wears HMD according to an embodiment from the top. It is a figure showing the YZ cross section which looked at the visual recognition area from the X direction in virtual space. It is a figure showing the XZ cross section which looked at the visual recognition area from the Y direction in virtual space. FIG. 2 is a block diagram illustrating a computer according to an embodiment as a modular configuration. It is a figure for demonstrating the hardware constitutions and module constitution of a server. It is a flowchart showing the process in a HMD system. It is a figure for demonstrating the process which tracks a hand. It is a figure for demonstrating operation | movement of a tracking module. It is a figure showing an example of the data structure of tracking data. It is a figure showing the visual field image which a user visually recognizes. It is a figure for demonstrating the display method of a user interface. It is a figure showing a visual field image when the user is operating UI. It is a figure for demonstrating a user's operation | movement in the real space corresponding to FIG. It is a flowchart showing the process which displays a user interface on an avatar object. It is a figure showing the other display mode of a screen object. It is a figure for demonstrating the process which does not display UI. It is a figure for demonstrating operation based on a user's eyes | visual_axis. In a network, it is a figure showing typically the situation where each of a plurality of HMD provides virtual space to each of a plurality of users. It is a figure showing the screen object of the other user at the time of an out camera mode. It is a figure showing the screen object of the other user at the time of in-camera mode. It is a flowchart showing the process which changes the display mode of a screen object according to a mode. It is a figure for demonstrating the arrangement | positioning relationship between the avatar object and screen object in virtual space. It is a figure showing the visual field image which a user visually recognizes in the state of Drawing 26. It is a figure for demonstrating the arrangement | positioning relationship between the avatar object and screen object in another situation. It is a flowchart showing the process which changes the display mode of another screen object according to the position of an avatar object. It is a figure showing an example of the data structure of picked-up image DB according to a certain embodiment.

  Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. Each embodiment described below may be combined appropriately and appropriately.

[Configuration of HMD system]
A configuration of an HMD (Head-Mounted Device) system 100 will be described with reference to FIG. FIG. 1 shows an outline of the configuration of the HMD system 100. The HMD system 100 is provided as a home system or a business system.

HMD system 100 includes HM D Se Tsu DOO 105A, 105B, 105C, and 105D, the network 19 and the server 150. Each of the HMD sets 105A, 105B, 105C, and 105D is configured to be able to communicate with the server 150 via the network 19. Hereinafter, the HMD sets 105A, 105B, 105C, and 105D are collectively referred to as the HMD set 105. The number of HMD sets 105 constituting the HMD system 100 is not limited to four, and may be three or less or five or more.

  The HMD set 105 includes an HMD 110, an HMD sensor 120, and a computer 200. The HMD 110 includes a monitor 112, a camera 116, a speaker 118, a microphone 119, and a gaze sensor 140. In another aspect, the HMD 110 further includes a sensor 114.

  In one aspect, the computer 200 can be connected to the Internet and other networks 19, and can communicate with the server 150 and other computers (for example, computers of other HMD sets 105) connected to the network 19.

  The HMD 110 may be worn on the head of the user 190 and provide a virtual space to the user 190 during operation. More specifically, the HMD 110 displays a right-eye image and a left-eye image on the monitor 112, respectively. When each eye of the user 190 visually recognizes each image, the user 190 can recognize the image as a three-dimensional image based on the parallax of both eyes. The HMD 110 may include both a so-called head mounted display including a monitor and a head mounted device on which a terminal having a smartphone or other monitor can be attached.

  The monitor 112 is realized as, for example, a non-transmissive display device. In one aspect, the monitor 112 is disposed on the main body of the HMD 110 so as to be positioned in front of both eyes of the user 190. Therefore, when the user 190 visually recognizes the three-dimensional image displayed on the monitor 112, the user 190 can be immersed in the virtual space. In some embodiments, the virtual space includes, for example, an image of a background, objects that can be manipulated by the user 190, and menus that can be selected by the user 190. In an embodiment, the monitor 112 may be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor included in a so-called smartphone or other information display terminal.

  In another aspect, the monitor 112 can be realized as a transmissive display device. In this case, the HMD 110 may be an open type such as a glasses type instead of a sealed type that covers the eyes of the user 190 as shown in FIG. The transmissive monitor 112 can temporarily function as a non-transmissive display device by adjusting the transmittance. Further, the monitor 112 may include a configuration in which a part of an image constituting the virtual space and the real space are displayed simultaneously. For example, the monitor 112 may display an image of the real space photographed by a camera mounted on the HMD 110, or may make the real space visible by setting a part of the transmittance high.

  In one aspect, the monitor 112 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 112 may be configured to display a right-eye image and a left-eye image together. In this case, the monitor 112 includes a high-speed shutter. The high-speed shutter operates so that an image for the right eye and an image for the left eye can be displayed alternately so that the image is recognized only by one of the eyes.

  The camera 116 is configured to be able to acquire depth information of an object. As an example, the camera 116 acquires depth information of an object according to a TOF (Time Of Flight) method. As another example, the camera 116 acquires depth information of an object according to a pattern irradiation method. In some embodiments, the camera 116 may be a stereo camera that can capture an object from two or more different directions. Further, the camera 116 may be an infrared camera that is invisible to humans. The camera 116 is attached to the HMD 110 and photographs a part of the user 190's body. Hereinafter, as an example, the camera 116 captures the hand of the user 190. The camera 116 outputs the acquired depth information of the hand of the user 190 to the computer 200.

  The speaker 118 converts the audio signal into audio and outputs it to the user 190. The microphone 119 converts the utterance of the user 190 into an audio signal that is an electrical signal and outputs it to the computer 200. In another aspect, HMD 110 may be configured to include an earphone instead of speaker 118.

  The HMD sensor 120 detects the position and inclination of the HMD 110. In this case, the HMD 110 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the light emitted from each light source and detecting the position and orientation of the HMD 110.

  In other aspects, HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and inclination of the HMD 110 by executing image analysis processing using image information of the HMD 110 output from the camera.

  In still another aspect, the computer 200 may be configured to detect the inclination of the HMD 110 based on the output of the sensor 114 instead of the output of the HMD sensor 120. The sensor 114 is realized by, for example, an angular velocity sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, or a combination of these sensors. The computer 200 detects the inclination of the HMD 110 based on the output of the sensor 114. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects angular velocities around the three axes of the HMD 110 in real space over time. The computer 200 calculates the inclination of the HMD 110 by calculating temporal changes of the angles around the three axes of the HMD 110 based on the angular velocities.

  The gaze sensor 140 detects a direction (line of sight) in which the line of sight of the right eye and the left eye of the user 190 is directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, the gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris with respect to the irradiated light. . The gaze sensor 140 can detect the line of sight of the user 190 based on each detected rotation angle.

  Server 150 may send a program to computer 200. In another aspect, the server 150 may communicate with other computers 200 for providing virtual reality to HMDs used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's operation with another computer 200, and a plurality of users are common in the same virtual space. Allows you to enjoy the game.

[Hardware configuration]
A computer 200 according to this embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of a hardware configuration of computer 200 according to an aspect. The computer 200 includes a processor 10, a memory 11, a storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

  The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on a signal given to the computer 200 or based on the establishment of a predetermined condition. In one aspect, the processor 10 is realized as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array), or other device.

  The memory 11 temporarily stores programs and data. The program is loaded from the storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In one aspect, the memory 11 is realized as a RAM (Random Access Memory) or other volatile memory.

  The storage 12 holds programs and data permanently. The storage 12 is realized as, for example, a ROM (Read-Only Memory), a hard disk device, a flash memory, and other nonvolatile storage devices. The programs stored in the storage 12 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data and objects for defining the virtual space.

  In another aspect, the storage 12 may be realized as a removable storage device such as a memory card. In still another aspect, a configuration using a program and data stored in an external storage device may be used instead of the storage 12 built in the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used as in an amusement facility, it is possible to update programs and data collectively.

  In some embodiments, the input / output interface 13 communicates signals between the HMD 110 and the HMD sensor 120. In an aspect, the sensor 114, the camera 116, the speaker 118, and the microphone 119 included in the HMD 110 can communicate with the computer 200 via the input / output interface 13 of the HMD 110. In one aspect, the input / output interface 13 is implemented using a USB (Universal Serial Bus), a DVI (Digital Visual Interface), an HDMI (registered trademark) (High-Definition Multimedia Interface), or other terminals. The input / output interface 13 is not limited to that described above.

  The communication interface 14 is connected to the network 19 and communicates with other computers (for example, the server 150) connected to the network 19. In one aspect, the communication interface 14 is realized as, for example, a local area network (LAN) or other wired communication interface, or a wireless communication interface such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication), or the like. Is done. The communication interface 14 is not limited to the above.

  In one aspect, the processor 10 accesses the storage 12, loads one or more programs stored in the storage 12 into the memory 11, and executes a series of instructions included in the program. The one or more programs may include an operating system of the computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space, and the like. The processor 10 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 112 based on the signal.

  In the example illustrated in FIG. 2, the computer 200 is configured to be provided outside the HMD 110. However, in another aspect, the computer 200 may be incorporated in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including the monitor 112 may function as the computer 200.

  Further, the computer 200 may be configured to be used in common for a plurality of HMDs 110. According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space.

  In an embodiment, in the HMD system 100, a global coordinate system is preset. The global coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-rear direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (up-down direction), and the front-rear direction in the global coordinate system are defined as an x-axis, a y-axis, and a z-axis, respectively. More specifically, in the global coordinate system, the x axis is parallel to the horizontal direction of the real space. The y axis is parallel to the vertical direction of the real space. The z axis is parallel to the front-rear direction of the real space.

  In one aspect, the sensor 114 is configured by a combination of a triaxial angular velocity sensor and a triaxial acceleration sensor. The computer 200 calculates an angle with respect to the reference direction (for example, the gravity (vertical) direction) of the HMD 110 based on the outputs of these sensors. Thereby, the computer 200 can acquire the inclination of the HMD 110 in the global coordinate system.

  In another aspect, HMD sensor 120 includes an infrared sensor. When the infrared sensor detects the infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the position of each light source (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120 can detect temporal changes in the position and inclination of the HMD 110 using each value detected over time.

  In one aspect, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between a plurality of points (for example, the distance between the points). Can be specified as a relative position with respect to the HMD sensor 120. Moreover, the processor 10 can determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

  The global coordinate system is parallel to the real space coordinate system. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination around the three axes of the HMD 110 in the global coordinate system. The computer 200 sets the uvw visual field coordinate system to the HMD 110 based on the inclination of the HMD 110 in the global coordinate system. The uvw visual field coordinate system set in the HMD 110 corresponds to a viewpoint coordinate system when the user 190 wearing the HMD 110 views an object in the virtual space.

[Uvw visual field coordinate system]
The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a schematic diagram conceptually showing the uvw visual field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and inclination of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 10 sets the uvw visual field coordinate system to the HMD 110 based on the detected value.

  As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system with the head (origin) of the user 190 wearing the HMD 110 as the center (origin). More specifically, the HMD 110 includes a horizontal direction, a vertical direction, and a front-rear direction (x-axis, y-axis, z-axis) that define the global coordinate system by an inclination around each axis of the HMD 110 in the global coordinate system. Three directions newly obtained by tilting around the axis are set as the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw visual field coordinate system in the HMD 110.

  In a certain situation, when the user 190 wearing the HMD 110 stands upright and is viewing the front, the processor 10 sets the uvw visual field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch axis (u-axis) and yaw axis (v-axis) of the uvw visual field coordinate system in the HMD 110. , And the roll axis (w axis).

  After the uvw visual field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the inclination (the amount of change in inclination) of the HMD 110 in the set uvw visual field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110. The pitch angle (θu) represents the inclination angle of the HMD 110 around the pitch axis in the uvw visual field coordinate system. The yaw angle (θv) represents the inclination angle of the HMD 110 around the yaw axis in the uvw visual field coordinate system. The roll angle (θw) represents the inclination angle of the HMD 110 around the roll axis in the uvw visual field coordinate system.

  The computer 200 sets the uvw visual field coordinate system in the HMD 110 after the HMD 110 has moved to the HMD 110 based on the tilt angle of the HMD 110. The relationship between the HMD 110 and the uvw visual field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

[Virtual space]
The virtual space will be further described with reference to FIG. FIG. 4 is a schematic diagram conceptually showing one aspect of expressing the virtual space 2 according to an embodiment. The virtual space 2 has a spherical structure that covers the entire 360 ° direction of the center 21. In FIG. 4, the upper half of the celestial sphere in the virtual space 2 is illustrated in order not to complicate the description. In the virtual space 2, each mesh is defined. The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system defined in the virtual space 2. The computer 200 develops each partial image constituting the panoramic image 22 on each corresponding mesh. Thereby, the user 190 visually recognizes the virtual space 2.

  In one aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, vertical direction (vertical direction), and front-rear direction in the XYZ coordinate system are defined as an X axis, a Y axis, and a Z axis, respectively. Therefore, the X axis (horizontal direction) of the XYZ coordinate system is parallel to the x axis of the global coordinate system, the Y axis (vertical direction) of the XYZ coordinate system is parallel to the y axis of the global coordinate system, and The Z axis (front-rear direction) is parallel to the z axis of the global coordinate system.

  When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 1 is disposed at the center 21 of the virtual space 2. In one aspect, the processor 10 displays an image captured by the virtual camera 1 on the monitor 112 of the HMD 110. The virtual camera 1 similarly moves in the virtual space 2 in conjunction with the movement of the HMD 110 in the real space. Thereby, changes in the position and orientation of the HMD 110 in the real space can be similarly reproduced in the virtual space 2.

  As with the HMD 110, the uvw visual field coordinate system is defined for the virtual camera 1. The uvw visual field coordinate system of the virtual camera 1 in the virtual space 2 is defined so as to be linked to the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 1 also changes accordingly. The virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space.

  The processor 10 of the computer 200 defines the viewing area 23 in the virtual space 2 based on the position of the virtual camera 1 and the tilt direction of the virtual camera 1, in other words, the reference line of sight 5 indicating the shooting direction of the virtual camera 1. To do. The visual recognition area 23 corresponds to an area of the virtual space 2 that is visually recognized by the user 190 wearing the HMD 110. As described above, the uvw visual field coordinate system of the virtual camera 1 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, the reference line of sight 5 is determined by the inclination of the HMD 110.

  The line of sight of the user 190 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 190 visually recognizes the object. The uvw visual field coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 112. Further, the uvw visual field coordinate system of the virtual camera 1 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, the HMD system 100 according to an aspect can regard the line of sight of the user 190 detected by the gaze sensor 140 as the line of sight of the user 190 in the uvw visual field coordinate system of the virtual camera 1.

[User's line of sight]
The determination of the line of sight of the user 190 will be described with reference to FIG. FIG. 5 is a schematic diagram showing the head of the user 190 wearing the HMD 110 according to an embodiment from above.

  In one aspect, gaze sensor 140 detects each line of sight of user 190's right eye and left eye. In a certain aspect, when the user 190 is looking near, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll axis w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll axis w. The gaze sensor 140 transmits the detection result to the computer 200.

  When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the line-of-sight detection result, the computer 200 identifies the point of sight N1 that is the intersection of the lines of sight R1 and L1 based on the detection value. On the other hand, when the detected values of the lines of sight R2 and L2 are received from the gaze sensor 140, the computer 200 specifies the intersection of the lines of sight R2 and L2 as the point of sight. The computer 200 specifies the line of sight N0 of the user 190 based on the specified position of the gazing point N1. For example, the computer 200 detects, as the line of sight N0, the extending direction of the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1. The line of sight N0 is a direction in which the user 190 is actually pointing the line of sight with both eyes. The line of sight N0 corresponds to the direction in which the user 190 is actually pointing the line of sight with respect to the visual recognition area 23.

  In another aspect, HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 2.

  In still another aspect, the HMD system 100 may include a communication circuit for connecting to the Internet or a call function for connecting to a telephone line.

[Visibility area]
The visual recognition area 23 will be described with reference to FIGS. FIG. 6 shows a YZ cross section of the visual recognition area 23 viewed from the X direction in the virtual space 2. FIG. 7 represents an XZ cross section of the visual recognition area 23 viewed from the Y direction in the virtual space 2.

  As shown in FIG. 6, the visual recognition area 23 in the YZ cross section includes an area 24. The region 24 is defined by the arrangement position of the virtual camera 1, the reference line of sight 5, and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α around the reference line of sight 5 in the virtual space as the region 24.

  As shown in FIG. 7, the visual recognition area 23 in the XZ cross section includes an area 25. The region 25 is defined by the arrangement position of the virtual camera 1, the reference line of sight 5, and the XZ section of the virtual space 2. The processor 10 defines a range including the azimuth angle β around the reference line of sight 5 in the virtual space 2 as a region 25. The polar angles α and β are determined according to the arrangement position of the virtual camera 1 and the orientation of the virtual camera 1.

  In one aspect, the HMD system 100 provides the user 190 with a visual field in the virtual space by displaying the visual field image 26 on the monitor 112 based on a signal from the computer 200. The view image 26 corresponds to a portion of the panoramic image 22 that is superimposed on the viewing area 23. When the user 190 moves the HMD 110 worn on the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the visual recognition area 23 in the virtual space 2 changes. As a result, the view field image 26 displayed on the monitor 112 is updated to an image that is superimposed on the viewing area 23 of the panoramic image 22 in the direction in which the user 190 faces in the virtual space 2. The user 190 can visually recognize a desired direction in the virtual space 2.

  Thus, the orientation (tilt) of the virtual camera 1 corresponds to the line of sight of the user 190 (reference line of sight 5) in the virtual space 2, and the position where the virtual camera 1 is arranged corresponds to the viewpoint of the user 190 in the virtual space 2. To do. Therefore, by moving the virtual camera 1 (including an operation for changing the arrangement position and an operation for changing the orientation), the image displayed on the monitor 112 is updated, and the field of view of the user 190 is moved.

  While wearing the HMD 110, the user 190 can visually recognize only the panoramic image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the HMD system 100 can give the user 190 a high sense of immersion in the virtual space 2.

  In one aspect, the processor 10 can move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 10 specifies an image area (that is, the visual recognition area 23 in the virtual space 2) projected on the monitor 112 of the HMD 110 based on the position and orientation of the virtual camera 1 in the virtual space 2.

  According to an embodiment, the virtual camera 1 may include two virtual cameras: a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In addition, appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In this embodiment, the virtual camera 1 includes two virtual cameras, and the roll axis (w) generated by combining the roll axes of the two virtual cameras is adapted to the roll axis (w) of the HMD 110. The technical idea which concerns on this indication is illustrated as what is comprised in this way.

[HMD control device]
A control device of the HMD 110 will be described with reference to FIG. In an embodiment, the control device is realized by a computer 200 having a known configuration. FIG. 8 is a block diagram illustrating a computer 200 according to an embodiment as a modular configuration.

  As shown in FIG. 8, the computer 200 includes a display control module 220, a virtual space control module 230, a memory module 240, and a communication control module 250. The display control module 220 includes a virtual camera control module 221, a view area determination module 222, a view image generation module 223, a reference line of sight identification module 224, a motion detection module 225, and a tracking module 226 as submodules. . The virtual space control module 230 includes a virtual space definition module 231, a virtual object generation module 232, an operation object control module 233, an avatar control module 234, and a photographing module 235 as submodules.

  In an embodiment, the display control module 220 and the virtual space control module 230 are realized by the processor 10. In another embodiment, multiple processors 10 may operate as the display control module 220 and the virtual space control module 230. The memory module 240 is realized by the memory 11 or the storage 12. The communication control module 250 is realized by the communication interface 14.

  In one aspect, the display control module 220 controls image display on the monitor 112 of the HMD 110.

  The virtual camera control module 221 arranges the virtual camera 1 in the virtual space 2. Further, the virtual camera control module 221 controls the position of the virtual camera 1 in the virtual space 2 and the tilt (shooting direction) of the virtual camera 1. The visual field area determination module 222 defines the visual recognition area 23 according to the position and inclination of the virtual camera 1. The view image generation module 223 generates a view image 26 displayed on the monitor 112 based on the determined viewing area 23. That is, the position of the virtual camera 1 corresponds to the viewpoint of the user 190 in the virtual space 2.

  The reference line-of-sight specifying module 224 specifies the inclination of the HMD 110 (the direction in which the user 190's head is facing) based on the output of the sensor 114 or the HMD sensor 120. In another aspect, the reference line-of-sight identification module 224 may identify the line of sight of the user 190 based on a signal from the gaze sensor 140.

  Based on the output of the HMD sensor 120, the motion detection module 225 can calculate a displacement amount with respect to a reference state of the HMD 110 (for example, a state (initial state) when the virtual space 2 is defined). The virtual camera control module 221 can control the position and inclination of the virtual camera 1 based on the calculated displacement amount.

  The tracking module 226 detects (tracks) the position of a part of the body of the user 190. In an embodiment, the tracking module 226 detects the position of the hand of the user 190 in the uvw visual field coordinate system set in the HMD 110 based on the depth information input from the camera 116. The operation of the tracking module 226 will be described later.

  The virtual space control module 230 controls the virtual space 2 provided to the user 190. The virtual space definition module 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2.

  The virtual object generation module 232 generates an object arranged in the virtual space 2. The objects may include, for example, forests, mountains and other landscapes, animals, etc. that are arranged according to the progress of the game story.

  The operation object control module 233 arranges an operation object for accepting an operation of the user 190 in the virtual space 2 in the virtual space 2. For example, the user 190 operates an object placed in the virtual space 2 by operating the operation object. In one aspect, the operation object may include a hand object corresponding to the hand of the user 190 wearing the HMD 110, for example. In one aspect, the operation object may correspond to a hand portion of an avatar object described later. In another aspect, the operation object includes an object (for example, a stick) held by the avatar object.

  The avatar control module 234 generates data for placing an avatar object corresponding to a user of another computer 200 connected via a network in the virtual space 2. In one aspect, the avatar control module 234 generates data for placing an avatar object corresponding to the user 190 in the virtual space 2. In one aspect, the avatar control module 234 generates an avatar object that imitates the user 190 based on the image of the user 190. In another aspect, the avatar control module 234 displays, in the virtual space 2, an avatar object that has been selected by the user 190 from a plurality of types of avatar objects (for example, an object imitating an animal or an object of a deformed person). Generate data for placement.

  The avatar control module 234 reflects the movement of the HMD 110 on the avatar object based on the output of the HMD sensor 120 (or sensor 114). For example, the avatar control module 234 detects that the position of the HMD 110 has moved, and generates data for moving the avatar object. Also, the avatar control module 234 reflects the movement of the hand of the user 190 in the real space on the hand of the avatar object based on the output of the tracking module 226. The avatar control module 234 controls the movement of the avatar object corresponding to the user of the other computer based on the data input from the other computer 200.

  The imaging module 235 images the virtual space 2. More specifically, the shooting module 235 places an object having a shooting function in the virtual space 2 and performs shooting using the object. An image generated by shooting is stored in the storage 12.

  The virtual space control module 230 detects the collision when each of the objects arranged in the virtual space 2 collides with another object. The virtual space control module 230 can detect, for example, the timing when a certain object and another object touch each other, and performs a predetermined process when the detection is performed. The virtual space control module 230 can detect the timing when the object is away from the touched state, and performs a predetermined process when the detection is made. The virtual space control module 230 can detect that the object is in a touched state. For example, when the operation object touches another object, the operation object control module 233 detects that the operation object touches another object, and performs a predetermined process.

  The memory module 240 holds data used for the computer 200 to provide the virtual space 2 to the user 190. In one aspect, the memory module 240 holds space information 241, object information 242, user information 243, and a captured image DB 244.

  The space information 241 holds one or more templates defined for providing the virtual space 2.

  The object information 242 holds a plurality of types of panoramic images 22, modeling data for configuring objects arranged in the virtual space 2, and object position information. The panoramic image 22 may include an unreal space image and a real space image.

  The object information 242 may further store motion detection data and tracking data. The motion detection data is data representing the position and inclination of the HMD 110 (output of the HMD sensor 120). The tracking data is data indicating the position of the hand of the user 190 acquired by the camera 116 and the tracking module 226. Such information can be updated at any time in the output cycle of the HMD sensor 120 or the imaging cycle of the camera 116.

  The user information 243 holds a program for causing the computer 200 to function as a control device of the HMD system 100, an application program that uses each content held in the object information 242, and the like. The user information 243 may include a user ID for identifying the user 190 (for example, an IP address or a MAC address set in the computer 200).

  The captured image DB 244 manages images (captured images) generated by the imaging module 235.

  Data and programs stored in the memory module 240 are input by the user 190 of the HMD 110. Alternatively, the processor 10 downloads a program or data from a computer (for example, the server 150) operated by a provider providing the content, and stores the downloaded program or data in the memory module 240.

  The communication control module 250 can communicate with the server 150 and other information communication devices via the network 19.

  In an aspect, a part of the display control module 220 and the virtual space control module 230 can be realized using, for example, Unity (registered trademark) provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

  Processing in the computer 200 is realized by hardware and software executed by the processor 10. Such software may be stored in advance in a memory module 240 such as a hard disk. The software may be stored in a CD-ROM or other non-volatile computer-readable data recording medium and distributed as a program product. Alternatively, the software may be provided as a program product that can be downloaded by an information provider connected to the Internet or other networks. Such software is read from a data recording medium by an optical disk drive or other data reader, or downloaded from the server 150 or other computer via the communication control module 250 and then temporarily stored in the storage module. . The software is read from the storage module by the processor 10 and stored in the RAM in the form of an executable program. The processor 10 executes the program.

[Structure of server 150]
FIG. 9 is a diagram for explaining the hardware configuration and module configuration of the server 150. In an embodiment, the server 150 includes a communication interface 910, a processor 920, and a storage 930 as main components.

  The communication interface 910 functions as a communication module for wireless communication that performs modulation / demodulation processing for transmitting / receiving signals to / from an external communication device such as the computer 200. The communication interface 910 is realized by a tuner, a high frequency circuit, or the like.

  The processor 920 controls the operation of the server 150. The processor 920 functions as a transmission / reception unit 922, a server processing unit 924, and a matching unit 926 by executing various control programs stored in the storage 930.

  The transmission / reception unit 922 transmits / receives various information to / from each computer 200. For example, the transmission / reception unit 922 transmits a request to place an object in the virtual space 2, a request to delete the object from the virtual space 2, a request to move the object, a user voice, and the like to each computer 200.

  The server processing unit 924 performs processing necessary for a plurality of users to share the virtual space 2. For example, the server processing unit 924 updates position information 936 described later based on information received from the computer 200.

  The matching unit 926 performs a series of processes for associating a plurality of users. For example, when a plurality of users perform an input operation for sharing the same virtual space 2, the matching unit 926 performs processing for associating user IDs of these users with each other.

  The storage 930 holds virtual space designation information 932, object designation information 934, avatar object information 935, user information 938, and a captured image DB 939.

  The virtual space designation information 932 is information used by the virtual space definition module 231 of the computer 200 to define the virtual space 2. In one aspect, the virtual space designation information 932 may include identification information of one or more templates that the computer 200 holds as the space information 241. In another aspect, the virtual space designation information 932 may include information for designating the size of the virtual space 2 and the panoramic image 22 developed on each mesh constituting the virtual space 2.

  The object designation information 934 is information for designating an object to be arranged (generated) by the virtual object generation module 232 of the computer 200 in the virtual space 2. In another aspect, the object designation information 934 may include modeling data for the virtual object generation module 232 to generate an object.

  The avatar object information 935 includes position information 936 and inclination information 937. The position information 936 represents the position (coordinates) of each avatar object in the virtual space 2. The inclination information 937 represents the inclination of the head of each avatar object in the virtual space 2. The inclination of the head of the avatar object is the inclination of the virtual camera 1 (reference line of sight 5). The avatar object information 935 can be updated as needed based on information input from the computer 200.

  The user information 938 is information about the user 190 of the computer 200. The user information 938 includes, for example, user IDs that identify the plurality of users 190 from each other. The captured image DB 939 includes information stored in the captured image DB 244 of each computer 200.

[Control structure of computer 200]
With reference to FIG. 10, the update method of the visual field image in the computer 200 is demonstrated. FIG. 10 is a flowchart showing processing in the HMD system 100.

  In step S <b> 1010, the processor 10 of the computer 200 defines the virtual space 2 as the virtual space definition module 231. For example, the virtual space definition module 231 defines the virtual space 2 with reference to the space information 241 based on the virtual space designation information 932 received from the server 150.

  In step S <b> 1020, the processor 10 places the virtual camera 1 in the virtual space 2 as the virtual camera control module 221. At this time, the processor 10 can place the virtual camera 1 in the center 21 defined in advance in the virtual space 2 in the work area of the memory.

  In step S1030, the processor 10 generates view image data for displaying the initial view image 26 as the view image generation module 223. The processor 10 outputs the generated view field image data to the HMD 110.

  In step S <b> 1032, the monitor 112 of the HMD 110 displays the view field image 26 based on the signal input from the computer 200. As a result, the user 190 wearing the HMD 110 recognizes the virtual space 2.

  In step S1034, HMD sensor 120 detects the position and inclination of HMD 110. The HMD sensor 120 outputs the detection result to the computer 200 as motion detection data.

  In step S1040, processor 10 detects the position and tilt of HMD 110 based on the motion detection data input from HMD sensor 120. The processor 10 can store the input motion detection data in the object information 242. Further, the processor 10 changes the position and inclination of the virtual camera 1 in the virtual space 2 so as to be interlocked with the detected position and inclination as the virtual camera control module 221. The viewing area determination module 222 defines the viewing area 23 according to the position and inclination of the virtual camera 1 after the change. Thereby, the field-of-view image 26 photographed by the virtual camera 1 is updated.

  In step S1050, the processor 10 generates, as the view image generation module 223, view image data for displaying the view image 26 captured by the virtual camera 1 whose inclination has been changed, and outputs the generated view image data to the HMD 110. To do.

  In step S1052, the monitor 112 of the HMD 110 displays the updated view image based on the received view image data. Thereby, the field of view of the user 190 in the virtual space 2 is updated.

[Hand tracking]
Hereinafter, processing for tracking hand movement will be described with reference to FIGS. FIG. 11 is a diagram for explaining the process of tracking the hand.

  Hereinafter, the user of the HMD set 105A is represented as a user 190A. In addition, A is added to the reference symbol of each component regarding the HMD set 105A. For example, the HMD 110A is included in the HMD set 105A.

  Referring to FIG. 11, user 190A wears HMD 110A in real space. A camera 116A is mounted on the HMD 110A. The camera 116A acquires depth information of an object included in the space 1100 in front of the HMD 110A. In the example illustrated in FIG. 11, the camera 116 </ b> A acquires depth information of the hand of the user 190 </ b> A included in the space 1100.

  The tracking module 226A generates hand position information (hereinafter also referred to as “tracking data”) based on the depth information. The camera 116A is mounted on the HMD 110A. Therefore, the tracking data indicates a position in the uvw visual field coordinate system set in the HMD 110A.

  FIG. 12 is a diagram for explaining the operation of the tracking module 226A. In one aspect, tracking module 226A tracks bone movement of user 190A's hand based on depth information input from camera 116A. In the example shown in FIG. 12, the tracking module 226A detects the positions of the joints a, b, c..., X of the hand of the user 190A and the position of the feature point y of the bone constituting the arm. . As an example, the feature point y may be the tip of the rib on the side close to the hand. As another example, the feature point y can be an elbow.

  The tracking module 226A is configured to be able to recognize the shape of the hand (finger movement) of the user 190A based on the positional relationship between the joints a to x. For example, the tracking module 226 may indicate that the hand of the user 190A is pointing to a finger, that the hand is open, that the hand is closed, that the hand is pinching something, and that the hand is twisted. And that the hand is in the shape of a handshake. Further, the tracking module 226A can determine whether the recognized hand is the left hand or the right hand based on the positional relationship between the joints a to d and the other joints. Further, the tracking module 226A can accurately track the direction in which the arm of the user 190A extends by specifying the linear direction extending from the joint j to the feature point y. Such a camera 116A and a tracking module 226A can be realized by, for example, a LeapMotion (registered trademark) provided by LeapMotion.

  FIG. 13 shows an example of the data structure of tracking data. The tracking module 226A acquires tracking data for each of the joints a to x and the feature point y. These tracking data represent position information in the uvw visual field coordinate system set in the HMD 110A. The computer 200A stores the acquired tracking data in the object information 242.

  FIG. 14 shows a view field image 1400 visually recognized by the user 190A. The view image 1400 corresponds to a part of the virtual space 2A. An avatar object corresponding to the user 190A is arranged in the virtual space 2A. The virtual camera 1A that captures the view image of the user 190A is arranged at the position of the avatar object (for example, the position of the eyes). In other words, the avatar object is arranged in the viewpoint of the user 190A (the position of the virtual camera 1A) in the virtual space 2A. Thereby, the virtual camera 1A and the avatar object are arranged at the same position in principle. Therefore, when the camera 116A captures the hand and arm of the user 190A, the field-of-view image 1400 shown in FIG. 14 includes the hand and arm of the avatar object. More specifically, the field-of-view image 1400 includes a right hand 1410, a right arm 1420, a left hand 1430, and a left arm 1440.

  The tracking module 226A generates tracking data representing the movement of the hand and arm in the real space of the user 190A at a predetermined interval (for example, 60 times per second). The avatar control module 234A moves the hand and arm of the avatar object based on the tracking data. Therefore, for example, when the user 190A moves the right hand and the right arm in the vertical direction in the real space, the right hand 1410 and the right arm 1420 arranged in the virtual space 2A also move in the vertical direction (Y direction).

[Display UI]
FIG. 15 is a diagram for explaining a user interface display method. A view field image 1500 shown in FIG. 15 corresponds to a part of the virtual space 2A. The view image 1500 is different from the view image 1400 shown in FIG. 14 in that it further includes a screen object 1510 and a UI (user interface) 1520.

  The screen object 1510 is arranged in the virtual space 2A by the photographing module 235A. The screen object 1510 is a rectangular object as an example, and has a front surface and a back surface. The screen object 1510 has a photographing function, and the front surface functions as a preview screen. In one aspect, the normal direction of the front surface is set as the shooting direction of the screen object 1510. In another aspect, the screen object 1510 may be configured to display a preview screen on both sides.

  Note that the screen object 1510 only needs to have a shooting function and a preview screen, and may not have the shape shown in FIG. For example, the screen object 1510 may be the shape of a camera (a shape including a lens, a shutter button, etc.) or the shape of a portable information processing terminal (smart phone, tablet).

  The imaging module 235A sets the screen object 1510 to the in-camera mode or the out-camera mode. The screen object 1510 in the in-camera mode photographs its front surface direction. On the other hand, the screen object 1510 in the out-camera mode shoots its back direction.

  In the example shown in FIG. 15, the screen object 1510 is set to the in-camera mode, and the avatar object 1000A corresponding to the user 190A is displayed on the preview screen. In the example shown in FIG. 15, the avatar object includes the entire body. However, in another aspect, the avatar object may be configured by only a part of the body. For example, the processor 10A may be configured to place an arm object (which also functions as an operation object) as an avatar object in the virtual space 2A.

  In one aspect, the UI 1520 relates to shooting by the screen object 1510. The user 190 </ b> A operates shooting using the screen object 1510 using the UI 1520. For example, the user 190A operates the UI 1520 to set settings such as movement of the screen object 1510, zoom magnification, filters (eg, polarization filter, ND filter), shooting timing, shooting mode (still image shooting / moving image shooting), and the like. Do.

  The user 190A operates the UI 1520 using the right hand 1410 or the left hand 1430 of the avatar object 1000A as an operation object.

  In one aspect, the virtual space control module 230A displays the UI 1520 on either the right arm 1420 or the left arm 1440 of the avatar object 1000A. As an example, the user 190A registers his / her dominant hand in the computer 200A in advance. The virtual space control module 230A displays the UI 1520 on the arm that is not the dominant hand. Thus, the user 190A can comfortably operate the UI 1520 by operating the operation object corresponding to the dominant hand.

  In another aspect, there may be a case where only one of the arms of the user 190A exists in the sensing range of the camera 116A (the space 1100 in FIG. 11). In such a case, the virtual space control module 230A displays the UI 1520 on the arm of the avatar object 1000A corresponding to this one arm. The user 190A looks at the UI 1520 and moves the other arm to the sensing range of the camera 116A in order to operate the UI 1520.

  By the way, it is also conceivable that the user interface is arranged in any one of the virtual spaces 2A like a UI 2110 described later with reference to FIG. However, when the virtual camera 1 is away from the UI 2110, the user 190A may not be able to clearly see the UI 2110. In addition, the user 190 </ b> A needs to operate the virtual camera 1 after approaching the UI 2110. For these reasons, the user 190A may not be able to operate the UI 2110 comfortably.

  A UI 1520 according to an embodiment may solve these problems. The avatar object 1000A is arranged in the virtual camera 1A (the user 190A's view in the virtual space 2A). Therefore, regardless of where the virtual camera 1A is arranged in the virtual space 2A, the user 190A can easily view and operate the UI 1520 simply by looking at the arm (sensing the arm with the camera 116A). Therefore, the user 190A can comfortably operate the UI 1520.

  In addition, the UI 1520 is not constantly displayed on the monitor 112A, but is displayed at a desired timing of the user 190A (timing when the user's 190A arm is photographed by the camera 116). Therefore, the computer 200A can prevent the UI 1520 from blocking the field of view of the user 190A in the virtual space 2A.

[Tactile feedback]
Next, with reference to FIG. 16 and FIG. 17, it will be described that feedback of the user 190A operating the UI 1520 can be obtained as a tactile sensation.

  FIG. 16 shows a view field image 1600 when the user 190 </ b> A is operating the UI 1520. FIG. 17 is a diagram for explaining the operation of the user 190A in the real space corresponding to FIG.

  Referring to FIG. 16, right hand 1410 of avatar object 1000 </ b> A operates UI 1520 displayed on left arm 1440. As described with reference to FIGS. 11 to 14, the movement of the hand and arm of the avatar object 1000 </ b> A in the virtual space 2 </ b> A reflects the movement of the hand and arm of 190 </ b> A in the real space. Therefore, as shown in FIG. 17, the user 190 </ b> A touches the left arm 1040 with the right hand 1010 when performing the operation of FIG. 16.

  That is, the user 190 </ b> A can obtain feedback that the UI 1520 has been operated by the tactile sensation of the right hand 1010 and the left arm 1040. Thereby, the user 190A can grasp that the UI 1520 has been operated reliably.

  A UI 2110 shown in FIG. 21 will be described as a comparison of the above processing. The UI 2110 is arranged in the hollow of the virtual space 2A. In operating the UI 2110, the user 190A moves the right hand 1010 or the left hand 1030 in a hollow state. Therefore, the user 190 </ b> A cannot obtain tactile feedback when operating the UI 2110. In such a case, the user 190 </ b> A may not be sure that his / her operation has been input to the computer 200 </ b> A, and may feel anxiety. The computer 200A according to the embodiment can prevent such a situation by displaying the UI 1520 on the arm of the avatar object 1000A.

  Note that the computer 200A may temporarily change the color of the UI 1520 or output a notification sound from the speaker 118A when the UI 1520 is operated. Thus, the user 190A can obtain visual or auditory feedback when operating the UI 1520.

  In the above example, the UI 1520 is displayed on the arm of the avatar object 1000A, but may be displayed on a part of the body of the avatar object 1000A. For example, the UI 1520 may be displayed on the palm, back of the hand, or foot of the avatar object.

[Process to display the user interface]
FIG. 18 is a flowchart showing processing for displaying a user interface on an avatar object. The processing shown in FIG. 18 is realized by the processor 10A executing a control program stored in the storage 12.

  In step S1810, the processor 10A defines the virtual space 2A. In step S1815, the processor 10A places the avatar object 1000A corresponding to the user 190A in the virtual space 2A. More specifically, the processor 10A arranges the avatar object 1000A in the viewpoint (position of the virtual camera 1A) in the virtual space 2A of the user 190A. The hand of the avatar object 1000A functions as an operation object.

  In step S1820, the processor 10A places the screen object 1510 in the virtual space 2A. In one aspect, the processor 10A executes a process of arranging the screen object 1510 in the virtual space 2A and a process of deleting from the virtual space 2A according to the operation of the user 190A.

  In step S1825, the processor 10A detects the movement of the hand and arm of the user 190A based on the output of the camera 116A as the tracking module 226A.

  In step S1830, processor 10A moves the hand and arm of avatar object 1000A based on the detection result. Thereby, the hand and arm of the avatar object 1000A are interlocked with the hand and arm of the user 190A in the real space.

  In step S1835, processor 10A determines whether or not the hand of avatar object 1000A holds screen object 1510. In one aspect, when the right hand 1410 or the left hand 1430 comes into contact with the screen object 1510, the processor 10A determines that the touched hand holds the object. In another aspect, processor 10A determines that this hand has held the object in response to the closed shape of the hand in contact with screen object 1510.

  If processor 10A determines that the hand has held screen object 1510 (YES in step S1835), processing proceeds to step S1840. Otherwise (NO in step S1835), processor 10A returns the process to step S1835.

  In step S1840, the processor 10A displays the UI 1520 on the arm of the avatar object 1000A. As an example, the processor 10A displays the UI 1520 on the arm on the side holding the screen object 1510.

  In step S1842, the processor 10A links the screen object 1510 to the held hand. In this step, the user 190A sets the screen object 1510 functioning as a camera at an arbitrary position and inclination in the virtual space 2A. Thereby, the user 190A determines the composition of shooting.

  In step S1843, the processor 10A determines whether or not the hand of the avatar object 1000A has touched the UI 1520. More specifically, the processor 10A determines whether or not the hand has touched the shooting button included in the UI 1520.

  If processor 10A determines that the hand has touched UI 1520 (YES in step S1843), processing proceeds to step S1860. Otherwise (NO in step S1843), processor 10A advances the process to step S1845.

  In step S1845, processor 10A determines whether or not the holding state of screen object 1510 is released. In one aspect, the processor 10A determines that the holding state of the screen object 1510 has been released in response to the open shape of the hand in contact with the screen object 1510. In another aspect, processor 10A determines that holding state of screen object 1510 has been released in response to the hand of avatar object 1000A coming in contact with the release button included in UI 1520.

  If processor 10A determines that the holding state of screen object 1510 has been released (YES in step S1845), processing proceeds to step S1850. Otherwise (NO in step S1845), processor 10A returns the process to step S1842.

  Note that in another aspect, the processor 10A may return the process to S1835 after deleting the UI 1520 displayed on the arm in response to the release of the holding state of the screen object 1510. According to this configuration, the user 190A can visually recognize the UI 1520 only while holding the screen object 1510 on the virtual space 2A.

  In step S1850, processor 10A fixes screen object 1510 at the position where the held state is released. If the holding state of the screen object 1510 is released in the hollow of the virtual space 2A, the screen object 1510 is fixed in a hollow state without being affected by gravity. According to this configuration, the computer 200A can increase the degree of freedom of the composition of shooting. Further, since the avatar object 1000A does not hold the screen object 1510, the user 190A can take a desired posture.

  The computer 200A arranges the screen object 1510 at the position designated by the user 190A by executing the processes of steps S1842, S1845, and S1850.

  In step S1855, the processor 10A determines whether or not the hand of the avatar object 1000A has touched the UI 1520. More specifically, the processor 10A determines whether or not the hand has touched the shooting button of the UI 1520.

  If processor 10A determines that the hand has touched UI 1520 (YES in step S1855), processing proceeds to step S1860. Otherwise (NO in step S1855), processor 10A returns the process to step S1855.

  In step S1860, the processor 10A executes shooting by the screen object 1510 after a predetermined time (for example, 3 seconds) has elapsed since the hand touched the UI 1520 (the shooting button). The processor 10A further saves an image generated by shooting in the storage 12A.

  The processor 10A stores the captured image generated by the imaging module 235A in the captured image DB 244A. At that time, the processor 10A stores the captured image and the predetermined information in association with each other. The predetermined information includes a panoramic image ID for identifying the panoramic image 22 developed in the virtual space 2A at the time of shooting, and a shooting image position where the shooting direction of the screen object 1510 and the celestial sphere of the virtual space 2A intersect.

  Further, when the panorama image 22 is a moving image, the predetermined information includes a shooting timing at which shooting was performed during the reproduction of the panorama image 22. In one aspect, the processor 10A transmits information stored in the captured image DB 244 to the server 150 together with the user ID of the user 190A. The server 150 stores the received information in the captured image DB 939.

[Photographed when the user is stationary]
In the above example, the processor 10A shoots when the UI 1520 (the shooting button) and the hand of the avatar object 1000A come into contact with each other (steps S1843 and S1855), but the shooting trigger is not limited to this. For example, the processor 10A is configured to take a picture when detecting that the movement of the user 190A has stopped for a predetermined time.

  In one aspect, the processor 10A determines that the movement of the user 190A is stopped when the movement of the hand and / or head of the user 190A is stopped. The processor 10A determines that the hand of the user 190A is stopped when the displacement amount per unit time of the tracking data is less than a predetermined displacement amount. Further, the processor 10A indicates that the head of the user 190A is stopped when the amount of displacement per unit time of the HMD 110A (the position of one of the plurality of light sources provided therein) is less than a predetermined amount of displacement. to decide.

  Based on the above, the user 190 </ b> A can perform shooting in a desired posture without bothering to operate the UI 1520.

[Selfie stick]
FIG. 19 shows another display mode of the screen object 1510. A view field image 1900 shown in FIG. 19 corresponds to a part of the virtual space 2A. In one aspect, the processor 10A places a support member 1910 that supports the object together with the screen object 1510 in the virtual space 2A.

  When the screen object 1510 is arranged alone in the virtual space 2A, the user 190A may not understand at a glance that the object functions as a camera. Therefore, the computer 200A according to an embodiment arranges (displays) the screen object 1510 together with the support member 1910, thereby increasing the possibility that the user 190A recognizes the shooting function of the screen object 1510.

  Self-taking sticks (also referred to as selfie sticks or selka sticks) that support smartphones (or devices having photographing functions) are widely recognized in the world. Therefore, when the screen object 1510 having a screen (preview screen) and the rod-shaped support member 1910 are presented together, the possibility that the user recognizes the shooting function is increased.

  Support member 1910 is configured to be held by avatar object 1000A. In the example shown in FIG. 19, the right hand 1410 of the avatar object 1000 </ b> A holds the support member 1910. The processor 10 </ b> A moves the screen object 1510 and the support member 1910 in conjunction with the right hand 1410 holding the support member 1910.

  In one aspect, the processor 10A displays a UI 1520 on the arm of the avatar object 1000A on the side holding the support member 1910. Therefore, a UI 1520 is displayed on the right arm 1420 in FIG. The determination as to whether or not the hand of the avatar object 1000A is holding the support member 1910 is executed by the same process as in step S1835.

  In the state of FIG. 19, the user 190A operates the UI 1520 by operating the left hand 1430 of the avatar object 1000A. In one aspect, the UI 1520 includes a button that can adjust the length of the support member 1910. The user 190A moves the hand holding the support member 1910 to an arbitrary position and angle, and further adjusts the length of the support member 1910 to determine the composition of the shooting. According to this configuration, the computer 200A can increase the degree of freedom of the composition of shooting.

  In one aspect, the processor 10A may be configured to display the UI 1520 on the arm only while the support member 1910 is held by the hand of the avatar object 1000A.

  FIG. 20 is a diagram for explaining processing for not displaying the UI 1520. Referring to FIG. 20, view image 2000 of user 190 </ b> A does not include screen object 1510. In such a case, the processor 10A is configured not to execute the process of displaying the UI 1520 on the arm of the avatar object 1000A. More specifically, the processor 10A does not execute the process of displaying the UI 1520 when the screen object 1510 is not arranged in the visual recognition area 23 determined from the position and inclination of the virtual camera 1A.

  In another aspect, the processor 10A is configured not to execute the process of displaying the UI 1520 when the screen object 1510 is not arranged in the virtual space 2A. As described in step S1820 in FIG. 18, the screen object 1510 may or may not be arranged in the virtual space 2A. Only when the screen object 1510 is arranged in the virtual space 2A, the processor 10A executes a process of displaying a UI 1520 for controlling the screen object 1510.

  Note that the user interface displayed on the avatar object 1000A is not limited to the control of the screen object 1510. For example, the content of the UI 2110 (menu) shown in FIG. 21 may be displayed on a part of the body of the avatar object 1000A.

[Operation by closing eyes]
FIG. 21 is a diagram for explaining an operation based on the line of sight of the user 190A. In one aspect, the view field image 2100 visually recognized by the user 190A includes a UI 2110. The UI 2110 includes a tutorial button 2120, a setting button 2130, a return button 2140, and an end button 2150.

  The UI 2110 is a user interface including a plurality of layers of content. When the processor 10A receives the selection of the return button 2140, the processor 10A displays the content of the previous layer. Further, when the processor 10A receives selection of the end button 2150, the processor 21A deletes the UI 2110 from the virtual space 2A.

  The user 190A selects these buttons with the hand of the avatar object 1000A or with a line of sight. More specifically, when the gazing point of the user 190A detected by the gazing sensor 140A is directed to any of these buttons for a predetermined time, the processor 10A determines that the button has been selected. .

  However, the user 190A may feel uncomfortable with these operations. Specifically, the user 190 </ b> A may feel fatigued when operating his / her arm hollow. In addition, the user 190A may find it troublesome to perform an operation with a gazing point.

  Therefore, the processor 10A according to an embodiment accepts an input of a negative operation by the user 190A in response to the time when the eyes of the user 190A are closed exceeding a predetermined time.

  The processor 10A detects that the eyes of the user 190A are closed based on the output of the gaze sensor 140A. As an example, the gaze sensor 140A detects reflected light of near-infrared light emitted to the eyes of the user 190A, and the processor 10A detects that the eyes of the user 190A are closed based on the detection result.

  The negative operation includes selection of a return button 2140 or an end button 2150 as an example. As another example, the negative operation includes an operation of deleting an object arranged in the virtual space 2A, an operation of changing the panoramic image 22 developed in the virtual space 2A, and the like. As yet another example, the negative operation includes an operation of deleting the UI 1520 for operating the screen object 1510 from the arm of the avatar object 1000A. In general, people tend to turn away (close) their eyes because of disgust. Therefore, the user can intuitively understand that a negative operation is input to the computer 200A by closing the eyes. Based on the above, the user 190A can easily operate the UI 2110.

[Display mode of other users' screen objects]
The series of processes are processes that can be executed independently by the computer 200A. In one aspect, a plurality of users communicate with each other using each avatar object in the virtual space. At that time, the user wants to take an image including his avatar object and another user's avatar object. However, other users may have difficulty in recognizing that the user's camera (screen object) is in the in-camera mode (self-shooting mode). In addition, other users may feel uncomfortable when their field of view is blocked by the user's camera. Therefore, a display mode of a camera (screen object) when a plurality of users communicate in a virtual space will be described below.

  FIG. 22 schematically illustrates a situation where each of the plurality of HMDs 110 provides a virtual space to each of the plurality of users 190 in the network 19. Hereinafter, a user of the HMD set 105B is represented as a user 190B, a user of the HMD set 105C is represented as a user 190C, and a user of the HMD set 105D is represented as a user 190D. Further, B is added to the reference symbol of each component relating to the HMD set 105B, C is added to the reference symbol of each component relating to the HMD set 105C, and D is added to the reference symbol of each component relating to the HMD set 105D. .

  Referring to FIG. 22, each of computers 200 </ b> A to 200 </ b> D provides virtual spaces 2 </ b> A to 2 </ b> D to users 190 </ b> A to 190 </ b> D via HMDs 110 </ b> A to 110 </ b> D, respectively. In the example shown in FIG. 22, the computer 200A (user 190A) and the computer 200B (user 190B) share the same virtual space. Therefore, the content expanded in the virtual space 2A and the content expanded in the virtual space 2B are the same. For example, the panoramic image 22A expanded in the virtual space 2A and the panoramic image 22B expanded in the virtual space 2B are the same image. In the virtual space 2A and the virtual space 2B, an avatar object 1000A of the user 190A and an avatar object 1000B of the user 190B exist. It should be noted that the avatar object 1000A in the virtual space 2A and the avatar object 1000B in the virtual space 2B are each equipped with an HMD, but this is for ease of explanation. Not installed.

  In one aspect, computers 200A and 200B transmit and receive motion detection data and tracking data to each other. Thereby, the computer 200A can link the operation of the avatar object 1000B arranged in the virtual space 2A with the operation of the user 190B. That is, the user 190A can recognize the operation of the user 190B via the avatar object 1000B. In addition, the computer 200B can link the operation of the avatar object 1000A arranged in the virtual space 2B with the operation of the user 190A.

(Display mode of screen object according to mode)
FIG. 23 shows another user's screen object in the out-camera mode. FIG. 24 shows another user's screen object in the in-camera mode.

  Referring to FIG. 23, the view field image 2300 corresponds to a part of the virtual space 2A. The view image 2300 includes an avatar object 1000B corresponding to the user 190B, a screen object 2310, and a tree object 2320. Screen object 2310 is operated by user 190B. In this way, cameras (screen objects) corresponding to each of a plurality of users can be arranged in the virtual space.

  In the example of FIG. 23, the screen object 2310 is set to the out camera mode. Therefore, on the preview screen of the screen object 2310, a tree object 2320 that exists in the space in the reverse direction of the screen object 2310 is displayed.

  Since the user 190B sets the mode of the screen object 2310 by himself, the user 190B can understand whether the screen object 2310 is in the in-camera mode or the out-camera mode. On the other hand, it is difficult for the user 190A to determine at a glance whether the screen object 2310 is in the out-camera mode or the in-camera mode. This is because the user 190 </ b> A compares the preview screen of the screen object 2310 with those arranged before and after the screen object 2310 to determine which mode the screen object 2310 is set to. In yet another aspect, the avatar object is not imitating the appearance of the corresponding user, but is selected from any one of predetermined types. Therefore, when the type of the avatar object 1000A is the same as the type of the avatar object 1000B, it is difficult for the user 190A to determine whether or not the avatar object displayed on the preview screen is an avatar object operated by the user.

  Therefore, the computer 200A according to an embodiment changes the display mode of the screen object 2310 operated by another user according to the mode.

  Referring to FIG. 24, screen object 2310 included in view image 2400 is set to the in-camera mode. Therefore, on the preview screen of the screen object 2310, the avatar objects 1000A and 1000B existing in the space in the front surface direction of the screen object 2310 are displayed.

  In one aspect, when the screen object 2310 is set to the in-camera mode, the processor 10A displays the screen object 2310 in a manner supported by the support member 2415. The processor 10A further displays these objects in a manner in which the hand of the avatar object 1000B is holding the support member 2415.

  Based on the above, the user 190A can recognize at a glance that the screen object 2310 is set to the in-camera mode by visually recognizing that the screen object 2310 is supported by the support member 2415. As described above, a self-shooting stick that assists self-shooting with a camera set in the in-camera mode is widely recognized. Therefore, the user 190A can easily understand that the support member 2415 is recognized as a selfie stick and the screen object 2310 is set to the in-camera mode.

  Furthermore, as shown in FIG. 23, the screen object 2310 set to the out-camera mode is displayed in a mode that is not supported by the support member 2415. Therefore, the user 190A can easily understand that the screen object 2310 is set to the out-camera mode by visually recognizing that the screen object 2310 is displayed alone.

  FIG. 25 is a flowchart illustrating a process of changing the display mode of the screen object according to the mode. Note that the processing shown in FIG. 25 is executed by the processor 10A and the processor 10B. These processes are based on the assumption that the computer 200 </ b> A and the computer 200 </ b> B are communicating via the server 150.

  In step S2510, the processor 10A defines the virtual space 2A. In step S2515, the processor 10B defines the virtual space 2B.

  In step S2520, the processor 10B places a screen object 2310 (indicated as “other screen object” in FIG. 25) operated by the user 190B in the virtual space 2B. As an example, the screen object 2310 is set to the out camera mode. The processor 10B further transmits information on the screen object 2310 to the computer 200A. The information of the screen object 2310 includes, for example, modeling data, position information, tilt information, and a mode signal. The mode signal represents whether the screen object 2310 is in the in-camera mode or the out-camera mode.

  In step S2525, the processor 10A places the screen object 2310 in the virtual space 2A based on the information received from the computer 200B. At this time, the processor 10A refers to the mode signal and displays the screen object 2310 in a mode that is not supported by the support member 2415 (that is, the screen object 2310 alone).

  In step S2530, the processor 10B determines whether or not an input of an instruction to switch the mode of the screen object 2310 is received from the user 190B. In one aspect, processor 10B makes this determination based on an operation of a user interface displayed on a part of the body of avatar object 1000B.

  If processor 10B determines that an instruction has been input (YES in step S2530), it transmits a mode signal to computer 200A. Otherwise (NO in step S2530), processor 10B advances the process to step S2545.

  In step S2532, the processor 10A determines whether or not a mode signal has been received from the computer 200B. If processor 10A determines that a mode signal has been received (YES in step S2532), processing proceeds to step S2535. Otherwise (NO in step S2532), processor 10A advances the process to step S2540.

  In step S2535, the processor 10A displays the screen object 2310 in a manner corresponding to the received mode signal. In one aspect, when the mode signal represents the in-camera mode, the processor 10A displays the screen object 2310 in a manner supported by the support member 2415. Further, when the mode signal represents the out-camera mode, the processor 10A displays the screen object 2310 in a mode that is not supported by the support member 2415.

  In another aspect, processor 10A may be configured to change the color or transparency of screen object 2310 in accordance with the mode signal. In yet another aspect, the processor 10A may be configured to change the shape of the screen object 2310 in response to the mode signal.

  In step S2540, processor 10A updates the view image and outputs the updated view image to HMD 110A. In step S2545, the processor 10B updates the view image and outputs the updated view image to the HMD 110B.

  According to the above, the user 190A can determine at a glance the mode of the screen object 2310 operated by another user.

  For example, the user 190A sees the screen object 2310 set to the in-camera mode, and recognizes that the user 190B is going to take a self-portrait (shooting the avatar object 1000B) on the virtual space 2B. Thereby, the user 190A performs an operation of bringing the avatar object 1000A closer to the avatar object 1000B. As a result, the user 190B can acquire an image including the avatar objects 1000A and 1000B. The computer 200B transmits the generated image to the computer 200A. The users 190A and 190B can promote communication in the virtual space using the obtained image as a topic.

(Screen object display mode according to position)
As shown in FIGS. 23 and 24, in one aspect, the processor 10A is configured to place another screen object operated by another user in the virtual space 2A. In such a case, the user 190A may be prevented from viewing the panoramic image 22A by another screen object. Hereinafter, processing that can solve such a problem will be described.

  FIG. 26 is a diagram for explaining the arrangement relationship between the avatar object and the screen object in the virtual space 2A. Referring to FIG. 26, avatar object 1000B holds support member 2415 of screen object 2310.

  The computer 200A according to an embodiment controls the display mode of the screen object 2310 based on the relative relationship between the position of the screen object 2310 and the position of the avatar object 1000A.

  The virtual space 2A is divided into a space 2610 and a space 2620 by a plane 2600 on which a surface (for example, a front surface) of the screen object 2310 exists. In one aspect, the space 2610 is defined as the rear of the screen object 2310 and the space 2620 is defined as the front of the screen object 2310.

  In the example shown in FIG. 26, the avatar object 1000A is arranged behind the screen object 2310 (space 2610).

  FIG. 27 shows a view field image 2700 visually recognized by the user 190A in the state of FIG. Referring to FIG. 27, processor 10A according to an embodiment displays only a frame (for example, a frame at four corners) of screen object 2310 when avatar object 1000A is arranged behind screen object 2310. Therefore, the user 190A can visually recognize the panoramic image 22A and the avatar object 1000B arranged on the back side of the screen object 2310.

  According to this configuration, the computer 200 </ b> A can suppress the screen object 2310 from blocking the field of view of the user 190 </ b> A.

  In the example shown in FIG. 27, the support member 2415 is displayed. However, in another aspect, the support member 2415 may not be displayed.

  FIG. 28 is a diagram for explaining an arrangement relationship between an avatar object and a screen object in another aspect. Referring to FIG. 28, avatar object 1000A is disposed in front of screen object 2310 (space 2620). In such a case, the processor 10A displays a preview screen of the screen object 2310 as shown in FIG. Therefore, the user 190A can adjust the position of the avatar object 1000A in the angle of view while checking the preview screen of the screen object 2310.

  Based on the above, the computer 200A can provide an appropriate view image to the user 190A based on the relative relationship between the position of the screen object 2310 and the position of the avatar object 1000A. Therefore, the user 190A is less likely to feel discomfort with the screen object 2310.

  FIG. 29 is a flowchart showing a process of changing the display mode of another screen object according to the position of the avatar object. In addition, the same code | symbol is attached | subjected about the same process as the above-mentioned process among the processes shown by FIG. Therefore, the description about the process is not repeated.

  In step S2910, processor 10A arranges avatar object 1000A (indicated as “own avatar object” in FIG. 29) corresponding to user 190A in virtual space 2A. The processor 10A further transmits information of the avatar object 1000A (for example, data for modeling, position information, etc.) to the computer 200B.

  In step S2915, processor 10B arranges avatar object 1000A in virtual space 2B based on the received information about avatar object 1000A.

  In steps S2920 and S2925, as in steps S2910 and S2915, an avatar object 1000B (indicated as “other avatar object” in FIG. 29) is generated in the virtual spaces 2A and 2B.

  In step S2930, the processor 10B updates the position information of the screen object 2310 based on the motion detection data detected by the HMD sensor 120B or the tracking data generated by the tracking module 226B. The processor 10B further transmits the updated position information of the screen object 2310 to the computer 200A.

  In step S2935, the processor 10A updates the position information of the screen object 2310 arranged in the virtual space 2A based on the information received from the computer 200B.

  In step S2940, processor 10A updates the position information of avatar object 1000A. As an example, the processor 10A updates this information based on motion detection data detected by the HMD sensor 120.

  In step S2945, the processor 10A controls the display mode of the screen object 2310 based on the relative relationship between the position of the screen object 2310 and the position of the avatar object 1000A. As an example, when the avatar object 1000A is located in front of the screen object 2310, the processor 10A displays the screen object 2310 in a form including a preview screen. On the other hand, when the avatar object 1000A is located behind the screen object 2310, the processor 10A can display only the frame of the screen object 2310.

  In one aspect, the processor 10A may be configured to change the display mode (color, transparency) of the frame of the screen object 2310 depending on whether the avatar object 1000A is in front of or behind the screen object 2310. Good.

[Manage shot images]
FIG. 30 shows an example of the data structure of the photographed image DB 939 according to an embodiment. In the example illustrated in FIG. 30, the captured image DB 939 stores a captured image, a user ID, a panoramic image ID, a captured image position, and a capturing timing in association with each other. These data are information received from each computer 200.

  The user ID identifies the user 190 of the computer 200 that generated the captured image. The panorama image ID identifies the panorama image 22 developed in the virtual space 2 at the time of shooting. The photographed image position represents a position where the photographing direction of the automatic camera and the celestial sphere in the virtual space 2 intersect. In other words, the captured image position represents the center position of the panoramic image 22 included in the captured image. The shooting timing specifies the timing at which shooting is performed on the panoramic image 22 when the panoramic image 22 developed in the virtual space 2 is a moving image.

  When the captured image includes a plurality of avatar objects, the captured image DB 939 also holds a user ID corresponding to each of these avatar objects.

  The user 190 captures an object (for example, an avatar object of another person or a panoramic image 22) that has shown interest. Therefore, the administrator of the server 150 can grasp the object of interest of the user 190 based on the captured image DB 939.

  In one aspect, the processor 920 of the server 150 identifies an object (content) included in the photographed image by a known machine learning method, and stores the identification result in the photographed image DB 939. As an example, the processor 920 identifies a bounding box that surrounds an object included in the captured image using the Selective Search method. The processor 920 calculates a feature amount derived by a CNN (Convolutional Neural Network) from the image cut out by the bounding box. The processor 920 specifies an object included in the bounding box using the calculated feature amount and a discriminator according to a plurality of SVMs (Support Vector Machines).

  Based on the above, the administrator of the server 150 can easily understand the object (object) of the user 190's interest. Further, the processor 920 of the server 150 performs processing for distributing an advertisement that the user 190 may be interested in based on the identified object and recommending the panoramic image 22 that the user 190 will be interested in. It may be configured as follows.

[Constitution]
The technical features disclosed above can be summarized as follows.

  (Configuration 1) According to an embodiment, a method executed by the computer 200 to provide the virtual space 2 to the HMD 110 is provided. In this method, a step of defining the virtual space 2 (S1010), a step of displaying a view field image on the monitor 112 of the HMD 110 to provide the virtual space 2 to the user 190 of the HMD 110 (S1032), and a motion of the HMD 110 are detected. A step (S1034), a step (S1040) of updating a view field image in conjunction with the detected movement, a step (S1815) of placing an avatar object 1000 corresponding to the user 190 in the view of the user 190 in the virtual space 2; And a step of displaying a UI 1520 for accepting an operation from the user 190 on the avatar object 1000 (S1840). The avatar object 1000 can be composed of only a part of the body.

  (Configuration 2) The method according to (Configuration 1) includes a step (S1825) of detecting the movement of the first part (for example, the right hand) of the user 190, and an operation object (link) in conjunction with the detected movement of the first part. For example, the step of moving the right hand part of the avatar object (S1830), the step of detecting the operation of the second part (for example, the left arm) of the user 190 (S1825), and the action of the detected second part are linked. The step of moving the part corresponding to the second part of the avatar object 1000 (S1830) is further provided. The step of displaying the UI 1520 includes displaying the UI 1520 that can accept an operation by the operation object at a location corresponding to the second portion of the avatar object 1000.

  (Configuration 3) In (Configuration 2), the first portion includes one of the right hand and the left hand. The second portion includes at least one of the other, which is different from one of the right hand and the left hand, and the arm on the other side.

  (Configuration 4) A method according to any one of (Configuration 1) to (Configuration 3) includes a step (S1820) of arranging a screen object having a photographing function and capable of displaying a preview image in the virtual space 2, and a screen object. A step (S1860) of storing the photographed image in a memory. The UI 1520 relates to shooting using a screen object.

  (Configuration 5) In the method according to (Configuration 4), the step of detecting the operation of the first part of the user 190 (S1825) and the step of moving the operation object so as to be interlocked with the detected operation of the first part (S1830). And further comprising. The step of displaying the UI 1520 includes displaying the UI 1520 on the avatar object 1000 (S1840) in response to the operation object holding the screen object (YES in S1835).

  (Configuration 6) In (Configuration 4) or (Configuration 5), the step of arranging the screen object in the virtual space 2 includes arranging the screen object at a position designated by the user 190 (S1845, S1850).

  (Configuration 7) In (Configuration 6), the UI 1520 includes a shooting button. The step of performing photographing with the screen object includes performing photographing after a predetermined time has elapsed since the photographing button was selected (S1855, 1860).

  (Configuration 8) The method according to (Configuration 4) or (Configuration 5) further includes a step of arranging a support member 1910 that supports the screen object and can be held by the avatar object 1000 in the virtual space 2. The UI 1520 is configured to be able to adjust the length of the support member (FIG. 19).

  (Configuration 9) In any one of (Configuration 4) to (Configuration 8), the step of performing imaging with the screen object executes imaging when it is detected that the movement of the HMD 110 has stopped for a predetermined time. including.

  (Configuration 10) In the method according to any one of (Configuration 1) to (Configuration 9), a step of detecting whether or not the user 190 closes his eyes and a time during which the user 190 closes his eyes are predetermined. A step of accepting an input of a negative operation by the user 190 in response to exceeding the time.

  (Configuration 11) The method according to any one of (Configuration 4) to (Configuration 10) is executed by the computer 200A. The computer 200A displays the other screen object 2310 different from the screen object 1510, which is operated by the user 190B of the other computer 200B that can communicate with the computer 200A, in the virtual space 2A (S2510), and the other computer 200B. And (2530) receiving a mode signal indicating whether the other screen object 2310 is in the in-camera mode or the out-camera mode. In the step of displaying the other screen object 2310 in the virtual space 2A, when the mode signal represents the in-camera mode, the other screen object 2310 is displayed in the first mode, and the mode signal represents the out-camera mode. In addition, displaying another screen object 2310 in a second mode different from the first mode (S2535).

  (Configuration 12) In (Configuration 11), the first mode includes a mode in which another screen object 2310 is supported by a support member 2415 (FIG. 24). The second mode includes a mode in which the other screen object 2310 is not supported by the support member 2415.

  (Configuration 13) The method according to any one of (Configuration 4) to (Configuration 12) is executed by the computer 200A. The computer 200A is further operated by a user 190B of another computer 200B that can communicate with the computer, and further includes a step of displaying another screen object 2310 capable of displaying a preview image in the virtual space 2A. The step of displaying another screen object in the virtual space 2 includes controlling the display mode of the screen object 2310 based on the relative relationship between the position of the other screen object 2310 and the position of the avatar object 1000A (S2945). .

  (Configuration 14) In (Configuration 13), the step of displaying another screen object 2310 in the virtual space 2A displays only the frame of the screen object 2310 when the avatar object 1000A is positioned behind the other screen object 2310. (FIG. 27).

  It should be thought that embodiment disclosed this time is an illustration and restrictive at no points. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  1 virtual camera, 2 virtual space, 5 reference line of sight, 10,920 processor, 11 memory, 12,930 storage, 13 input / output interface, 14,910 communication interface, 22 panoramic image, 23 viewing area, 26, 1400, 1500, 1600, 1900, 2000, 2100, 2300, 2400, 2700 View image, 100 HMD system, 105 HMD set, 112 monitor, 114 sensor, 116 camera, 118 speaker, 119 microphone, 120 HMD sensor, 140 gaze sensor, 150 server, 190 user, 200 computer, 220 display control module, 221 virtual camera control module, 222 view area determination module, 223 view image generation module, 224 reference line of sight Constant module, 225 motion detection module, 226 tracking module, 230 virtual space control module, 231 virtual space definition module, 232 virtual object generation module, 233 operation object control module, 234 avatar control module, 235 shooting module, 240 memory module, 241 Spatial information, 242 Object information, 243, 938 User information, 250 Communication control module, 922 Transmission / reception unit, 924 Server processing unit, 926 Matching unit, 932 Virtual space designation information, 934 Object designation information, 935 Avatar object information, 936 Position information 937 Tilt information, 1000 Avatar object, 1010, 1410 Right hand, 1030, 1430 Left hand, 1040, 1440 Left arm, 1100, 2610, 2620 Space, 1420 Right arm, 1510, 2310 Screen object, 1910, 2415 Support member, 2120 Tutorial button, 2130 Setting button, 2140 button, 2150 End button.

Claims (16)

A method performed by a computer to provide a virtual space for a head mounted device comprising:
Defining a virtual space;
Providing a virtual space to a user of the head mounted device by displaying a field of view image on a non-transmissive display of the head mounted device;
Detecting the movement of the head mounted device;
Updating the view image in conjunction with the detected movement;
Placing an avatar object corresponding to the user in the user's perspective in the virtual space;
For example Bei and displaying a user interface for receiving an operation from the user to the avatar object,
The virtual space includes a controlled object whose operation is controlled by the user interface,
In conjunction with the movement of the head mounted device, the field of view from the user's viewpoint in the virtual space is controlled,
The view image is an image corresponding to the view,
The method, wherein the user interface cannot be displayed on the avatar object when the view does not include the control target object, and can be displayed on the avatar object when the view includes the control target object . Detecting the movement of the first part of the user;
Moving an operation object to interlock with the detected movement of the first portion;
Detecting the movement of the second part of the user;
Moving a position corresponding to the second part of the avatar object to interlock with the detected movement of the second part,
The method according to claim 1, wherein displaying the user interface includes displaying the user interface capable of accepting an operation by the operation object at a location corresponding to the second portion of the avatar object. The first portion includes one of a right hand and a left hand,
The method according to claim 2, wherein the second portion includes at least one of the other of the right hand and the left hand different from the one and the arm on the other side. Detecting whether the user has closed eyes;
4. The method according to claim 1, further comprising a step of receiving an input of a negative operation by the user in response to a time when the user closes his eyes exceeds a predetermined time. The method described in 1. The control target object is a screen object having a photographing function and capable of displaying a preview image,
Placing said screen object in the virtual space,
Storing an image photographed by the screen object in a memory;
The method according to claim 1, wherein the user interface relates to shooting by the screen object. A method performed by a computer to provide a virtual space for a head mounted device comprising:
Defining a virtual space;
Providing a virtual space to a user of the head mounted device by displaying a field of view image on a non-transmissive display of the head mounted device;
Detecting the movement of the head mounted device;
Updating the view image in conjunction with the detected movement;
Placing an avatar object corresponding to the user in the user's perspective in the virtual space;
Displaying a user interface for accepting an operation from the user on the avatar object;
Arranging a screen object having a photographing function and capable of displaying a preview image in the virtual space;
Storing an image photographed by the screen object in a memory,
The user interface relates to shooting by the screen object,
Detecting the movement of the first part of the user;
Further comprising a step of moving an operation object to interlock with the detected movement of the first part,
Step of displaying the user interface includes the operation object in response to the holding of the screen object, and displays the user interface on the avatar object, Methods.   The method according to claim 5 or 6, wherein the step of arranging a screen object in the virtual space includes arranging the screen object at a position specified by the user. The user interface includes a shooting button,
The method according to claim 7, wherein the step of performing photographing with the screen object includes performing photographing after a predetermined time elapses after the photographing button is selected. A method performed by a computer to provide a virtual space for a head mounted device comprising:
Defining a virtual space;
Providing a virtual space to a user of the head mounted device by displaying a field of view image on a non-transmissive display of the head mounted device;
Detecting the movement of the head mounted device;
Updating the view image in conjunction with the detected movement;
Placing an avatar object corresponding to the user in the user's perspective in the virtual space;
Displaying a user interface for accepting an operation from the user on the avatar object;
Arranging a screen object having a photographing function and capable of displaying a preview image in the virtual space;
Storing an image photographed by the screen object in a memory,
The user interface relates to shooting by the screen object,
A step of supporting the screen object and arranging a support member that can be held by the avatar object in the virtual space;
Wherein the user interface is adjustably configured the length of the support member, Methods. A method performed by a computer to provide a virtual space for a head mounted device comprising:
Defining a virtual space;
Providing a virtual space to a user of the head mounted device by displaying a field of view image on a non-transmissive display of the head mounted device;
Detecting the movement of the head mounted device;
Updating the view image in conjunction with the detected movement;
Placing an avatar object corresponding to the user in the user's perspective in the virtual space;
Displaying a user interface for accepting an operation from the user on the avatar object;
Arranging a screen object having a photographing function and capable of displaying a preview image in the virtual space;
Storing an image photographed by the screen object in a memory,
The user interface relates to shooting by the screen object,
Performing the imaging by the screen object includes the motion of the head-mounted device performs the shooting when it is detected that has been stopped for a predetermined time, Methods. A method performed by a computer to provide a virtual space for a head mounted device comprising:
Defining a virtual space;
Providing a virtual space to a user of the head mounted device by displaying a field of view image on a non-transmissive display of the head mounted device;
Detecting the movement of the head mounted device;
Updating the view image in conjunction with the detected movement;
Placing an avatar object corresponding to the user in the user's perspective in the virtual space;
Displaying a user interface for accepting an operation from the user on the avatar object;
Arranging a screen object having a photographing function and capable of displaying a preview image in the virtual space;
Storing an image photographed by the screen object in a memory,
The user interface relates to shooting by the screen object,
Displaying in the virtual space another screen object different from the screen object operated by a user of another computer that can communicate with the computer;
Receiving from the other computer a mode signal indicating whether the other screen object is in an in-camera mode or an out-camera mode;
Displaying the other screen object in the virtual space comprises:
When the mode signal represents an in-camera mode, displaying the other screen object in a first manner;
When the mode signal represents the out-camera mode, and displaying in a different second aspect and the first aspect of the other screen objects, Methods. The first aspect includes an aspect in which the other screen object is supported by a support member,
The method according to claim 11, wherein the second aspect includes an aspect in which the other screen object is not supported by the support member.
A method performed by a computer to provide a virtual space for a head mounted device comprising:
Defining a virtual space;
Providing a virtual space to a user of the head mounted device by displaying a field of view image on a non-transmissive display of the head mounted device;
Detecting the movement of the head mounted device;
Updating the view image in conjunction with the detected movement;
Placing an avatar object corresponding to the user in the user's perspective in the virtual space;
Displaying a user interface for accepting an operation from the user on the avatar object;
Arranging a screen object having a photographing function and capable of displaying a preview image in the virtual space;
Storing an image photographed by the screen object in a memory,
The user interface relates to shooting by the screen object,
Further comprising the step of displaying, in the virtual space, another screen object that is operated by a user of another computer that can communicate with the computer and can display a preview image;
Step of displaying the other screen objects in the virtual space may include controlling the display mode of the screen object based the position of the other screen objects, the relative relationship between the position of the avatar object, square Law.   The step of displaying the other screen object in the virtual space includes displaying only a frame of the screen object when the avatar object is located behind the other screen object. the method of.   The program for making a computer implement | achieve the method of any one of Claims 1-14. A memory storing the program according to claim 15;
An information processing apparatus comprising: a processor for executing the program.

Images