JP6262283B2 - Method, program, and recording medium for providing virtual space - Google Patents

Description

  The present disclosure relates to a method, a program, and a recording medium that provide a virtual space.

  Patent Document 1 discloses a technique for allowing a user to visually recognize content played back in a virtual space through a head-mounted display.

JP2013-258614A (released on December 26, 2013)

  In the above-described conventional technique, when the user moves the HMD, the user's visual recognition location in the virtual space can be changed, so that the user's immersion feeling in the virtual space can be enhanced. On the other hand, there is a need for a device for improving the operability in the virtual space so that the user can more easily visually recognize an arbitrary place in the virtual space while increasing the immersion space in the virtual space.

  This indication is made in order to solve the above-mentioned subject. The purpose is to further improve the operability in the virtual space.

  In order to solve the above problems, a method of providing a virtual space according to the present disclosure is a method of providing a virtual space to a user wearing a head mounted display (hereinafter, HMD), in the first mode, In the second mode, the visual image based on the roll direction of the virtual camera that is arranged in the virtual space and interlocked with the HMD is displayed on the HMD. In the second mode, at least the rotation direction is specified based on the user operation, and the operation is detected. While continuing to be performed, the HMD has a step of updating the visual field image while continuously rotating the virtual camera or the space coordinate system of the virtual space in the rotation direction.

  According to this indication, operativity in virtual space can be raised more.

It is a figure which shows the structure of a HMD system. It is a figure which shows the hardware constitutions of a control circuit part. It is a figure which illustrates the visual field coordinate system set to HMD. It is a figure explaining the outline | summary of the virtual space provided to a user. It is a figure which shows the cross section of a visual field area | region. It is a figure which illustrates the method of determining a user's gaze direction. It is a block diagram which shows the functional structure of a control circuit part. It is a sequence diagram which shows the flow of the process in which an HMD system provides a virtual space to a user. It is a sequence diagram which shows the flow of a process which the HMD system reproduces | regenerates the moving image content which the user selected through the platform in virtual space in virtual space. It is a figure explaining the selection and reproduction | regeneration of the moving image content through the platform in virtual space. It is a sequence diagram which shows the flow of a process when a HMD system transfers to rotation mode. It is a sequence diagram which shows the flow of a process when a control circuit part starts rotation of a virtual camera in rotation mode. It is a figure which shows the virtual space and visual field image after a virtual camera rotates. It is a sequence diagram showing a flow of processing when the control circuit unit further rotates the virtual camera or stops the rotation of the virtual camera in the rotation mode. It is a figure which shows the virtual space and visual field image after a virtual camera further rotates. It is a figure which shows the virtual space and visual field image after rotation of a virtual camera stopped.

[Details of the embodiment of the present invention]
A specific example of a method and program for providing a virtual space according to an embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to these exemplifications, but is defined by the scope of claims for patent, and it is intended that all modifications within the meaning and scope equivalent to the scope of claims for patent are included in the present invention. In the following description, the same reference numerals are given to the same elements in the description of the drawings, and repeated description is not repeated.

(Configuration of HMD system 100)
FIG. 1 is a diagram showing a configuration of the HMD system 100. As shown in this figure, the HMD system 100 includes an HMD 110, an HMD sensor 120, a control circuit unit 200, and a controller 300.

  The HMD 110 is worn on the user's head. The HMD 110 includes a display 112 which is a non-transmissive display device, a sensor 114, and a gaze sensor 130. The HMD 110 displays a right-eye image and a left-eye image on the display 112, thereby allowing the user to visually recognize a three-dimensional image that is stereoscopically viewed by the user based on the parallax between both eyes of the user. This provides a virtual space to the user. Since the display 112 is disposed in front of the user's eyes, the user can immerse in the virtual space through an image displayed on the display 112. The virtual space may include a background, various objects that can be operated by the user, menu images, and the like.

  The display 112 may include a right-eye sub-display that displays a right-eye image and a left-eye sub-display that displays a left-eye image. Alternatively, the display 112 may be a single display device that displays the right-eye image and the left-eye image on a common screen. As such a display device, for example, a display device that alternately displays a right-eye image and a left-eye image independently by switching a shutter so that the display image can be recognized only by one eye. .

(Hard structure of the control circuit unit 200)
FIG. 2 is a diagram illustrating a hardware configuration of the control circuit unit 200. The control circuit unit 200 is a computer for causing the HMD 110 to provide a virtual space. As shown in FIG. 2, the control circuit unit 200 includes a processor, a memory, a storage, an input / output interface, and a communication interface. These are connected to each other in the control circuit unit 200 through a bus as a data transmission path.

  The processor includes a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), and the like, and controls operations of the control circuit unit 200 and the HMD system 100 as a whole.

  The memory functions as main memory. The memory stores a program processed by the processor and control data (such as calculation parameters). The memory may include a ROM (Read Only Memory) or a RAM (Random Access Memory).

  The storage functions as auxiliary storage. The storage stores a program for controlling the operation of the entire HMD system 100, various simulation programs, a user authentication program, and various data (images, objects, etc.) for defining a virtual space. Furthermore, a database including a table for managing various data may be constructed in the storage. The storage can be configured to include a flash memory or an HDD (Hard Disc Drive).

  The input / output interface includes various wired connection terminals such as a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and various wireless connection terminals. These processing circuits are included. The input / output interface connects the HMD 110, various sensors including the HMD sensor 120, and the controller 300 to each other.

  The communication interface includes various wired connection terminals for communicating with an external device via the network NW, and various processing circuits for wireless connection. The communication interface is configured to conform to various communication standards and protocols for communicating via a LAN (Local Area Network) or the Internet.

  The control circuit unit 200 provides a virtual space to the user by loading a predetermined application program stored in the storage into a memory and executing it. When the program is executed, the memory and the storage store various programs for operating various objects arranged in the virtual space and displaying and controlling various menu images and the like.

  The control circuit unit 200 may or may not be mounted on the HMD 110. That is, the control circuit unit 200 may be another hardware independent of the HMD 110 (for example, a personal computer or a server device that can communicate with the HMD 110 through a network). The control circuit unit 200 may be a device in which one or a plurality of functions are implemented by cooperation of a plurality of hardware. Alternatively, only a part of all the functions of the control circuit unit 200 may be mounted on the HMD 110, and the remaining functions may be mounted on different hardware.

  A global coordinate system (reference coordinate system, xyz coordinate system) is set in advance for each element such as the HMD 110 constituting the HMD system 100. This global coordinate system has three reference directions (axes) parallel to the vertical direction, the horizontal direction orthogonal to the vertical direction, and the front-rear direction orthogonal to both the vertical direction and the horizontal direction in the real space. In the present embodiment, since the global coordinate system is a kind of viewpoint coordinate system, the horizontal direction, vertical direction (vertical direction), and front-rear direction in the global coordinate system are set as an x-axis, a y-axis, and a z-axis, respectively. Specifically, the x-axis of the global coordinate system is parallel to the horizontal direction of the real space, the y-axis is parallel to the vertical direction of the real space, and the z-axis is parallel to the front-rear direction of the real space.

  The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. With this function, the HMD sensor 120 detects the position and inclination of the HMD 110 in the real space. In order to realize this detection, the HMD 110 includes a plurality of light sources (not shown). Each light source is, for example, an LED that emits infrared rays. The HMD sensor 120 includes, for example, an infrared sensor. The HMD sensor 120 detects the detection point of the HMD 110 by detecting the infrared ray irradiated from the light source of the HMD 110 with the infrared sensor. Further, based on the detection value of the detection point of the HMD 110, the position and inclination of the HMD 110 in the real space according to the user's movement are detected. The HMD sensor 120 can determine the time change of the position and inclination of the HMD 110 based on the change with time of the detected value.

  The HMD sensor 120 may include an optical camera. In this case, the HMD sensor 120 detects the position and inclination of the HMD 110 based on the image information of the HMD 110 obtained by the optical camera.

  Instead of the HMD sensor 120, the HMD 110 may detect its position and inclination using the sensor 114. In this case, the sensor 114 may be an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, or a gyro sensor, for example. The HMD 110 uses at least one of these. When the sensor 114 is an angular velocity sensor, the sensor 114 detects the angular velocity around the three axes in the real space of the HMD 110 over time according to the movement of the HMD 110. The HMD 110 can determine the temporal change of the angle around the three axes of the HMD 110 based on the detected value of the angular velocity, and can detect the inclination of the HMD 110 based on the temporal change of the angle.

  When the HMD 110 detects the position and inclination of the HMD 110 based on the detection value of the sensor 114, the HMD sensor 120 is not necessary for the HMD system 100. Conversely, when the HMD sensor 120 arranged at a position away from the HMD 110 detects the position and inclination of the HMD 110, the sensor 114 is not necessary for the HMD 110.

  As described above, the global coordinate system is parallel to the real space coordinate system. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination around the three axes of the HMD 110 in the global coordinate system. The HMD sensor 120 sets the uvw visual field coordinate system to the HMD 110 based on the detected value of the inclination of the HMD sensor 120 in the global coordinate system. The uvw visual field coordinate system set in the HMD 110 corresponds to a viewpoint coordinate system when a user wearing the HMD 110 views an object.

(Uuv visual field coordinate system)
FIG. 3 is a diagram illustrating a uwv visual field coordinate system set in the HMD 110. The HMD sensor 120 detects the position and inclination of the HMD 110 in the global coordinate system when the HMD 110 is activated. Then, a three-dimensional uvw visual field coordinate system based on the detected tilt value is set in the HMD 110. As shown in FIG. 3, the HMD sensor 120 sets a three-dimensional uvw visual field coordinate system around the head of the user wearing the HMD 110 as the center (origin) in the HMD 110. Specifically, the horizontal direction, the vertical direction, and the front-rear direction (x axis, y axis, z axis) that define the global coordinate system are respectively set around each axis by an inclination around each axis of the HMD 110 in the global coordinate system. Three new directions obtained by tilting are set as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw visual field coordinate system in the HMD 110.

  As shown in FIG. 3, when the user wearing the HMD 110 stands upright and visually recognizes the front, the HMD sensor 120 sets the uvw visual field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), the vertical direction (y-axis), and the front-back direction (z-axis) of the global coordinate system are the same as the pitch direction (u-axis) and yaw direction (v of the uvw visual field coordinate system in the HMD 110. Axis) and the roll direction (w-axis).

  After setting the uvw visual field coordinate system in the HMD 110, the HMD sensor 120 can detect the inclination (the amount of change in inclination) of the HMD 110 in the currently set uvw visual field coordinate system in accordance with the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the currently set uvw visual field coordinate system as the inclination of the HMD 110. The pitch angle (θu) is an inclination angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) is an inclination angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) is an inclination angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

  Based on the detected value of the inclination of the HMD 110, the HMD sensor 120 newly sets the uvw visual field coordinate system in the HMD 110 after moving to the HMD 110. The relationship between the HMD 110 and the uvw visual field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system similarly change in conjunction with the change.

  The HMD sensor 120 determines the position of the HMD 110 in the real space relative to the HMD sensor 120 based on the infrared light intensity acquired by the infrared sensor and the relative positional relationship (distance between the detection points) between the plurality of detection points. You may specify as a position. The origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) may be determined based on the specified relative position. The HMD sensor 120 detects the inclination of the HMD 110 in the real space based on the relative positional relationship between the plurality of detection points, and further, based on the detected value, the uvw visual field coordinates of the HMD 110 in the real space (global coordinate system). The orientation of the system may be determined.

(Outline of virtual space 2)
FIG. 4 is a diagram illustrating an outline of the virtual space 2 provided to the user. As shown in this figure, the virtual space 2 has a spherical structure that covers the entire 360 ° direction of the center 21. FIG. 4 illustrates only the upper half celestial sphere in the entire virtual space 2. A plurality of substantially square or substantially rectangular meshes are associated with the virtual space 2. The position of each mesh in the virtual space 2 is defined in advance as coordinates in a space coordinate system (XYZ coordinate system) defined in the virtual space 2. The control circuit unit 200 associates each partial image constituting content (still image, moving image, etc.) that can be developed in the virtual space 2 with each corresponding mesh in the virtual space 2, thereby enabling the virtual space image that can be visually recognized by the user. The virtual space 2 in which 22 is expanded is provided to the user.

  The virtual space 2 defines an XYZ space coordinate system with the center 21 as the origin. The XYZ coordinate system is parallel to the global coordinate system, for example. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, vertical direction (vertical direction), and front-rear direction in the XYZ coordinate system are defined as an X axis, a Y axis, and a Z axis, respectively. That is, the X axis (horizontal direction) of the XYZ coordinate system is parallel to the x axis of the global coordinate system, the Y axis (vertical direction) of the XYZ coordinate system is parallel to the y axis of the global coordinate system, and The Z axis (front-rear direction) is parallel to the z axis of the global coordinate system.

  When the HMD 110 is activated (initial state), the virtual camera 1 is arranged at the center 21 of the virtual space 2. The virtual camera 1 moves similarly in the virtual space 2 in conjunction with the movement of the HMD 110 in the real space. Thereby, changes in the position and orientation of the HMD 110 in the real space are similarly reproduced in the virtual space 2.

  As with the HMD 110, the uvw visual field coordinate system is defined for the virtual camera 1. The uvw visual field coordinate system of the virtual camera 1 in the virtual space 2 is defined to change to the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 1 also changes accordingly. The virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user wearing the HMD 110 in the real space.

  The orientation of the virtual camera 1 in the virtual space 2 is determined according to the position and inclination of the virtual camera 1 in the virtual space 2. As a result, a line of sight (reference line of sight 5) as a reference when the user visually recognizes the virtual space image 22 developed in the virtual space 2 is determined. The control circuit unit 200 determines the visual field region 23 in the virtual space 2 based on the reference visual line 5. The visual field area 23 is an area corresponding to the visual field of the user wearing the HMD 110 in the virtual space 2.

  FIG. 5 is a view showing a cross section of the visual field region 23. FIG. 5A shows a YZ cross section of the visual field region 23 viewed from the X direction in the virtual space 2. FIG. 5B shows an XZ cross-section of the visual field region 23 viewed from the Y direction in the virtual space 2. The visual field region 23 is defined by the first region 24 (see FIG. 5A) that is a range defined by the reference line of sight 5 and the YZ section of the virtual space 2, and the reference line of sight 5 and the XZ section of the virtual space 2. And a second region 25 (see FIG. 5B) which is a defined range. The control circuit unit 200 sets a range including the polar angle α around the reference line of sight 5 in the virtual space 2 as the first region 24. In addition, a range including the azimuth angle β around the reference line of sight 5 in the virtual space 2 is set as the second region 25.

  The HMD system 100 provides the user with the virtual space 2 by causing the display image 112 of the HMD 110 to display a view image 26 that is a portion of the virtual space image 22 that is superimposed on the view region 23. If the user moves the HMD 110, the virtual camera 1 also moves in conjunction with it, and as a result, the position of the visual field area 23 in the virtual space 2 changes. As a result, the view image 26 displayed on the display 112 is updated to an image that is superimposed on a portion of the virtual space image 22 where the user faces (= view region 23). Therefore, the user can visually recognize a desired location in the virtual space 2.

  While wearing the HMD 110, the user visually recognizes only the virtual space image 22 developed in the virtual space 2 without seeing the real world. Therefore, the HMD system 100 can give a high immersive feeling to the virtual space 2 to the user.

  The control circuit unit 200 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user wearing the HMD 110 in the real space. In this case, the control circuit unit 200 identifies the visual field region 23 that is visually recognized by the user by being projected on the display 112 of the HMD 110 in the virtual space 2 based on the position and orientation of the virtual camera 1 in the virtual space 2.

  The virtual camera 1 preferably includes a right-eye virtual camera that provides a right-eye image and a left-eye virtual camera that provides a left-eye image. Furthermore, it is preferable that appropriate parallax is set for the two virtual cameras so that the user can recognize the three-dimensional virtual space 2. In the present embodiment, only the virtual camera 1 in which the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD 110 is represented. Will be shown and described.

(Gaze direction detection)
The gaze sensor 130 has an eye tracking function that detects the direction (gaze direction) in which the line of sight of the user's right eye and left eye is directed. As the gaze sensor 130, a known sensor having an eye tracking function can be employed. The gaze sensor 130 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 130 may be, for example, a sensor that detects the rotation angle of each eyeball by irradiating the user's right eye and left eye with infrared light and receiving reflected light from the cornea and iris with respect to the irradiated light. The gaze sensor 130 can detect the direction of the user's line of sight based on each detected rotation angle.

  The user's gaze direction detected by the gaze sensor 130 is a direction in the viewpoint coordinate system when the user visually recognizes the object. As described above, the uvw visual field coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user visually recognizes the display 112. Further, the uvw visual field coordinate system of the virtual camera 1 is linked to the uvw visual field coordinate system of the HMD 110. Therefore, in the HMD system 100, the user's line-of-sight direction detected by the gaze sensor 130 can be regarded as the user's line-of-sight direction in the uvw visual field coordinate system of the virtual camera 1.

  FIG. 6 is a diagram illustrating a method of determining the user's line-of-sight direction. As shown in this figure, the gaze sensor 130 detects the line of sight of the right eye and the left eye of the user U. When the user U is looking nearby, the gaze sensor 130 detects the line of sight R1 and L1 of the user U. When the user looks far away, the gaze sensor 130 identifies the line of sight R2 and L2 that are smaller in angle with the roll direction (w) of the HMD 110 than the line of sight R1 and L1 of the user. The gaze sensor 130 transmits the detection value to the control circuit unit 200.

  When receiving the line of sight R1 and L1 as the line-of-sight detection value, the control circuit unit 200 specifies the gazing point N1 that is the intersection of the two. On the other hand, also when the lines of sight R2 and L2 are received, the gazing point N1 (not shown) that is the intersection of the two is specified. The control circuit unit 200 detects the gaze direction N0 of the user U based on the identified gazing point N1. For example, the control circuit unit 200 detects, as the line-of-sight direction N0, the direction in which the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user U and the gazing point N1 extends. The line-of-sight direction N0 is a direction in which the user U actually points the line of sight with both eyes. The line-of-sight direction N0 is also a direction in which the user U actually directs his / her line of sight with respect to the visual field region 23.

  The HMD system 100 may include a microphone and a speaker in any element constituting the HMD system 100. Thereby, the user can give a voice instruction to the virtual space 2. Further, in order to receive a broadcast of a television program on a virtual television in a virtual space, the HMD system 100 may include a television receiver as any element. Further, it may include a communication function or the like for displaying an electronic mail or the like acquired by the user.

(Controller 300)
The controller 300 is a device that can transmit various commands based on user operations to the control circuit unit 200. The controller 300 can be a portable terminal capable of wired or wireless communication. The controller 300 may be, for example, a smartphone, a PDA (Personal Digital Assistant), a tablet computer, a game console, a general-purpose PC (Personal Computer), or the like. The controller 300 is preferably a device including a touch panel, and any terminal including a processor, a memory, a storage, a communication unit, and a touch panel in which a display unit and an input unit are integrated with each other may be employed. . The user inputs various touch operations including tap, swipe, and hold on the touch panel of the controller 300, thereby affecting various objects and UI (User Interface) arranged in the virtual space 2. be able to.

(Functional configuration of control circuit unit 200)
FIG. 7 is a block diagram showing a functional configuration of the control circuit unit 200. The control circuit unit 200 controls the virtual space 2 provided to the user by using various data received from the HMD sensor 120, the gaze sensor 130, and the controller 300, and displays an image on the display 112 of the HMD 110. Control. As illustrated in FIG. 7, the control circuit unit 200 includes a detection unit 210, a display control unit 220, a virtual space control unit 230, a storage unit 240, and a communication unit 250. The control circuit unit 200 functions as a detection unit 210, a display control unit 220, a virtual space control unit 230, a storage unit 240, and a communication unit 250 by the cooperation of the hardware illustrated in FIG. The functions of the detection unit 210, the display control unit 220, and the virtual space control unit 230 can be realized mainly by the cooperation of the processor and the memory. The function of the storage unit 240 can be realized mainly by the cooperation of the memory and the storage. The function of the communication unit 250 can be realized mainly by the cooperation of the processor and the communication interface.

  The detection unit 210 receives detection values from various sensors (such as the HMD sensor 120) connected to the control circuit unit 200. Moreover, the predetermined process using the received detected value is performed as needed. The detection unit 210 includes an HMD detection unit 211, a line-of-sight detection unit 212, and an operation reception unit 213. The HMD detection unit 211 receives detection values from the HMD 110 and the HMD sensor 120, respectively. The line-of-sight detection unit 212 receives the detection value from the gaze sensor 130. The operation reception unit 213 receives the operation by receiving a command transmitted in response to a user operation on the controller 300.

  The display control unit 220 controls image display on the display 112 of the HMD 110. The display control unit 220 includes a virtual camera control unit 221, a visual field region determination unit 222, and a visual field image generation unit 223. The virtual camera control unit 221 arranges the virtual camera 1 in the virtual space 2 and controls the behavior of the virtual camera 1 in the virtual space 2. The view area determination unit 222 determines the view area 23. The view image generation unit 223 generates a view image 26 displayed on the display 112 based on the determined view area 23.

  The virtual space control unit 230 controls the virtual space 2 provided to the user. The virtual space control unit 230 includes a virtual space defining unit 231, a line-of-sight management unit 232, a content specifying unit 233, an operation object control unit 234, and a rotation control unit 235. The virtual space defining unit 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2 provided to the user. The line-of-sight management unit 232 manages the user's line of sight in the virtual space 2. The content specifying unit 233 specifies the content to be reproduced in the virtual space 2. The operation object control unit 234 controls the operation object 27 in the virtual space 2. The rotation control unit 235 controls the rotation of the virtual camera 1 or the virtual space 2 based on the operation on the operation object 27.

  The storage unit 240 stores various data used by the control circuit unit 200 to provide the virtual space 2 to the user. The storage unit 240 includes a template storage unit 241 and a content storage unit 242. The template storage unit 241 stores various template data. The content storage unit 242 stores various types of content.

  The template data is data representing a template of the virtual space 2. The template data has spatial structure data that defines the spatial structure of the virtual space 2. The spatial structure data is data that defines the spatial structure of a 360-degree celestial sphere centered on the center 21, for example. The template data further includes data defining the XYZ coordinate system of the virtual space 2. The template data further includes coordinate data for specifying the position of each mesh constituting the celestial sphere in the XYZ coordinate system. The template data further includes a flag indicating whether or not an object can be arranged in the virtual space 2.

  The content is content that can be reproduced in the virtual space 2. Examples of content include platform content and viewing content.

  The platform content is content related to an environment (platform) for allowing the user to select viewing content that the user wants to view in the virtual space 2. By reproducing the platform content in the virtual space 2, a platform for content selection is provided to the user. The platform content has at least a background image and data defining an object.

  The viewing content is, for example, a still image content or a moving image content. The still image content has a background image. The moving image content includes at least an image (still image) of each frame. The moving image content may further include audio data.

  The moving image content is content generated by an omnidirectional camera, for example. An omnidirectional camera is a camera that can generate images in all directions at once by photographing all directions in a real space around the lens of the camera at once. Each image constituting the moving image content obtained by the omnidirectional camera is distorted, but when the moving image content is reproduced in the virtual space 2, the distortion of each image is canceled by the lens constituting the display 112 of the HMD 110. It is solved by. Therefore, at the time of reproduction of the moving image content, the user can visually recognize a natural image without distortion in the virtual space 2.

  In each content, an initial direction facing an image to be shown to the user in the initial state (at the time of activation) of the HMD 110 is defined in advance. The initial direction defined in the moving image content generated by the omnidirectional camera usually matches the predetermined shooting direction defined in the omnidirectional camera used for shooting the moving image content. The initial direction can be changed to a direction different from the shooting direction. Specifically, after shooting with the omnidirectional camera, the obtained moving image content is appropriately edited, whereby the direction deviated from the shooting direction can be defined as the moving image content.

  The communication unit 250 transmits / receives data to / from an external device 400 (for example, a server) via the network NW.

(Process of providing virtual space 2)
FIG. 8 is a sequence diagram showing a flow of processing in which the HMD system 100 provides the virtual space 2 to the user. The virtual space 2 is basically provided to the user through the cooperation of the HMD 110 and the control circuit unit 200. When the process shown in FIG. 8 is started, first, in S1, the virtual space defining unit 231 generates virtual space data representing the virtual space 2 provided to the user, thereby defining the virtual space 2. The generation procedure is as follows. First, the virtual space defining unit 231 defines the prototype of the virtual space 2 by acquiring the template data of the virtual space 2 from the template storage unit 241. The virtual space defining unit 231 further acquires content to be played back in the virtual space 2 from the content storage unit 242. The virtual space defining unit 231 generates virtual space data defining the virtual space 2 by adapting the acquired content to the acquired template data. The virtual space defining unit 231 appropriately associates each partial image constituting the background image included in the content with the management data of each mesh constituting the celestial sphere of the virtual space 2 in the virtual space data. The virtual space defining unit 231 preferably associates each partial image with each mesh so that the initial direction defined in the content matches the Z direction in the XYZ coordinate system of the virtual space 2.

  The virtual space defining unit 231 further adds management data of each object included in the content to the virtual space data as necessary. At that time, coordinates representing the position where the corresponding object is arranged in the virtual space 2 are set in each management data. Thereby, each object is arranged at the position of the coordinate in the virtual space 2.

  After that, when the HMD 110 is activated by the user, the HMD sensor 120 detects the position and inclination of the HMD 110 in the initial state in S2, and outputs the detected value to the control circuit unit 200 in S3. The HMD detection unit 211 receives this detection value. Thereafter, in S4, the virtual camera control unit 221 initializes the virtual camera 1 in the virtual space 2.

  The initialization procedure is as follows. First, the virtual camera control unit 221 places the virtual camera 1 at an initial position in the virtual space 2 (eg, the center 21 in FIG. 4). Next, the orientation of the virtual camera 1 in the virtual space 2 is set. At that time, the virtual camera control unit 221 specifies the uvw visual field coordinate system of the HMD 110 in the initial state based on the detection value from the HMD sensor 120 and sets the uvw visual field coordinate system that matches the uvw visual field coordinate system of the HMD 110 to the virtual camera 1. The orientation of the virtual camera 1 may be set by setting to. When setting the uvw visual field coordinate system for the virtual camera 1, the virtual camera control unit 221 adapts the roll direction (w axis) of the virtual camera 1 to the Z direction (Z axis) of the XYZ coordinate system. Specifically, the virtual camera control unit 221 matches the direction obtained by projecting the roll direction of the virtual camera 1 on the XZ plane with the Z direction of the XYZ coordinate system, and the roll direction of the virtual camera 1 with respect to the XZ plane. Is matched with the inclination of the roll direction of the HMD 110 with respect to the horizontal plane. As a result of such adaptation processing, the roll direction of the virtual camera 2 in the initial state is adapted to the initial direction of the content, so that the horizontal direction that the user first faces after the start of content reproduction matches the initial direction of the content Can be made.

  When the initialization process of the virtual camera 1 is completed, the visual field region determination unit 222 determines the visual field region 23 in the virtual space 2 based on the uvw visual field coordinate system of the virtual camera 1. Specifically, the roll direction (w axis) of the uvw visual field coordinate system of the virtual camera 1 is specified as the reference visual line 5 of the user, and the visual field region 23 is determined based on the reference visual line 5. In S <b> 5, the visual field image generation unit 223 corresponds to a portion projected on the visual field region 23 in the virtual space 2 out of the entire virtual space image 22 developed in the virtual space 2 by processing the virtual space data. A view image 26 is generated (rendered). In S6, the visual field image generation unit 223 outputs the generated visual field image 26 to the HMD 110 as an initial visual field image. In S <b> 7, the HMD 110 displays the received initial view image on the display 112. Thereby, the user visually recognizes the initial view image.

  Thereafter, in S8, the HMD sensor 120 detects the current position and inclination of the HMD 110, and outputs these detected values to the control circuit unit 200 in S9. The HMD detection unit 211 receives each detection value. The virtual camera control unit 221 specifies the current uvw visual field coordinate system in the HMD 110 based on the position and tilt detection values of the HMD 110. Furthermore, in S <b> 10, the virtual camera control unit 221 specifies the roll direction (w axis) of the uvw visual field coordinate system in the XYZ coordinate system as the visual field direction of the HMD 110.

  In the present embodiment, in S <b> 11, the virtual camera control unit 221 identifies the identified visual field direction of the HMD 110 as the user's reference visual line 5 in the virtual space 2. In S <b> 12, the virtual camera control unit 221 controls the virtual camera 1 based on the identified reference line of sight 5. If the position (starting point) and direction of the reference line of sight 5 are the same as the initial state of the virtual camera 1, the virtual camera control unit 221 maintains the position and direction of the virtual camera 1 as they are. On the other hand, if the position (starting point) and / or direction of the reference line of sight 5 has changed from the initial state of the virtual camera 1, the position and / or inclination of the virtual camera 1 in the virtual space 2 is changed to the reference line of sight after the change. Change the position and / or inclination according to 5. Further, the uvw visual field coordinate system is reset for the virtual camera 1 after the control.

  In S <b> 13, the visual field region determination unit 222 determines the visual field region 23 in the virtual space 2 based on the identified reference visual line 5. Thereafter, in S14, the visual field image generation unit 223 projects (superimposes) the entire virtual space image 22 developed in the virtual space 2 onto the visual field region 23 in the virtual space 2 by processing the virtual space data. A view field image 26 is generated (rendered). In S15, the visual field image generation unit 223 outputs the generated visual field image 26 to the HMD 110 as a visual field image for update. In S <b> 16, the HMD 110 updates the view image 26 by displaying the received view image 26 on the display 112. Thereby, if a user moves HMD110, the view image 26 will be updated in response to it.

(Platform provision and content playback processing)
In the present embodiment, the HMD system 100 provides the user with an environment (platform) for the user to select content in the virtual space 2 that the user wants to view. When the user selects content to view through the platform developed in the virtual space 2, the control circuit unit 200 starts reproduction of the selected content in the virtual space 2. Details of the platform provision and content playback processing will be described below.

  FIG. 9 is a sequence diagram illustrating a flow of processing in which the HMD system 100 reproduces the moving image content selected by the user through the platform in the virtual space 2 in the virtual space 2. FIG. 10 is a diagram for explaining selection and playback of moving image content through the platform in the virtual space 2. When the processing shown in FIG. 9 is started, first, in S21, the virtual space defining unit 231 creates virtual space data representing the virtual space 2 for providing the platform, thereby defining the virtual space 2 for providing the platform. To do. The generation procedure is as follows. First, the virtual space defining unit 231 acquires the template data of the virtual space 2 corresponding to the platform from the template storage unit 241. Here, the template data of the virtual space 2 in which the object can be used is acquired.

  The virtual space defining unit 231 further acquires platform content related to the platform provided in the virtual space 2 from the content storage unit 242. The virtual space defining unit 231 generates virtual space data defining the virtual space 2 for providing the platform by adapting the acquired platform content to the acquired template data. In the virtual space data, the virtual space defining unit 231 appropriately associates each partial image constituting the background image included in the platform content with the management data of each mesh constituting the celestial sphere of the virtual space 2.

  The virtual space defining unit 231 further adds management data of each object included in the platform content to the virtual space data. Thus, each object is placed at a predetermined position in the virtual space 2 that provides the platform. One of the objects is an operation object operated by the user. As will be described in detail later, the operation object 27 is used to rotate the virtual camera 1 or the virtual space 2. The rest are thumbnail objects used to display a summary image (thumbnail) of the viewing content. Next, the virtual space defining unit 231 acquires thumbnails of a predetermined number of viewing contents (here, moving image contents) stored in the content storage unit 242. The virtual space defining unit 231 associates each acquired thumbnail with the management data of any thumbnail object in the virtual space data. Thereby, a thumbnail is associated with each thumbnail object arranged in the virtual space 2. In the following, for convenience of explanation, a thumbnail object arranged in the virtual space 2 and associated with a thumbnail may be simply referred to as a thumbnail.

  Each viewing content in which the corresponding thumbnail is associated with the object is a moving image content candidate (candidate moving image content) that can be selected for reproduction in the virtual space 2 by the user. The user can select the corresponding candidate moving image content through selection of the thumbnail. The control circuit unit 200 identifies the candidate video content selected by the user on the platform as video content to be played back in the virtual space 2.

  FIG. 10A shows an example of a virtual space 2 for providing a platform. This figure shows a virtual space 2 for providing a platform in which one operation object 27 and four thumbnails SN1 to SN4 are arranged. Each of these thumbnails SN1 to SN4 is an object associated with a summary image (thumbnail) of the corresponding moving image content. The shape of the operation object 27 is basically circular. As will be described in detail later, the operation object 27 is elastically deformed (expanded) in response to an operation on the operation object 27 by the user.

  After generating the virtual space data representing the virtual space 2 shown in FIG. 10A, the view image generation unit 223 generates the view image 26 based on the user's reference line of sight 5 in S22. This generation method is the same as the method described with reference to FIG. Here, a view field image 26 corresponding to the platform is generated. In FIG. 10A, among the thumbnails SN1 to N4, an operation object 27, thumbnails SN1, and SN2 are arranged inside the field of view area 23 defined by the user's reference line of sight 5. On the other hand, the thumbnails SN3 and SN4 are arranged outside the field-of-view area 23. Therefore, the view image generation unit 223 generates a view image 26 including the operation object 27 and thumbnails SN1 and SN2. As will be described in detail later, the visual field image generation unit 223 further generates a visual field image 26 including the gazing point 28. The gaze point 28 is information indicating which position in the view image 26 the user is gazing at.

  In S <b> 23, the visual field image generation unit 223 outputs the generated visual field image 26 to the HMD 110. In S <b> 24, the HMD 110 displays the received view field image 26 on the display 112. The user views the platform view image 26. When an object associated with a thumbnail is included in the view image 26, the thumbnail associated with the object is displayed on the display 112 when the view image 26 is displayed. Thereby, the user visually recognizes the view field image 26 including the thumbnail. When the operation object 27 is included in the view image 26, the operation object 27 is displayed on the display 112 when the view image 26 is displayed. Thereby, the user visually recognizes the view field image 26 including the operation object 27.

  FIG. 10B shows an example of the platform view image 26. The view image 26 shown in this figure includes an operation object 27, a gazing point 28, a thumbnail SN1, and a thumbnail SN2. Therefore, the user visually recognizes the operation object 27, the gazing point 28, the thumbnail SN1, and the thumbnail SN2 in the virtual space 2.

  The visual field image generation unit 223 determines the position of the gazing point 28 in the visual field image 26 based on the reference visual line 5 and the visual line direction N0. When the user is viewing the front, that is, when the reference line-of-sight 5 coincides with the line-of-sight direction N0, the visual field image generation unit 223 applies a note to the center of the visual field image 26 based on the reference visual line 5 when generating the visual field image 26. The viewpoint 28 is arranged. When the user does not change the line of sight while viewing the front, the reference line of sight 5 coincides with the line-of-sight direction N0, so that the gazing point 28 is always arranged at the center of the field-of-view image 26.

  On the other hand, the visual field image generation unit 223 determines the position of the gazing point 28 in the visual field image 26 based on the current visual line direction N0 when the visual line direction N0 deviates from the reference visual line 5 because the user moves the line of sight. Specifically, an intersection point where the line-of-sight direction N0 intersects in the visual field region 23 is specified, and then a visual field image 26 in which the gazing point 28 is applied to a position corresponding to the intersection point in the visual field image 26 is generated.

  When the user moves only the line of sight without moving the HMD 110 (while the reference line of sight 5 is fixed), the position of the gazing point 28 in the view field image 26 is changed following the movement of the line of sight. As a result, the view field image 26 is updated so that the gazing point 28 is displayed at the position where the user gazed in the view field image 26. In other words, the user can freely move the gazing point 28 in the view field image 26 by the movement of the line of sight. Accordingly, the user can accurately grasp which part of the view field image 26 is line of sight by confirming the gazing point 28.

  Although not shown in FIG. 9, if the user moves the HMD 110 thereafter, the view field image 26 is updated in conjunction with the movement. For example, when the user moves the HMD 110 and the position of the visual field area 23 changes to a position including the thumbnails SN1 and SN3, the visual field image 26 including the thumbnails SN1 and SN3 is displayed on the display 112. Therefore, the user can put the thumbnail of the moving image content that he / she wants to watch in his / her field of view by moving the HMD 110 as appropriate.

  After the platform view image 26 is displayed, the gaze sensor 130 detects the user's right eye gaze and left eye gaze in S25, and transmits each detected value to the control circuit unit 200 in S26. The line-of-sight detection unit 212 receives this detection value. In S <b> 27, the line-of-sight detection unit 212 identifies the user's line-of-sight direction N <b> 0 in the uvw visual field coordinate system of the virtual camera 1 using the received detection value.

  In S28, based on the line-of-sight direction N0 and each thumbnail included in the field-of-view area 23, the line-of-sight management unit 232 hits a specific thumbnail included in the field-of-view image 26 with a user's line of sight (gaze point 28) for a specified time or more. Determine whether or not. More specifically, the line-of-sight management unit 232 determines whether or not the point where the line-of-sight direction N0 in the visual field region 23 intersects is included in the display range (arrangement range) of a specific thumbnail included in the visual field region 23. If the determination result is YES, it is determined that the line of sight has hit a specific thumbnail included in the field-of-view image 26. If NO, it is determined that the line of sight has not hit the specific thumbnail.

  When S28 is No, the process of FIG. 9 returns to immediately before S25. Thereafter, the processes of S25 to S28 are repeated until S28 becomes YES. On the other hand, when S28 is YES, in S29, the content specifying unit 233 specifies the content corresponding to the thumbnail that has been determined that the line of sight has been hit for a specified time or more. For example, when the user focuses on the thumbnail SN1, the gazing point 28 is superimposed on the thumbnail SN1 in the view image 26. Furthermore, when the user focuses on the thumbnail SN1 for a predetermined time or more, the moving image content associated with the management data of the thumbnail SN1 is specified as the content corresponding to the thumbnail SN1.

  Thereafter, in S30, the virtual space defining unit 231 defines the virtual space 2 for reproducing the moving image content by generating virtual space data for reproducing the specified moving image content. The generation procedure is as follows. First, the virtual space defining unit 231 acquires the template data of the virtual space 2 corresponding to the moving image content from the template storage unit 241.

  The virtual space defining unit 231 acquires the moving image content specified by the content specifying unit 233 from the content storage unit 242. The virtual space defining unit 231 generates virtual space data that defines the virtual space 2 for reproducing moving image content by adapting the acquired moving image content to the acquired template data. The virtual space defining unit 231 appropriately associates each partial image constituting the first frame image included in the moving image content with the management data of each mesh constituting the celestial sphere of the virtual space 2 in the virtual space data.

  In the virtual space 2 defined by the virtual space data generated here, it is not assumed that an object is arranged. Furthermore, the moving image content does not include management data that defines the object. Therefore, in S30, the virtual space defining unit 231 generates virtual space data that does not include object management data.

  After generating the virtual space data representing the virtual space 2 for reproducing moving image content, the view image generating unit 223 generates the view image 26 based on the reference line of sight 5 of the user in S31. This generation method is the same as the method described with reference to FIG. Here, a view image 26 of the moving image content is generated. In S <b> 32, the visual field image generation unit 223 outputs the generated visual field image 26 to the HMD 110. In S <b> 33, the HMD 110 updates the view image 26 by displaying the received view image 26 on the display 112. Thereby, the reproduction of the moving image content in the virtual space 2 is started, and the user visually recognizes the view image 26 of the moving image content.

  Although not shown in FIG. 9, if the user moves the HMD 110 thereafter, the view field image 26 is updated in conjunction with the movement. Therefore, the user can visually recognize a partial image (view image 26) at a desired position in the omnidirectional image of each frame constituting the moving image content by appropriately moving the HMD 110.

  FIG. 10C shows an example of the visual field image 26 of the moving image content. The view image 26 shown in this figure is the view image 26 of the moving image content corresponding to the thumbnail SN1 selected by the user. As described above, when the user selects the thumbnail SN1 by the line of sight while viewing the platform view image 26 shown in FIG. 10B, the platform view image 26 displayed on the display 112 is shown in FIG. The view image 26 of the moving image content shown in FIG. That is, the user can view the moving image content corresponding to this in the virtual space 2 by selecting the thumbnail SN1 by moving the line of sight in the virtual space 2.

  As described above, the user can select the moving image content desired to view in the virtual space 2 through the moving image content selection platform in the virtual space 2. Therefore, the user does not need to select the moving image content that the user wants to view in the virtual space 2 while visually recognizing another general display connected to the control circuit unit 200 in the real space before wearing the HMD 110. Thereby, a user's immersion feeling with respect to the virtual space 2 can be improved further.

  In addition, if the user performs a predetermined operation on the HMD system 100 (for example, through the controller 300) during reproduction of the moving image content, the control circuit unit 200 provides the user with the virtual space 2 in which the moving image content is reproduced. Then, again, the virtual space 2 in which the platform for selecting moving image contents is developed is provided to the user. Thus, the user can view other moving image content in the virtual space 2 by selecting another thumbnail. Since the user does not need to remove the HMD 110 when switching the moving image content that the user wants to view, the user's immersion in the virtual space 2 can be further enhanced.

(Rotation control of virtual camera 1)
The control circuit unit 200 according to the present embodiment operates in the normal mode (first mode) or the rotation mode (second mode). The control method of the virtual camera 1 by the control circuit unit 200 is different between the normal mode and the rotation mode. In the normal mode, the control circuit unit 200 controls the tilt of the virtual camera 1 in the virtual space 2 in conjunction with the tilt of the HMD 110. Therefore, when the user tilts the HMD 110 in the normal mode, the virtual camera 1 is also tilted in conjunction with the tilt, and the view field image 26 corresponding to the tilt is displayed on the HMD 110.

  In the rotation mode, the control circuit unit 200 does not control the tilt of the virtual camera 1 in the virtual space 2 in conjunction with the tilt of the HMD 110. Therefore, even if the user tilts the HMD 110 in the rotation mode, the virtual camera 1 does not tilt in conjunction with the tilt. Instead, the control circuit unit 200 rotates the virtual camera 1 in the virtual space 2 according to the operation of the operation object 27 by the user. In the present embodiment, the user can operate the operation object 27 by the movement of the HMD 110. Therefore, the control circuit unit 200 rotates the virtual camera 1 according to the inclination of the HMD 110 in the rotation mode.

  In the normal mode, when the user stops the movement of the HMD 110, the movement of the virtual camera 1 also stops. Therefore, the update of the view image 26 is also stopped. On the other hand, in the rotation mode, even if the user stops the movement of the HMD 110, the rotation of the virtual camera 1 does not stop. That is, as long as the user continues to maintain the state in which the HMD 110 is further tilted from the state at the time of starting rotation, the virtual camera 1 continues to rotate. Thereby, the view image 26 is also continuously updated.

  FIG. 11 is a sequence diagram illustrating a processing flow when the HMD system 100 shifts to the rotation mode. Hereinafter, a case where the virtual space 2 in which the platform is reproduced is provided to the user will be described as an example.

  After the platform view image 26 is displayed, the HMD sensor 120 detects the position and inclination of the HMD 110 in S41, and outputs the detection value to the control circuit unit 200 in S42. The HMD detection unit 211 receives this detection value. The virtual camera control unit 221 specifies the reference line of sight 5 according to the above-described procedure based on the detected values of the position and inclination of the HMD 110. The virtual camera control unit 221 controls the virtual camera 1 based on the identified reference line of sight 5.

  Next, in S43, the operation object control unit 234 determines whether or not the reference line of sight 5 has hit the operation object 27 in the virtual space 2 for a specified time or more. In the case of NO in S <b> 43, the visual field area determination unit 222 determines the visual field area 23 in the virtual space 2 based on the identified reference visual line 5. Thereafter, in S44, the visual field image generation unit 223 generates the visual field image 26 and outputs the visual field image 26 to the HMD 110 in S45. In S <b> 46, the HMD 110 updates the view image 26 by displaying the received view image 26 on the display 112. Thereby, if the user moves the HMD 110 in the normal mode, the view image 26 is updated in conjunction with the movement.

  When the user moves the HMD 110, if the operation object 27 is included in the view area 23, the operation object 27 does not move in the same manner as other objects. On the other hand, when the user moves the HMD 110 and the operation object 27 moves out of the view area 23, the operation object 27 moves so as to return to the view area 23.

  The procedure for realizing this is as follows. First, the operation object control unit 234 detects that at least a part of the operation object 27 is located outside the field-of-view area 23. Next, when the operation object control unit 234 detects that at least a part of the operation object 27 is located outside the view field area 23, the operation object control unit 234 moves the operation object 27 into the view field area 23. Specifically, the position (coordinates) of the operation object 27 in the data defining the operation object 27 is updated to any coordinate in the current visual field region 23.

  The position where the operation object 27 returns to the view area 23 may be any position in the view area 23. It is preferable that the position is such that the movement amount is the smallest when the operation object 27 moves. For example, when the operation object 27 is positioned on the right side of the visual field area 23, the operation object control unit 234 preferably moves the operation object 27 to the right end of the visual field area 23.

  Thereby, in the normal mode, since the operation object 27 is always included in the view area 23, the view object 26 generated based on the view area 23 always includes the operation object 27. Therefore, when the user wants to select the operation object 27, the user can easily find the operation object 27. Further, as will be described later, when the operation object 27 is selected by the reference line of sight 5, the movement control of the HMD 110 for bringing the reference line of sight 5 to the operation object 27 can be minimized.

  On the other hand, in the case of YES in S43, the operation object control unit 234 detects that the operation object 27 has been selected by the user in S47. In this way, the user can select the operation object 27 by placing the line of sight on the operation object 27. Using the selection of the operation object 27 as a trigger, the control circuit unit 200 shifts to the rotation mode in S48. Thereby, the user can shift the control circuit unit 200 to the rotation mode by his / her own intention. After shifting to the rotation mode, the rotation of the virtual camera 1 is started in response to a user operation on the operation object 27.

  FIG. 12 is a sequence diagram showing a flow of processing when the control circuit unit 200 starts to rotate the virtual camera 1 in the rotation mode.

  After shifting to the rotation mode, the HMD sensor 120 detects the position and inclination of the HMD 110 in S51, and outputs the detected value to the control circuit unit 200 in S52. The HMD detection unit 211 receives this detection value. In S53, the operation object control unit 234 determines whether or not the inclination of the HMD 110 has changed. If NO in S53, the process in FIG. 12 returns to before S51. Therefore, after the operation object 27 is selected, the steps S51 to S53 are repeated until the user further tilts the HMD 110.

  On the other hand, in the case of YES in S53, the operation object control unit 234 specifies the amount of change in the inclination of the HMD 110 in S54. In S55, the operation object control unit 234 specifies the extension direction (predetermined direction) and the extension amount (predetermined amount) of the operation object 27 designated by the user based on the specified change amount of the tilt. In S56, the operation object control unit 234 extends the operation object 27 by the specified extension amount in the specified extension direction. At that time, the operation object control unit 234 expands the operation object 27 so that the operation object 27 gradually becomes narrower from the root of the operation object 27 toward the end portion in the expansion direction. Further, the display state of the operation object 27 is changed so that the reference line of sight 5 is superimposed on the end portion of the operation object 27 in the extension direction.

  Thereafter, in S <b> 57, the rotation control unit 235 determines the rotation direction of the virtual camera 1 based on the extension direction of the operation object 27 and determines the rotation speed of the virtual camera 1 based on the extension amount of the operation object 27. At this time, the operation object control unit 234 preferably specifies a direction parallel to the horizontal component in the extension direction as the rotation direction of the virtual camera 1. Thereby, the virtual camera 1 rotates only around the Y axis in the virtual space 2. Further, the virtual camera 1 can be prevented from rotating around the Z axis in the virtual space 2. Thereby, since it can prevent that the user who the ground of the virtual space 2 moves up and down visually at the time of the update of the visual field image 26, generation | occurrence | production of the user's sickness in the virtual space 2 can be prevented.

  In S58, the virtual camera control unit 221 starts the rotation of the virtual camera 1 at the determined rotation speed in the determined rotation direction. When the virtual camera 1 rotates, the roll direction of the virtual camera 1 in the virtual space 2 changes, so that the field-of-view area 23 is changed. The operation object control unit 234 controls the operation object 27 so that the operation object 27 is always arranged at the center of the visual field area 23 when the virtual camera 1 is rotated. Specifically, in the management data of the operation object 27, the coordinates of the operation object 27 are updated to the coordinates at the center of the visual field area 23. Thereby, both the virtual camera 1 and the operation object 27 rotate in the virtual space 2 in a state where the position of the operation object 27 in the uvw visual field coordinate system of the virtual camera 1 is fixed.

  The visual field area determination unit 222 determines the visual field area 23 based on the direction (roll direction) of the virtual camera 1 after rotation for a certain time. In step S59, the visual field image generation unit 223 generates a visual field image 26 based on the determined visual field region 23, and transmits the visual field image 26 to the HMD 110 in step S60. In S <b> 61, the HMD 110 updates the view image 26 by displaying the received view image 26 on the display 112. Thereby, the view image 26 before the rotation of the virtual camera 1 is updated to the view image 26 after the rotation of the virtual camera 1.

  FIG. 13 is a diagram illustrating the virtual space 2 and the view image 26 after the virtual camera 1 is rotated. FIG. 13A shows the virtual space 2 and FIG. 13B shows the view field image 26. Hereinafter, an example will be described in which after the operation object 27 is selected, the user performs an operation on the operation object 27 to expand the operation object 27 to the left side by twisting the neck to the left side. In this case, since the extension direction of the operation object 27 is the left direction when viewed from the user, the counterclockwise rotation direction with the Y axis as the rotation axis is determined as the rotation direction of the virtual camera 1. Therefore, the virtual camera 1 rotates counterclockwise about the Y axis as a rotation axis.

  By rotating the virtual camera 1 counterclockwise, the position of the visual field region 23 is shifted counterclockwise in the virtual space 2. As a result, as shown in FIG. 13A, the positions of the thumbnail SN1 and the thumbnail SN2 in the view field area 23 are shifted to the right relative to the position before the rotation is started. Since the virtual camera 1 is rotating, the operation object 27 is expanded.

  As shown in FIG. 13B, the view field image 26 based on the view field area 23 after the virtual camera 1 rotates includes a thumbnail SN1 and a thumbnail SN2. Since the positions of the thumbnail SN1 and the thumbnail SN2 in the view area 23 are shifted to the right side before the rotation of the virtual camera 1 is started, correspondingly, the positions of the thumbnail SN1 and the thumbnail SN2 in the view image 26 are also the virtual camera. 1 is shifted to the right than before the start of rotation. Therefore, the user can grasp that the virtual camera 1 has started to rotate counterclockwise.

  After the virtual camera 1 starts to rotate, the virtual camera 1 continues to rotate counterclockwise while the user continues to twist his neck to the left. On the other hand, when the user stops twisting his neck and turns his face to the front, the rotation of the virtual camera 1 is stopped. Hereinafter, the flow of these processes will be described.

  FIG. 14 is a sequence diagram illustrating a processing flow when the control circuit unit 200 further rotates the virtual camera 1 or stops the rotation of the virtual camera 1 in the rotation mode.

  While the virtual camera 1 is rotating, the HMD sensor 120 detects the position and inclination of the HMD 110 in S71, and outputs the detected value to the control circuit unit 200 in S72. The HMD detection unit 211 receives this detection value. In S <b> 73, the operation object control unit 234 determines whether or not the current inclination of the HMD 110 matches the inclination of the HMD 110 when the operation object 27 is selected. If NO in S73, the virtual camera control unit 221 further rotates the virtual camera 1 in S74.

  Thereafter, the visual field region determination unit 222 determines the visual field region 23 based on the direction (roll direction) of the virtual camera 1 after rotation for a predetermined time. In step S75, the visual field image generation unit 223 generates the visual field image 26 based on the determined visual field region 23, and transmits the visual field image 26 to the HMD 110 in step S76. In S <b> 77, the HMD 110 updates the view image 26 by displaying the received view image 26 on the display 112. Thereby, the view image 26 after rotating the virtual camera 1 for a certain time is updated to the view image 26 after further rotating the virtual camera 1.

  After S77, the process shown in FIG. 14 returns to before S71. Therefore, after the rotation of the virtual camera 1 is started, the processes of S71 to S77 are repeated until the user restores the inclination of the HMD 110 (returns to the inclination when the operation object 27 is selected). That is, as long as a user operation on the operation object 27 is continuously detected, the virtual camera control unit 221 continues to rotate the virtual camera 1.

  FIG. 15 is a diagram illustrating the virtual space 2 and the field-of-view image 26 after the virtual camera 1 has further rotated. FIG. 15A shows the virtual space 2, and FIG. 15B shows the view field image 26. As the virtual camera 1 further rotates counterclockwise, the position of the visual field region 23 is further shifted counterclockwise in the virtual space 2. As a result, the visual field region 23 is arranged at a position including the thumbnail SN1, the thumbnail SN3, and the thumbnail SN4 in the virtual space 2. Since the virtual camera 1 is rotating, the operation object 27 is still stretched.

  As shown in FIG. 15B, the view field image 26 based on the view field area 23 after the virtual camera 1 further rotates includes a thumbnail SN1, a thumbnail SN3, and a thumbnail SN4. Therefore, the user grasps that the thumbnail SN4 newly enters his field of view while the virtual camera 1 is rotating. Hereinafter, an example in which an operation for stopping the rotation of the virtual camera 1 at this time is performed on the operation object 27 will be described.

  If YES in S73, in S78, the operation object control unit 234 detects that the designation of the extension direction and extension amount to the operation object 27 by the user has been cancelled. Thereby, the operation object control unit 234 returns the shape of the operation object 27 to the original circle in S79. Next, in S80, the operation object control unit 234 cancels the selection of the operation object 27. Next, in S81, the virtual camera control unit 221 stops the rotation of the virtual camera 1.

  After the rotation of the virtual camera 1 is stopped, in S82, the control circuit unit 200 shifts to the normal mode. Thereby, the visual field region determination unit 222 identifies the visual field region 23 based on the orientation of the virtual camera 1 at the time when the rotation of the virtual camera 1 stops. The view image generation unit 223 generates a view image 26 based on the view region 23 in S83, and outputs the view image 26 to the HMD 110 in S84. In S85, the HMD 110 updates the view image 26 by displaying the received view image 26 on the display 112.

  FIG. 16 is a diagram showing the virtual space 2 and the view field image 26 after the rotation of the virtual camera 1 is stopped. FIG. 16A shows the virtual space 2, and FIG. 16B shows the view field image 26. FIG. 16 shows an example in which the rotation of the virtual camera 1 is stopped immediately after the virtual camera 1 is rotated to the state shown in FIG. After the rotation of the virtual camera 1 is stopped, the shape of the operation object 27 in the virtual space 2 returns to the original circle. Correspondingly, the shape of the operation object 27 included in the view field image 26 is also returned to the original circular shape. The operation object 27 is not selected, and the control circuit unit 200 operates in the normal mode. Therefore, the user can select a desired thumbnail in the view field image 26 by moving the line of sight as usual.

  The field-of-view image 26 shown in FIG. 16B includes a thumbnail SN4 that is located away from the thumbnail SN1 in the virtual space 2. Therefore, the user can easily put the thumbnail SN4 that could not be visually recognized before the rotation of the virtual camera 1 into his field of view by a simple operation. Further, the user can start viewing the moving image content corresponding to the thumbnail SN4 by gazing at the thumbnail SN4.

  As described above, the HMD system 100 according to the present invention can enhance the user's immersion in the virtual space 2 in the normal mode. Further, in the rotation mode, the user can visually recognize a desired location in the virtual space 2 with a simpler operation than in the normal mode. Since the HMD system 100 supports both the normal mode and the rotation mode, the operability in the virtual space 2 can be further improved while increasing the user's immersion in the virtual space 2. Further, since each thumbnail can be arranged in a wide range in the virtual space 2, the virtual space 2 can be used more effectively.

  In the rotation mode, the control circuit unit 200 may stop the rotation of the virtual camera 1 when the expansion amount of the operation object 27 exceeds the threshold value. Specifically, the operation object control unit 234 returns the operation object to the original shape and cancels the selection of the operation object when the expansion amount of the operation object 27 exceeds the threshold value in the second mode. The operation object control unit 234 may change the display state of the operation object 27 so as to cut the extended operation object 27 before returning the operation object 27 to its original shape. The virtual camera control unit 221 stops the rotation of the virtual camera 1 when the expansion amount of the operation object 27 exceeds the threshold value. Thereafter, the control circuit unit 200 shifts to the normal mode. Also in this case, the user can stop the virtual camera 1 with a simple operation.

(Example of rotating the virtual space 2)
The control circuit unit 200 may rotate the virtual space 2 instead of the virtual camera 1 based on a user operation on the operation object 27 in the rotation mode. In this case, the rotation control unit 235 notifies the virtual space defining unit 231 of the specified rotation direction and rotation speed. The virtual space defining unit 231 rotates the virtual space 2 with the notified rotation direction and rotation speed. Here, “rotation” means the rotation of the virtual space 2 with the axis passing through the center 21 of the virtual space 2 as the rotation axis.

  The virtual space defining unit 231 rotates the virtual space 2 by rotating the XYZ coordinate system of the virtual space 2 in the global coordinate system. At this time, by processing the data defining the XYZ coordinate system included in the virtual space data, the data is updated to data defining the rotated XYZ coordinate system. Since the position of each mesh in the virtual space 2 is defined in the management data of each mesh as coordinates in the XYZ coordinate system, each mesh is similarly rotated in the global coordinate system when the XYZ coordinate system rotates in the global coordinate system. To do. Similarly, since the positions of the thumbnails SN1 to SN4 arranged in the virtual space 2 are also defined in the management data of each thumbnail as coordinates in the XYZ coordinate system, the thumbnails are displayed by rotating the XYZ coordinate system in the global coordinate system. SN1 to SN4 also rotate in the same manner in the global coordinate system.

  The virtual space defining unit 231 does not rotate the virtual camera 1 together with the virtual space 2 when rotating the virtual space 2. That is, the virtual space 2 is rotated while the position and inclination of the virtual camera 1 in the global coordinate system are fixed. Therefore, when the virtual space 2 rotates, the roll direction of the virtual camera 1 in the virtual space 2 changes relatively (rotates relative to the virtual space 2). As a result, since the position of the visual field area 23 also changes relatively, the visual field image 26 based on the visual field area 23 also changes following the change.

  When the virtual space 2 rotates counterclockwise about the vertical direction as the axis of rotation, the position of the visual field area 23 in the virtual space 2 moves relatively clockwise in the virtual space 2. As a result, the view field image 26 is updated so as to flow from right to left as viewed from the user. Conversely, when the virtual space 2 rotates clockwise about the vertical direction as the axis of rotation, the position of the visual field region 23 in the virtual space 2 moves in a relatively counterclockwise direction in the virtual space 2. As a result, the view field image 26 is updated so as to flow from left to right as viewed from the user.

  The virtual space defining unit 231 performs control so that the operation object 27 is not rotated together with the virtual space 2 when rotating the virtual space 2. Specifically, the virtual space 2 is rotated while maintaining the positional relationship between the reference line of sight 5 and the operation object in the uvw visual field coordinate system of the virtual camera 1. In other words, when the virtual space 2 is rotated, the position of the operation object 27 in the virtual space 2 is updated so that the operation object 27 is always arranged at a point where the reference line of sight 5 intersects in the visual field region 23. As a result, no matter how much the virtual space 2 rotates in the rotation mode, the view object image 26 always includes the operation object 27. The user can recognize that the virtual space 2 is currently rotating based on an operation on the operation object 27 by constantly viewing the extended operation object 27 while the virtual space 2 is rotating.

  The rotation direction of the virtual space 2 based on the extension direction of the operation object 27 designated by the user is preferably opposite to the rotation direction of the virtual camera 1 when the same extension direction is designated. As a result, the direction in which the visual field image 26 flows can be matched between when the virtual camera 1 rotates and when the virtual space 2 rotates. Therefore, regardless of whether the virtual camera 1 or the virtual space 2 is rotated, the direction in which the visual field image 26 flows based on the same operation of the user with respect to the operation object 27 can be made consistent.

(Operation of the operation object 27 by line of sight)
In the normal mode, the user may select or operate the operation object 27 by moving the line of sight instead of the HMD 110. In this case, the user focuses the gazing point 28 on the operation object 27 in the view image 26 by directing a line of sight toward the operation object 27 included in the view image 26. The line-of-sight management unit 232 determines whether or not the gazing point 28 has hit the operation object 27 for a predetermined time or more. When the determination result is true, the operation object control unit 234 detects that the operation object 27 has been selected by the user. Thereby, the control circuit unit 200 shifts to the rotation mode.

  After selecting the operation object 27 by line of sight, the user can operate the operation object 27 by moving the line of sight. In this case, the user changes the line of sight in either direction when the operation object 27 is selected. The line-of-sight management unit 232 identifies the movement direction and amount of movement in the line-of-sight direction N0 based on the respective line-of-sight directions N0 before and after the line-of-sight change. The operation object control unit 234 determines the extension direction of the operation object 27 based on the movement direction in the line-of-sight direction N0, and further determines the extension amount of the operation object 27 based on the movement amount in the line-of-sight direction N0. The extension method of the operation object 27 and the rotation method of the virtual camera 1 (or the virtual space 2) after the extension direction and the extension amount are determined are the same as in the above-described example.

  In the above example, the user can rotate the virtual camera 1 or the virtual space 2 in a desired direction by changing the line of sight (moving the eyeball) without moving the HMD 110. For example, when the user moves his / her line of sight to the right side in the rotation mode, the virtual camera 1 can be rotated clockwise (the virtual space 2 is counterclockwise). On the other hand, when the user moves his / her line of sight to the left side in the rotation mode, the virtual camera 1 can be rotated counterclockwise (the virtual space 2 is rotated clockwise). As described above, the user can place a desired portion in the virtual space 2 in his / her field of view with a simpler operation.

(Rotation control by the controller 300)
The user can also select and operate the operation object 27 by operating the controller 300. In this case, in the normal mode, when the user performs an operation for selecting the operation object 27 on the controller 300, the operation object control unit 234 detects that the operation object 27 has been selected by the user. Further, in the rotation mode, when the user operates the controller 300 to perform an operation of designating the expansion direction and expansion amount of the operation object 27 to the operation object 27, the expansion direction and expansion of the operation object 27 are performed. Specify the amount. The expansion method of the operation object 27 and the rotation method of the virtual camera 1 (or the virtual space 2) after determining the expansion direction and the expansion amount are the same as the above-described example.

  In the above example, the user can rotate the virtual camera 1 or the virtual space 2 in a desired direction by operating the controller 300 without moving the HMD 110 and without changing the line of sight. For example, if the user performs an operation on the controller 300 to extend the operation object 27 to the right in the rotation mode, the virtual camera 1 can be rotated clockwise (the virtual space 2 is rotated counterclockwise). On the other hand, if the user performs an operation on the controller 300 to extend the operation object 27 to the left in the rotation mode, the virtual camera 1 can be rotated counterclockwise (the virtual space 2 is rotated clockwise).

  The controller 300 may include a right controller that the user has with the right hand and a left controller that the user has with the left hand. In this case, the control circuit unit 200 generates the user's virtual right hand in the virtual space 2 based on the detection value of the position and inclination of the right controller and the detection result of the user's pressing operation on each button included in the right controller. be able to. Similarly, the control circuit unit 200 generates the virtual left hand of the user in the virtual space 2 based on the detection value of the position and inclination of the left controller and the detection result of the user's pressing operation on each button included in the left controller. be able to. The virtual right hand and the virtual left hand are both objects.

  In this aspect, the user can select and operate the operation object 27 by acting on the operation object 27 with the virtual right hand or the virtual left hand. For example, when the user moves the right hand so that the virtual right hand contacts the operation object 27, the operation object control unit 234 detects that the operation object 27 has been selected by the user. Thereafter, the user performs an operation for pinching the operation object 27 with the virtual right hand on the controller 300 (for example, pressing any button of the right controller) and drags the operation object 27 in a predetermined direction with the virtual right hand. When the right hand is moved in such a manner, the operation object control unit 234 specifies the extension direction and the extension amount of the operation object 27 based on the detected values of the position of the right controller before and after the right hand is moved. The extension method of the operation object 27 and the rotation method of the virtual camera 1 (or the virtual space 2) after the extension direction and the extension amount are determined are the same as in the above-described example.

  In the above example, the user can rotate the virtual camera 1 or the virtual space 2 in a desired direction by operating the right controller without moving the HMD 110 and without changing the line of sight. For example, in the rotation mode, if the user operates the right controller so as to drag the operation object 27 in the right direction with the virtual right hand, the virtual camera 1 can be rotated clockwise (the virtual space 2 is rotated counterclockwise). . On the other hand, in the rotation mode, if the user operates the right controller so as to drag the operation object 27 to the left with the virtual right hand, the virtual camera 1 can be rotated counterclockwise (the virtual space 2 is rotated clockwise). In these cases, the user can rotate the virtual camera 1 or the virtual space 2 more intuitively.

(Rotation control by functional section)
A functional partition may be defined at any position on the celestial sphere constituting the virtual space 2 where the platform is developed. The functional partition is an area to which a predetermined function can be assigned. Hereinafter, an example in which the control circuit unit 200 shifts to the rotation mode when the user selects a functional section will be described. In the following example, a function for detecting whether or not the user's line of sight (viewing direction or line-of-sight direction N0) hits the functional section is assigned to the functional section. Further, a function for detecting the time when the user's line of sight hits the function section is also assigned to the function section.

  When the line-of-sight management unit 232 detects that the user's line of sight (the reference line of sight 5 or the line-of-sight direction N0) hits the functional section in the normal mode for a specified time or longer, the line-of-sight management unit 232 detects that the user has selected the functional section. Thereby, the control circuit unit 200 shifts to the rotation mode. Thereafter, the control circuit unit 200 starts to rotate the virtual camera 1 or the virtual space 2 based on some operation by the user. For example, the control circuit unit 200 determines whether or not the user's line of sight has deviated from the functional section selected by the user by gaze for a specified time or longer. When it is determined that the user's line of sight has deviated from the functional section, the control circuit unit 200 identifies the direction in which the line of sight has deviated. Then, the direction in which the line of sight is deviated is specified as the movement direction of the line of sight, and the virtual camera 1 or the virtual space 2 is rotated in the rotation direction based on the specified movement direction of the line of sight. If the functional partition is used as in this example, even when the virtual space 2 in which the operation object 27 cannot be defined is provided to the user, the virtual camera 1 or the virtual space 2 can be rotated based on the user's operation. it can.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims. Embodiments obtained by appropriately combining technical means disclosed in different embodiments are also included in the technical scope of the present invention. A new technical feature can also be formed by combining the technical means disclosed in each embodiment.

[Example of software implementation]
A control block (detection unit 210, display control unit 220, virtual space control unit 230, storage unit 240, and communication unit 250) of the control circuit unit 200 is a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. ) Or by software using a CPU (Central Processing Unit).

  In the latter case, the control block includes a CPU that executes instructions of a program that is software that realizes each function, a ROM (Read Only Memory) or a memory in which the program and various data are recorded so as to be readable by a computer (or CPU). A device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like are provided. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[Additional Notes]
(Item 1) A method of providing a virtual space to a user wearing a head-mounted display (hereinafter referred to as HMD), and in the first mode, a virtual camera roll arranged in the virtual space and interlocked with the HMD In the second mode, at least a direction of rotation is specified based on the user's operation, and the virtual camera or the virtual space is detected while the operation continues to be detected. Updating the field-of-view image displayed on the HMD while continuing to rotate in the rotation direction. The operability in the virtual space can be improved.

(Item 2)
In the step of displaying the view image on the HMD, the view image including the operation object arranged in the virtual space is displayed on the HMD, and the operation object is selected by the user in the first mode. The method according to item 1, further comprising a step of detecting a transition to the second mode after detecting that the operation object has been selected in the first mode. It is possible to shift to the second mode according to the user's intention.

  (Item 3) In the second mode, after it is detected that the operation object is selected, a step of specifying a predetermined direction designated by the user for the operation object; and in the second mode, The method according to item 2, further comprising the step of elastically deforming the operation object in the predetermined direction, wherein the rotation direction is determined based on the predetermined direction in the step of determining the rotation direction. The virtual camera or the virtual space can be rotated based on a user operation on the operation object.

(Item 4)
4. The method according to item 3, wherein in the step of determining the rotation direction, the rotation direction is determined based on a horizontal component in the predetermined direction. It is possible to prevent the user from getting drunk in the virtual space.

(Item 5)
In the second mode, detecting that the designation of the predetermined direction to the operation object has been released; and in the second mode, detecting that the designation of the predetermined direction has been released. A step of returning the operation object to the original shape and canceling the selection of the operation object; and in the second mode, when it is detected that the designation of the predetermined direction is released, The method according to item 4, further comprising the step of stopping rotation of the camera or the virtual space and shifting to the first mode. The user can stop the rotation of the virtual camera or the virtual space with a simple operation.

(Item 6)
In the second mode, after detecting that the operation object is selected, in the step of specifying the amount of change in the inclination of the HMD in the reference coordinate system and the step of specifying the predetermined direction, The method according to any one of items 3 to 5, wherein the predetermined direction is specified based on a change amount. The user can adjust the rotation speed method of the virtual camera or the virtual space by changing the direction in which the HMD is tilted.

(Item 7)
In the step of detecting that the designation of the predetermined direction has been canceled, the inclination of the HMD after the rotation of the virtual camera or virtual space is started is detected when the operation object is selected. The method according to Item 6, wherein when the inclination of the HMD coincides, it is detected that the designation of the predetermined direction is cancelled. The user can stop the rotation of the virtual camera or the virtual space with a simple operation.

(Item 8)
In the second mode, after detecting that the operation object is selected, a step of specifying a predetermined amount designated for the operation object by the user; and in the second mode, the predetermined amount And a step of determining a rotation speed when rotating the virtual camera or the virtual space, and in the step of elastically deforming the operation object, the operation object is expanded by the predetermined amount, and the view image The method according to any one of items 3 to 7, wherein in the step of updating, the virtual camera or the virtual space is continuously rotated at the rotation speed. The user can rotate the virtual camera or the virtual space at a desired rotation speed.

(Item 9)
In the second mode, when the predetermined amount exceeds a threshold value, the step of returning the operation object to the original shape and canceling the selection of the operation object; and in the second mode, the predetermined amount When the threshold value exceeds a threshold, the step of stopping the rotation of the virtual camera or the virtual space and the second mode, the rotation of the virtual camera or the virtual space is stopped, and then the mode is shifted to the first mode. The method of item 8, further comprising the step of: The user can stop the rotation of the virtual camera or the virtual space with a simple operation.

(Item 10)
In the second mode, after detecting that the operation object is selected, the method further includes a step of specifying a change amount of the inclination of the HMD in a reference coordinate system, and the step of specifying the predetermined amount The method according to item 8 or 9, wherein the predetermined amount is specified based on a change amount of the inclination. The user can adjust the rotation speed method of the virtual camera or the virtual space by changing the degree of tilting the HMD.

(Item 11)
In the first mode, detecting that at least a part of the operation object is located outside the field of view of the virtual camera; in the first mode, at least a part of the operation object is: The method according to any one of items 2 to 10, further comprising a step of moving the operation object into the visual field area when it is detected that the object is located outside the visual field area. This makes it easier for users to find objects.

(Item 12)
Detecting that a user has selected a functional partition defined in the virtual space in the first mode; and a user selecting a functional partition defined in the virtual space in the first mode. The method of item 1, further comprising the step of transitioning to the second mode if detected. Even when a virtual space in which an object cannot be used is provided, the virtual camera or the virtual space can be rotated based on a user operation.

(Item 13)
A program for causing a computer to execute each step of the method according to any one of items 1 to 12.

(Item 14)
A computer-readable recording medium on which the program of item 13 is recorded.

1 virtual camera, 2 virtual space, 5 reference line of sight, 21 center, 22 virtual space image, 23 view area, 24 first area, 25 second area, 26 view image, 100 HMD system, 110 HMD, 112 display, 114 sensor 120 HMD sensor, 130 gaze sensor, 200 control circuit unit, 210 detection unit, 211 HMD detection unit, 212 gaze detection unit, 213 operation reception unit, 220 display control unit, 221 virtual camera control unit, 222 visual field region determination unit, 223 Visibility image generation unit, 230 Virtual space control unit, 231 Virtual space definition unit, 231 Gaze management unit, 232 Content specification unit, 233 Operation object control unit, Storage unit, 241 Model storage unit, 242 Content storage unit, 250 Communication unit 300 controller

Claims (12)

A method of providing a virtual space to a user wearing a head mounted display (hereinafter, HMD),
In the first mode, causing the HMD to display a field-of-view image based on a roll direction of a virtual camera disposed in the virtual space and interlocked with the HMD;
In the second mode, at least the rotation direction is specified based on the user's operation, and the field of view is displayed on the HMD while continuously rotating the virtual camera or the virtual space in the rotation direction while the operation is continuously detected. Updating the image, and
In the step of displaying the visual field image on the HMD, the visual field image including the operation object arranged in the virtual space is displayed on the HMD,
Detecting that the operation object is selected by the user in the first mode;
After detecting that the operation object is selected in the first mode, the step of transitioning to the second mode;
After detecting that the operation object is selected in the second mode, further comprising the step of specifying a predetermined direction designated by the user as the operation object;
In the step of updating the view image, the rotation direction is specified based on the predetermined direction ,
The method further comprising the step of elastically deforming the operation object in the predetermined direction in the second mode .   The method according to claim 1, wherein in the step of updating the view image, the rotation direction is specified based on a horizontal component in the predetermined direction. Detecting in the second mode that the designation of the predetermined direction to the operation object has been canceled;
In the second mode, when it is detected that the designation of the predetermined direction has been canceled, the operation object is returned to the original shape and changed to cancel the selection of the operation object;
In the second mode, when it is detected that the designation of the predetermined direction has been canceled, the method further includes the step of stopping the rotation of the virtual camera or the virtual space and shifting to the first mode. The method according to claim 2. In the second mode, after detecting that the operation object is selected, specifying a change amount of the inclination of the HMD in a reference coordinate system;
The method according to claim 1, wherein in the step of specifying the predetermined direction, the predetermined direction is specified based on a change amount of the inclination.   In the step of detecting that the designation of the predetermined direction has been canceled, the inclination of the HMD after the rotation of the virtual camera or virtual space is started is detected when the operation object is selected. The method according to claim 4, further comprising detecting that the designation of the predetermined direction is canceled when the inclination of the HMD coincides. In the second mode, after detecting that the operation object has been selected, specifying a predetermined amount designated for the operation object by the user;
In the second mode, further comprising the step of specifying a rotation speed when rotating the virtual camera or the virtual space based on the predetermined amount;
In the step of elastically deforming the operation object, the operation object is extended by the predetermined amount,
Wherein in the step of updating the view field image, continue the virtual camera or a virtual space is rotated at the rotational speed, The method according to claim 1. In the second mode, when the predetermined amount exceeds a threshold, returning the operation object to an original shape, and deselecting the operation object;
Stopping the rotation of the virtual camera or the virtual space when the predetermined amount exceeds a threshold in the second mode;
The method according to claim 6 , further comprising the step of transitioning to the first mode after rotation of the virtual camera or the virtual space is stopped in the second mode. In the second mode, after detecting that the operation object is selected, specifying a change amount of the inclination of the HMD in a reference coordinate system;
The method according to claim 6 or 7 , wherein, in the step of specifying the predetermined amount, the predetermined amount is specified based on a change amount of the slope. In the first mode, detecting that at least a part of the operation object is located outside the field of view of the virtual camera;
In the first mode, when it is detected that at least a part of the operation object is located outside the visual field area, the operation object further includes a step of moving the operation object into the visual field area. Item 9. The method according to any one of Items 1 to 8 . Detecting that a user has selected a functional partition defined in the virtual space in the first mode;
The method according to claim 1, further comprising the step of transitioning to the second mode when it is detected in the first mode that a user has selected a functional partition defined in the virtual space. The program which makes a computer perform each step of the method of any one of Claims 1-10 . The computer-readable recording medium which recorded the program of Claim 11 .

Images