JP2023099570A - program and system - Google Patents

Abstract

To improve operability for visibility control.SOLUTION: A program causes a computer to execute: definition means of defining a virtual space containing a virtual viewpoint; video reproducing means of reproducing a video in the virtual space; movement detection means of detecting movement of a controller held by a user; receiving means of receiving, from the controller, an input for enabling control of reproduction of the video in accordance with the movement of the controller; reproduction control means of controlling the reproduction of the video according to the movement of the controller while the input for enabling the control is being received; and display control means of displaying a visual field image corresponding to a visual field viewed from the virtual viewpoint, on a device mounted on the user.SELECTED DRAWING: Figure 15

Description

The present disclosure relates to programs and systems.

In the virtual space provided to the user by the head-mounted display, the user's field of view is usually controlled by the movement of the head-mounted display. .).

Japanese Patent No. 5869177

The user may get sick during the virtual experience, and control of the field of view is required to reduce the sickness. Also, when using an external controller, an operation method that does not impair the sense of immersion in the virtual experience is required. Therefore, in order to improve the user's virtual experience, there is room for improvement as to how to control the user's field of view in the virtual space.

An object of the present disclosure is to improve the operability of view control.

According to one aspect of the present disclosure, a program comprises: definition means for defining a virtual space including a virtual viewpoint; video playback means for playing back a video in the virtual space; motion for detecting motion of a controller held by a user; detecting means; receiving means for receiving an input from the controller to enable control of reproduction of the moving image in accordance with the movement of the controller; and a display control unit for displaying a view image corresponding to the view from the virtual viewpoint on the device worn by the user.

According to the present disclosure, it is possible to improve the operability of view control.

1 is a schematic representation of the configuration of an HMD system according to an embodiment; FIG. It is a block diagram showing an example of the hardware constitutions of the computer according to an embodiment. FIG. 4 is a diagram conceptually representing a uvw viewing coordinate system set in an HMD according to an embodiment; FIG. 2 is a diagram conceptually representing one aspect of representing a virtual space according to an embodiment; FIG. 2 is a top view of a user's head wearing an HMD according to an embodiment; It is a figure showing the YZ cross section which looked at the field-of-view area from the X direction in virtual space. It is a figure showing the XZ cross section which looked at the field-of-view area from the Y direction in virtual space. FIG. 2 is a diagram representing a schematic configuration of a controller according to an embodiment; FIG. FIG. 4 illustrates an example of yaw, roll, and pitch directions defined for a user's right hand according to an embodiment; It is a block diagram showing an example of a hardware configuration of a server according to an embodiment. 1 is a block diagram representing a computer as a modular configuration according to an embodiment; FIG. 4 is a sequence chart representing part of the processing performed in the HMD set according to one embodiment; FIG. 2 is a schematic diagram showing a situation in which each HMD provides a virtual space to a user in a network; It is a figure which shows the user's 5A view image in FIG. 12(A). FIG. 4 is a sequence diagram showing processing performed in the HMD system according to one embodiment; 3 is a block diagram showing the detailed configuration of the modules of the computer according to one embodiment; FIG. 4 is a flowchart illustrating a process for controlling gaze direction according to one embodiment. FIG. 10 is a diagram illustrating an example of control of the line-of-sight direction according to the movement of the controller; FIG. 10 is a diagram illustrating an example of control of the line-of-sight direction according to the movement of the controller; FIG. 17 is a diagram showing an example of a field-of-view image in the state of FIG. 16; FIG. 18 is a diagram showing an example of a field-of-view image in the state of FIG. 17; 4 is a flow chart illustrating the process of controlling playback of moving images according to one embodiment. FIG. 10 is a diagram showing an example of a field-of-view image when controlling playback of a moving image; 4 is a flowchart illustrating a process for controlling gaze direction according to one embodiment. 4 is a flow chart illustrating an example of a field of view image displayed according to one embodiment. 4 is a flow chart illustrating an example of a field of view image displayed according to one embodiment. 4 is a flow chart illustrating an example of a field of view image displayed according to one embodiment.

Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. In one or more of the embodiments presented in this disclosure, elements of each embodiment may be combined with each other and the combined result shall also form part of the embodiments presented in this disclosure.

[Configuration of HMD system]
The configuration of an HMD (Head-Mounted Device) system 100 will be described with reference to FIG. FIG. 1 is a diagram showing a schematic configuration of an HMD system 100 according to this embodiment. The HMD system 100 is provided as a system for home use or as a system for business use.

HMD system 100 includes server 600 , HMD sets 110 A, 110 B, 110 C, and 110 D, external device 700 , and network 2 . Each of HMD sets 110A, 110B, 110C, and 110D is configured to communicate with server 600 and external device 700 via network 2 . Hereinafter, the HMD sets 110A, 110B, 110C, and 110D are collectively referred to as the HMD set 110 as well. The number of HMD sets 110 configuring the HMD system 100 is not limited to four, and may be three or less or five or more. HMD set 110 includes HMD 120 , computer 200 , HMD sensor 410 , display 430 and controller 300 . HMD 120 includes monitor 130 , gaze sensor 140 , first camera 150 , second camera 160 , microphone 170 , and speaker 180 . Controller 300 may include motion sensor 420 .

In one aspect, computer 200 is connectable to network 2 such as the Internet, and is capable of communicating with server 600 and other computers connected to network 2 . Other computers include, for example, computers of other HMD sets 110 and external devices 700 . In another aspect, HMD 120 may include sensor 190 instead of HMD sensor 410 .

The HMD 120 can be worn on the head of the user 5 and provide the user 5 with a virtual space during operation. More specifically, HMD 120 displays a right-eye image and a left-eye image on monitor 130 . When each eye of the user 5 views the respective image, the user 5 can perceive the image as a three-dimensional image based on the parallax of both eyes. The HMD 120 may include both a so-called head-mounted display having a monitor and a head-mounted device on which a smartphone or other terminal having a monitor can be attached.

The monitor 130 is implemented as, for example, a non-transmissive display device. In one aspect, the monitor 130 is arranged on the main body of the HMD 120 so as to be positioned in front of both eyes of the user 5 . Therefore, the user 5 can immerse himself in the virtual space by viewing the three-dimensional image displayed on the monitor 130 . In one aspect, the virtual space includes images of, for example, a background, objects that the user 5 can manipulate, and menus that the user 5 can select. In one aspect, the monitor 130 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smart phone or other information display terminal.

In another aspect, monitor 130 may be implemented as a transmissive display. In this case, the HMD 120 may be an open type, such as a glasses type, instead of a closed type that covers the eyes of the user 5 as shown in FIG. Transmissive monitor 130 may be temporarily configurable as a non-transmissive display by adjusting its transmittance. The monitor 130 may include a configuration for simultaneously displaying a portion of the image forming the virtual space and the real space. For example, the monitor 130 may display an image of the real space captured by a camera mounted on the HMD 120, or may make the real space visible by setting a partial transmittance high.

In one aspect, monitor 130 may include a sub-monitor for displaying images for the right eye and a sub-monitor for displaying images for the left eye. In another aspect, monitor 130 may be configured to display the image for the right eye and the image for the left eye as one. In this case, monitor 130 includes a high speed shutter. The high speed shutter operates to alternately display the right eye image and the left eye image so that the image is perceived by only one eye.

In one aspect, the HMD 120 includes multiple light sources (not shown). Each light source is implemented by, for example, an LED (Light Emitting Diode) that emits infrared rays. HMD sensor 410 has a position tracking function for detecting movement of HMD 120 . More specifically, HMD sensor 410 reads a plurality of infrared rays emitted by HMD 120 and detects the position and tilt of HMD 120 in the physical space.

In another aspect, HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the HMD 120 by executing image analysis processing using the image information of the HMD 120 output from the camera.

In another aspect, HMD 120 may include sensor 190 instead of HMD sensor 410 or in addition to HMD sensor 410 as a position detector. HMD 120 can detect the position and tilt of HMD 120 using sensor 190 . For example, if sensor 190 is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, HMD 120 can detect its own position and tilt using any of these sensors instead of HMD sensor 410 . As an example, when the sensor 190 is an angular velocity sensor, the angular velocity sensor temporally detects angular velocities around three axes of the HMD 120 in real space. The HMD 120 calculates temporal changes in angles around the three axes of the HMD 120 based on the angular velocities, and further calculates the tilt of the HMD 120 based on the temporal changes in the angles.

Gaze sensor 140 detects the directions in which the user's 5 right and left eyes are directed. That is, the gaze sensor 140 detects the line of sight of the user 5 . Detection of the line-of-sight direction is achieved by, for example, a known eye-tracking function. Gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right and left eyes of the user 5 with infrared light and receives reflected light from the cornea and iris of the irradiated light, thereby detecting the rotation angle of each eyeball. . The gaze sensor 140 can detect the line of sight of the user 5 based on each detected rotation angle.

The first camera 150 photographs the lower part of the user's 5 face. More specifically, first camera 150 captures the nose, mouth, and the like of user 5 . The second camera 160 photographs the eyes, eyebrows, etc. of the user 5 . The housing of the HMD 120 on the side of the user 5 is defined as the inside of the HMD 120 , and the housing of the HMD 120 on the side opposite to the user 5 is defined as the outside of the HMD 120 . In one aspect, first camera 150 may be positioned outside HMD 120 and second camera 160 may be positioned inside HMD 120 . Images generated by first camera 150 and second camera 160 are input to computer 200 . In another aspect, the first camera 150 and the second camera 160 may be realized as one camera, and the face of the user 5 may be photographed with this one camera.

The microphone 170 converts the speech of the user 5 into an audio signal (electrical signal) and outputs it to the computer 200 . The speaker 180 converts the audio signal into audio and outputs it to the user 5 . In another aspect, HMD 120 may include earphones instead of speaker 180 .

The controller 300 is connected to the computer 200 by wire or wirelessly. The controller 300 accepts command input from the user 5 to the computer 200 . In one aspect, controller 300 is configured to be grippable by user 5 . In another aspect, controller 300 is configured to be attachable to part of user's 5 body or clothing. In yet another aspect, controller 300 may be configured to output at least one of vibration, sound, and light based on signals transmitted from computer 200 . In yet another aspect, controller 300 receives an operation from user 5 for controlling the position and movement of an object placed in the virtual space.

In one aspect, controller 300 includes multiple light sources. Each light source is implemented, for example, by an LED that emits infrared rays. The HMD sensor 410 has a position tracking function. In this case, the HMD sensor 410 reads multiple infrared rays emitted by the controller 300 and detects the position and tilt of the controller 300 in the physical space. In another aspect, HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the controller 300 by executing image analysis processing using the image information of the controller 300 output from the camera.

Motion sensor 420 , in one aspect, is attached to the hand of user 5 to detect movement of the hand of user 5 . For example, the motion sensor 420 detects hand rotation speed, number of rotations, and the like. The detected signal is sent to computer 200 . Motion sensor 420 is provided in controller 300, for example. In one aspect, motion sensor 420 is provided, for example, in controller 300 configured to be grippable by user 5 . In another aspect, for safety in the real space, the controller 300 is attached to an object such as a glove that is attached to the hand of the user 5 so as not to fly off easily. In yet another aspect, sensors not worn by user 5 may detect movement of user's 5 hand. For example, a signal from a camera that captures the image of the user 5 may be input to the computer 200 as a signal representing the motion of the user 5 . Motion sensor 420 and computer 200 are, for example, wirelessly connected to each other. In the case of wireless communication, the form of communication is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

Display 430 displays an image similar to the image displayed on monitor 130 . This allows users other than the user 5 wearing the HMD 120 to view the same image as that of the user 5 . The image displayed on display 430 does not have to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 430 include a liquid crystal display and an organic EL monitor.

Server 600 may transmit programs to computer 200 . In another aspect, server 600 may communicate with other computers 200 to provide virtual reality to HMDs 120 used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's action with the other computers 200 via the server 600 to generate a plurality of games in the same virtual space. users to enjoy common games. Each computer 200 may communicate signals based on each user's actions with other computers 200 without going through the server 600 .

The external device 700 may be any device as long as it can communicate with the computer 200 . The external device 700 may be, for example, a device capable of communicating with the computer 200 via the network 2, or may be a device capable of directly communicating with the computer 200 through short-range wireless communication or wired connection. Examples of the external device 700 include, but are not limited to, smart devices, PCs (Personal Computers), and peripherals of the computer 200 .

[Computer hardware configuration]
A computer 200 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of computer 200 according to this embodiment. The computer 200 includes a processor 210, a memory 220, a storage 230, an input/output interface 240, and a communication interface 250 as main components. Each component is connected to bus 260 respectively.

Processor 210 executes a series of instructions contained in a program stored in memory 220 or storage 230 based on a signal given to computer 200 or based on the establishment of a predetermined condition. In one aspect, processor 210 is implemented as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), MPU (Micro Processor Unit), FPGA (Field-Programmable Gate Array), or other device.

Memory 220 temporarily stores programs and data. The program is loaded from storage 230, for example. Data includes data input to computer 200 and data generated by processor 210 . In one aspect, memory 220 is implemented as random access memory (RAM) or other volatile memory.

Storage 230 permanently holds programs and data. The storage 230 is implemented as, for example, a ROM (Read-Only Memory), hard disk device, flash memory, or other non-volatile storage device. The programs stored in the storage 230 include programs for providing a virtual space in the HMD system 100, simulation programs, game programs, user authentication programs, and programs for realizing communication with other computers 200. The data stored in the storage 230 includes data and objects for defining the virtual space.

In another aspect, storage 230 may be implemented as a removable storage device such as a memory card. In still another aspect, instead of storage 230 built into computer 200, a configuration using programs and data stored in an external storage device may be used. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used, such as an amusement facility, it is possible to collectively update programs and data.

Input/output interface 240 communicates signals between HMD 120 , HMD sensor 410 , motion sensor 420 and display 430 . Monitor 130 , gaze sensor 140 , first camera 150 , second camera 160 , microphone 170 and speaker 180 included in HMD 120 can communicate with computer 200 via input/output interface 240 of HMD 120 . In one aspect, the input/output interface 240 is implemented using a USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or other terminals. Input/output interface 240 is not limited to the one described above.

In some aspects, input/output interface 240 may also communicate with controller 300 . For example, input/output interface 240 receives signals output from controller 300 and motion sensor 420 . In another aspect, input/output interface 240 sends instructions output from processor 210 to controller 300 . The command instructs the controller 300 to vibrate, output sound, emit light, or the like. Upon receiving the command, the controller 300 performs any one of vibration, sound output, or light emission according to the command.

The communication interface 250 is connected to the network 2 and communicates with other computers connected to the network 2 (for example, the server 600). In one aspect, the communication interface 250 is implemented as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication) or other wireless communication interface. be done. Communication interface 250 is not limited to the one described above.

In one aspect, processor 210 accesses storage 230, loads one or more programs stored in storage 230 into memory 220, and executes the sequences of instructions contained in the programs. The one or more programs may include an operating system of computer 200, an application program for providing virtual space, game software executable in virtual space, and the like. Processor 210 sends a signal for providing virtual space to HMD 120 via input/output interface 240 . HMD 120 displays an image on monitor 130 based on the signal.

In the example shown in FIG. 2, computer 200 is provided outside HMD 120, but computer 200 may be built into HMD 120 in another aspect. As an example, a portable information communication terminal (for example, a smart phone) including monitor 130 may function as computer 200 .

The computer 200 may have a configuration that is commonly used by a plurality of HMDs 120 . According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so each user can enjoy the same application as other users in the same virtual space.

In one embodiment, in the HMD system 100, a real coordinate system, which is a coordinate system in real space, is set in advance. The real coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction perpendicular to the vertical direction, and the front-rear direction perpendicular to both the vertical and horizontal directions. The horizontal direction, vertical direction (vertical direction), and front-rear direction in the real coordinate system are defined as x-axis, y-axis, and z-axis, respectively. More specifically, in the real coordinate system, the x-axis is parallel to the horizontal direction of the real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the physical space.

In one aspect, HMD sensor 410 includes an infrared sensor. The presence of HMD 120 is detected when the infrared sensor detects infrared rays emitted from each light source of HMD 120 . The HMD sensor 410 further detects the position and inclination (orientation) of the HMD 120 in the physical space according to the movement of the user 5 wearing the HMD 120, based on the value of each point (each coordinate value in the real coordinate system). do. More specifically, the HMD sensor 410 can detect temporal changes in the position and tilt of the HMD 120 using each value detected over time.

Each tilt of HMD 120 detected by HMD sensor 410 corresponds to each tilt of HMD 120 around three axes in the real coordinate system. The HMD sensor 410 sets the uvw visual field coordinate system to the HMD 120 based on the tilt of the HMD 120 in the real coordinate system. The uvw visual field coordinate system set in the HMD 120 corresponds to the viewpoint coordinate system when the user 5 wearing the HMD 120 sees an object in virtual space.

[uvw visual field coordinate system]
The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw viewing coordinate system set in the HMD 120 according to one embodiment. HMD sensor 410 detects the position and tilt of HMD 120 in the real coordinate system when HMD 120 is activated. Processor 210 sets the uvw viewing coordinate system to HMD 120 based on the detected values.

As shown in FIG. 3 , the HMD 120 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 5 wearing the HMD 120 . More specifically, the HMD 120 rotates the horizontal direction, the vertical direction, and the front-rear direction (x-axis, y-axis, and z-axis) defining the real coordinate system by tilts around each axis of the HMD 120 within the real coordinate system. The three directions newly obtained by tilting around the axes are set as the pitch axis (u-axis), yaw-axis (v-axis), and roll-axis (w-axis) of the uvw visual field coordinate system in the HMD 120 .

In one aspect, when user 5 wearing HMD 120 is standing upright and looking straight ahead, processor 210 sets HMD 120 to a uvw viewing coordinate system that is parallel to the real coordinate system. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the real coordinate system are the pitch axis (u-axis) and yaw-axis (v-axis) of the uvw visual field coordinate system in the HMD 120. , and the roll axis (w-axis).

After the uvw visual field coordinate system is set on the HMD 120 , the HMD sensor 410 can detect the tilt of the HMD 120 in the set uvw visual field coordinate system based on the movement of the HMD 120 . In this case, the HMD sensor 410 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 120 in the uvw visual field coordinate system as the tilt of the HMD 120 . The pitch angle (θu) represents the tilt angle of the HMD 120 around the pitch axis in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 120 around the yaw axis in the uvw visual field coordinate system. A roll angle (θw) represents the tilt angle of the HMD 120 around the roll axis in the uvw visual field coordinate system.

The HMD sensor 410 sets the uvw visual field coordinate system in the HMD 120 after the HMD 120 moves based on the detected inclination of the HMD 120 to the HMD 120 . The relationship between the HMD 120 and the uvw visual field coordinate system of the HMD 120 is always constant regardless of the position and tilt of the HMD 120 . When the position and tilt of the HMD 120 change, the position and tilt of the uvw visual field coordinate system of the HMD 120 in the real coordinate system change in conjunction with the change in the position and tilt.

In one aspect, HMD sensor 410 detects HMD 120 based on the infrared light intensity and the relative positional relationship between a plurality of points (for example, the distance between points) obtained based on the output from the infrared sensor. position in the physical space may be specified as a relative position with respect to the HMD sensor 410 . Processor 210 may determine the origin of the uvw visual field coordinate system of HMD 120 in physical space (real coordinate system) based on the specified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually showing one aspect of expressing virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure covering the entire 360-degree direction of the center 12 . In FIG. 4, the celestial sphere in the upper half of the virtual space 11 is illustrated in order not to complicate the explanation. Each mesh is defined in the virtual space 11 . The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system, which is the global coordinate system defined in the virtual space 11 . The computer 200 associates each partial image that constitutes the panorama image 13 (still image, moving image, etc.) that can be developed in the virtual space 11 with each corresponding mesh in the virtual space 11 .

In one aspect, the virtual space 11 defines an XYZ coordinate system with the center 12 as the origin. The XYZ coordinate system is parallel to the real coordinate system, for example. The horizontal direction, vertical direction (vertical direction), and front-rear direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the The Z-axis (front-rear direction) is parallel to the z-axis of the real coordinate system.

When the HMD 120 is activated, that is, in the initial state of the HMD 120 , the virtual camera 14 is arranged at the center 12 of the virtual space 11 . In one aspect, processor 210 displays an image captured by virtual camera 14 on monitor 130 of HMD 120 . The virtual camera 14 similarly moves in the virtual space 11 in conjunction with the movement of the HMD 120 in the real space. Thereby, changes in the position and tilt of the HMD 120 in the real space can be similarly reproduced in the virtual space 11 .

A uvw field-of-view coordinate system is defined in the virtual camera 14 as in the case of the HMD 120 . The uvw visual field coordinate system of the virtual camera 14 in the virtual space 11 is defined to interlock with the uvw visual field coordinate system of the HMD 120 in the real space (real coordinate system). Therefore, when the tilt of HMD 120 changes, the tilt of virtual camera 14 also changes accordingly. The virtual camera 14 can also move in the virtual space 11 in conjunction with the movement of the user 5 wearing the HMD 120 in the real space.

The processor 210 of the computer 200 defines the field of view area 15 in the virtual space 11 based on the position and tilt of the virtual camera 14 (reference line of sight 16). The field of view area 15 corresponds to an area of the virtual space 11 that is visually recognized by the user 5 wearing the HMD 120 . That is, the position of the virtual camera 14 can be said to be the viewpoint of the user 5 in the virtual space 11 .

The line of sight of the user 5 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 5 visually recognizes an object. The uvw visual field coordinate system of the HMD 120 is equal to the viewpoint coordinate system when the user 5 visually recognizes the monitor 130 . The uvw visual field coordinate system of the virtual camera 14 is interlocked with the uvw visual field coordinate system of the HMD 120 . Therefore, the HMD system 100 according to one aspect can regard the line of sight of the user 5 detected by the gaze sensor 140 as the line of sight of the user 5 in the uvw field coordinate system of the virtual camera 14 .

[User's line of sight]
Determination of the line of sight of the user 5 will be described with reference to FIG. FIG. 5 is a top view of the head of user 5 wearing HMD 120 according to an embodiment.

In one aspect, gaze sensor 140 detects the line of sight of user 5's right and left eyes. In one aspect, when user 5 is looking near, gaze sensor 140 detects lines of sight R1 and L1. In another aspect, when user 5 is looking far away, gaze sensor 140 detects lines of sight R2 and L2. In this case, the angle between the lines of sight R2 and L2 with respect to the roll axis w is smaller than the angle between the lines of sight R1 and L1 with respect to the roll axis w. Gaze sensor 140 transmits the detection result to computer 200 .

When computer 200 receives detection values of lines of sight R1 and L1 from gaze sensor 140 as a detection result of lines of sight, computer 200 identifies point of gaze N1, which is the intersection of lines of sight R1 and L1, based on the detected values. On the other hand, when computer 200 receives detection values of lines of sight R2 and L2 from gaze sensor 140, computer 200 identifies the intersection of lines of sight R2 and L2 as the point of gaze. The computer 200 identifies the line of sight N0 of the user 5 based on the identified position of the gaze point N1. The computer 200 detects, for example, the extending direction of a straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 5 and the gaze point N1 as the line of sight N0. The line of sight N0 is the direction in which the user 5 is actually looking with both eyes. The line of sight N<b>0 corresponds to the direction in which the user 5 actually looks toward the field of view area 15 .

In another aspect, HMD system 100 may include a television broadcast reception tuner. With such a configuration, the HMD system 100 can display television programs in the virtual space 11 .

In still another aspect, the HMD system 100 may have a communication circuit for connecting to the Internet or a calling function for connecting to a telephone line.

[Vision area]
The field of view area 15 will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a diagram showing a YZ cross section of the visual field area 15 viewed from the X direction in the virtual space 11. As shown in FIG. FIG. 7 is a diagram showing an XZ cross section of the visual field area 15 viewed from the Y direction in the virtual space 11. As shown in FIG.

As shown in FIG. 6 , the field of view area 15 in the YZ cross section includes area 18 . A region 18 is defined by the position of the virtual camera 14 , the reference line of sight 16 and the YZ section of the virtual space 11 . The processor 210 defines an area 18 centered on the reference line of sight 16 in the virtual space and including the polar angle α.

As shown in FIG. 7 , the viewing area 15 in the XZ cross section includes area 19 . A region 19 is defined by the position of the virtual camera 14 , the reference line of sight 16 and the XZ section of the virtual space 11 . The processor 210 defines a range including the azimuth angle β around the reference line of sight 16 in the virtual space 11 as the region 19 . The polar angles α and β are determined according to the position of the virtual camera 14 and the tilt (orientation) of the virtual camera 14 .

In one aspect, HMD system 100 provides user 5 with a view in virtual space 11 by displaying view image 17 on monitor 130 based on a signal from computer 200 . The field-of-view image 17 is an image corresponding to a portion of the panoramic image 13 corresponding to the field-of-view area 15 . When the user 5 moves the HMD 120 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the viewing area 15 in the virtual space 11 changes. As a result, the field-of-view image 17 displayed on the monitor 130 is updated to an image of the panorama image 13 that is superimposed on the field-of-view area 15 in the direction the user 5 faces in the virtual space 11 . The user 5 can visually recognize a desired direction in the virtual space 11 .

Thus, the tilt of the virtual camera 14 corresponds to the line of sight (reference line of sight 16 ) of the user 5 in the virtual space 11 , and the position where the virtual camera 14 is arranged corresponds to the viewpoint of the user 5 in the virtual space 11 . Therefore, by changing the position or tilt of the virtual camera 14, the image displayed on the monitor 130 is updated and the field of view of the user 5 is moved.

While wearing the HMD 120, the user 5 can visually recognize only the panoramic image 13 developed in the virtual space 11 without visually recognizing the real world. Therefore, the HMD system 100 can give the user 5 a high sense of immersion in the virtual space 11 .

In one aspect, processor 210 can move virtual camera 14 in virtual space 11 in conjunction with movement of user 5 wearing HMD 120 in the physical space. In this case, processor 210 identifies an image area (viewing area 15 ) projected on monitor 130 of HMD 120 based on the position and tilt of virtual camera 14 in virtual space 11 .

In one aspect, virtual camera 14 may include two virtual cameras, a virtual camera for providing images for the right eye and a virtual camera for providing images for the left eye. Appropriate parallax is set for the two virtual cameras so that the user 5 can recognize the three-dimensional virtual space 11 . In another aspect, virtual camera 14 may be realized by one virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by one virtual camera. In the present embodiment, the virtual camera 14 includes two virtual cameras, and the roll axis (w) generated by synthesizing the roll axes of the two virtual cameras is adapted to the roll axis (w) of the HMD 120. The technical idea according to the present disclosure is exemplified as configured as follows.

[controller]
An example of the controller 300 will be described with reference to FIG. FIG. 8 is a diagram showing a schematic configuration of controller 300 according to an embodiment.

As shown in FIG. 8, in one aspect, controller 300 may include a right controller 300R and a left controller (not shown). Right controller 300R is operated by user 5's right hand. The left controller is operated with the user's 5 left hand. In one aspect, right controller 300R and left controller are symmetrically configured as separate devices. Therefore, the user 5 can freely move the right hand holding the right controller 300R and the left hand holding the left controller. In another aspect, controller 300 may be an integrated controller that accepts two-handed operation. The right controller 300R will be described below.

Right controller 300R includes grip 310 , frame 320 , and top surface 330 . The grip 310 is configured to be gripped by the user's 5 right hand. For example, grip 310 may be held by user 5's right palm and three fingers (middle finger, ring finger, little finger).

Grip 310 includes buttons 340 and 350 and motion sensor 420 . Button 340 is arranged on the side surface of grip 310 and is operated by the middle finger of the right hand. Button 350 is arranged on the front surface of grip 310 and is operated by the index finger of the right hand. In one aspect, buttons 340, 350 are configured as trigger buttons. Motion sensor 420 is built into the housing of grip 310 . The grip 310 may not include the motion sensor 420 if the motion of the user 5 can be detected from around the user 5 by a camera or other device.

Frame 320 includes a plurality of infrared LEDs 360 arranged along its circumference. Infrared LED 360 emits infrared light in accordance with the progress of a program using controller 300 during execution of the program. Infrared rays emitted from infrared LED 360 can be used to detect the positions and orientations (inclination and orientation) of right controller 300R and left controller. Although the example shown in FIG. 8 shows infrared LEDs 360 arranged in two rows, the number of arrays is not limited to that shown in FIG. A single row or three or more row arrays may be used.

Top surface 330 includes buttons 370 and 380 and analog stick 390 . Buttons 370 and 380 are configured as push buttons. Buttons 370 and 380 accept operations by user 5's right thumb. In a certain aspect, the analog stick 390 accepts an operation in any direction of 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object placed in the virtual space 11 .

In one aspect, right controller 300R and left controller include batteries for powering infrared LEDs 360 and other components. Batteries include, but are not limited to, rechargeable, button-type, dry cell-type, and the like. In another aspect, right controller 300R and left controller may be connected to a USB interface of computer 200, for example. In this case, right controller 300R and left controller do not require batteries.

As shown in states (A) and (B) of FIG. 8, for example, the yaw, roll, and pitch directions are defined for the user's 5 right hand. When the user 5 extends the thumb and index finger, the direction in which the thumb extends is the yaw direction, the direction in which the index finger extends is the roll direction, and the direction perpendicular to the plane defined by the yaw direction axis and the roll direction axis is the pitch direction. defined as

[Server hardware configuration]
Server 600 according to the present embodiment will be described with reference to FIG. FIG. 9 is a block diagram showing an example hardware configuration of server 600 according to an embodiment. The server 600 includes a processor 610, a memory 620, a storage 630, an input/output interface 640, and a communication interface 650 as main components. Each component is respectively connected to bus 660 .

Processor 610 executes a series of instructions contained in a program stored in memory 620 or storage 630 based on a signal provided to server 600 or based on the establishment of a predetermined condition. In one aspect, processor 610 is implemented as a CPU, GPU, MPU, FPGA, or other device.

Memory 620 temporarily stores programs and data. The program is loaded from storage 630, for example. Data includes data input to server 600 and data generated by processor 610 . In one aspect, memory 620 is implemented as RAM or other volatile memory.

Storage 630 permanently holds programs and data. The storage 630 is implemented as, for example, a ROM, hard disk device, flash memory, or other non-volatile storage device. The programs stored in the storage 630 may include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with the computer 200. The data stored in the storage 630 may include data and objects for defining the virtual space.

In another aspect, storage 630 may be implemented as a removable storage device such as a memory card. In still another aspect, instead of storage 630 built into server 600, a configuration using programs and data stored in an external storage device may be used. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used, such as an amusement facility, it is possible to collectively update programs and data.

Input/output interface 640 communicates signals with input/output devices. In one aspect, input/output interface 640 is implemented using USB, DVI, HDMI, or other terminals. Input/output interface 640 is not limited to the one described above.

Communication interface 650 is connected to network 2 and communicates with computer 200 connected to network 2 . In one aspect, communication interface 650 is implemented as, for example, a LAN or other wired communication interface, or a WiFi, Bluetooth, NFC or other wireless communication interface. Communication interface 650 is not limited to those described above.

In one aspect, processor 610 accesses storage 630, loads one or more programs stored in storage 630 into memory 620, and executes the sequences of instructions contained in the programs. The one or more programs may include an operating system of server 600, an application program for providing virtual space, game software executable in virtual space, and the like. Processor 610 may send signals to computer 200 via input/output interface 640 to provide the virtual space.

[HMD control device]
A control device for the HMD 120 will be described with reference to FIG. In one embodiment, the controller is implemented by computer 200 having a well-known configuration. FIG. 10 is a block diagram representing a modular configuration of computer 200 according to one embodiment.

As shown in FIG. 10, computer 200 comprises control module 510 , rendering module 520 , memory module 530 and communication control module 540 . In one aspect, control module 510 and rendering module 520 are implemented by processor 210 . In another aspect, multiple processors 210 may act as control module 510 and rendering module 520 . Memory module 530 is implemented by memory 220 or storage 230 . Communication control module 540 is implemented by communication interface 250 .

A control module 510 controls the virtual space 11 provided to the user 5 . The control module 510 defines the virtual space 11 in the HMD system 100 using virtual space data representing the virtual space 11 . Virtual space data is stored, for example, in memory module 530 . The control module 510 may generate virtual space data or acquire virtual space data from the server 600 or the like.

The control module 510 places objects in the virtual space 11 using object data representing the objects. Object data is stored, for example, in memory module 530 . The control module 510 may generate object data or acquire object data from the server 600 or the like. The objects include, for example, an avatar object that is the alter ego of the user 5, a character object, an operation object such as a virtual hand operated by the controller 300, landscapes including forests, mountains, etc. arranged according to the progress of the story of the game, townscapes, and animals. etc.

The control module 510 places the avatar object of the user 5 of the other computer 200 connected via the network 2 in the virtual space 11 . In one aspect, control module 510 places user 5's avatar object in virtual space 11 . In one aspect, the control module 510 places an avatar object that resembles the user 5 in the virtual space 11 based on the image containing the user 5 . In another aspect, the control module 510 arranges in the virtual space 11 an avatar object that has been selected by the user 5 from among multiple types of avatar objects (for example, an object imitating an animal or a deformed human object). do.

Control module 510 identifies the tilt of HMD 120 based on the output of HMD sensor 410 . In another aspect, control module 510 identifies the tilt of HMD 120 based on the output of sensor 190, which functions as a motion sensor. The control module 510 detects the facial organs of the user 5 (eg, mouth, eyes, eyebrows) from the facial images of the user 5 generated by the first camera 150 and the second camera 160 . The control module 510 detects the motion (shape) of each detected organ.

The control module 510 detects the line of sight of the user 5 in the virtual space 11 based on the signal from the gaze sensor 140 . The control module 510 detects the viewpoint position (coordinate values in the XYZ coordinate system) where the detected line of sight of the user 5 and the celestial sphere of the virtual space 11 intersect. More specifically, the control module 510 detects the viewpoint position based on the line of sight of the user 5 defined by the uvw coordinate system and the position and tilt of the virtual camera 14 . Control module 510 transmits the detected viewpoint position to server 600 . In another aspect, control module 510 may be configured to transmit gaze information representing the gaze of user 5 to server 600 . In this case, the viewpoint position can be calculated based on the line-of-sight information received by the server 600 .

The control module 510 reflects the movement of the HMD 120 detected by the HMD sensor 410 on the avatar object. For example, the control module 510 detects that the HMD 120 is tilted, and tilts and arranges the avatar object. The control module 510 reflects the motion of the detected facial features on the face of the avatar object placed in the virtual space 11 . The control module 510 receives line-of-sight information of the other user 5 from the server 600 and reflects the line-of-sight information of the avatar object of the other user 5 . In one aspect, control module 510 reflects movements of controller 300 on avatar objects and operation objects. In this case, controller 300 includes a motion sensor, an acceleration sensor, a plurality of light emitting elements (for example, infrared LEDs), or the like for detecting movement of controller 300 .

The control module 510 arranges in the virtual space 11 an operation object for receiving an operation by the user 5 in the virtual space 11 . The user 5 operates an object placed in the virtual space 11, for example, by operating the operation object. In one aspect, the manipulation object may include, for example, a hand object, which is a virtual hand corresponding to the user's 5 hand. In one aspect, the control module 510 moves the hand object in the virtual space 11 based on the output of the motion sensor 420 in conjunction with the hand movement of the user 5 in the real space. In one aspect, the manipulation object may correspond to a hand portion of the avatar object.

The control module 510 detects a collision when each of the objects placed in the virtual space 11 collides with another object. The control module 510 can detect, for example, the timing at which the collision area of a certain object touches the collision area of another object, and performs predetermined processing when the detection is made. The control module 510 can detect the timing when the objects are separated from the touching state, and performs a predetermined process when the detection is made. The control module 510 can detect that objects are touching. For example, when an operation object and another object touch each other, the control module 510 detects that the operation object and the other object have touched each other, and performs predetermined processing.

In one aspect, control module 510 controls image display on monitor 130 of HMD 120 . For example, control module 510 places virtual camera 14 in virtual space 11 . The control module 510 controls the position of the virtual camera 14 in the virtual space 11 and the tilt (orientation) of the virtual camera 14 . The control module 510 defines the field of view area 15 according to the inclination of the head of the user 5 wearing the HMD 120 and the position of the virtual camera 14 . Rendering module 520 generates view image 17 displayed on monitor 130 based on determined view area 15 . The field-of-view image 17 generated by the rendering module 520 is output to the HMD 120 by the communication control module 540 .

When the control module 510 detects an utterance by the user 5 using the microphone 170 from the HMD 120, the control module 510 specifies the computer 200 to which the audio data corresponding to the utterance is to be transmitted. The audio data is sent to the computer 200 specified by control module 510 . When the control module 510 receives voice data from another user's computer 200 via the network 2 , the control module 510 outputs voice (utterance) corresponding to the voice data from the speaker 180 .

Memory module 530 holds data used by computer 200 to provide virtual space 11 to user 5 . In one aspect, memory module 530 holds spatial information, object information, and user information.

The spatial information holds one or more templates defined to provide the virtual space 11. FIG.

The object information includes a plurality of panoramic images 13 forming the virtual space 11 and object data for arranging objects in the virtual space 11 . Panorama image 13 may include still images and moving images. The panorama image 13 may include an image of unreal space and an image of real space. Images of unreal space include, for example, images generated by computer graphics.

User information holds a user ID that identifies the user 5 . The user ID may be, for example, an IP (Internet Protocol) address or a MAC (Media Access Control) address set in the computer 200 used by the user. In another aspect, the user ID can be set by the user. The user information includes programs and the like for causing the computer 200 to function as a control device for the HMD system 100 .

Data and programs stored in memory module 530 are input by user 5 of HMD 120 . Alternatively, processor 210 downloads a program or data from a computer (for example, server 600 ) operated by a content provider, and stores the downloaded program or data in memory module 530 .

Communication control module 540 can communicate with server 600 and other information communication devices via network 2 .

In one aspect, control module 510 and rendering module 520 may be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, control module 510 and rendering module 520 may be implemented as a combination of circuit elements that implement each process.

Processing in computer 200 is realized by hardware and software executed by processor 210 . Such software may be pre-stored on a hard disk or other memory module 530 . The software may be distributed as a program product stored in a CD-ROM or other computer-readable non-volatile data recording medium. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other network. Such software is read from a data recording medium by an optical disk drive or other data reading device, or downloaded from the server 600 or other computer via the communication control module 540, and then temporarily stored in the storage module. . The software is read from the storage module by processor 210 and stored in RAM in the form of executable programs. Processor 210 executes the program.

[Control Structure of HMD System]
The control structure of the HMD set 110 will be described with reference to FIG. FIG. 11 is a sequence chart representing part of the processing performed in HMD set 110 according to one embodiment.

As shown in FIG. 11, at step S1110, processor 210 of computer 200, as control module 510, identifies virtual space data and defines virtual space 11. As shown in FIG.

At step S1120, processor 210 initializes virtual camera . For example, the processor 210 places the virtual camera 14 at the predefined center 12 in the virtual space 11 in the work area of the memory, and directs the line of sight of the virtual camera 14 in the direction the user 5 is facing.

At step S1130, processor 210, as rendering module 520, generates view image data for displaying an initial view image. The generated visual field image data is output to HMD 120 by communication control module 540 .

In step S<b>1132 , monitor 130 of HMD 120 displays a field-of-view image based on the field-of-view image data received from computer 200 . The user 5 wearing the HMD 120 can recognize the virtual space 11 by viewing the visual field image.

In step S<b>1134 , HMD sensor 410 detects the position and tilt of HMD 120 based on a plurality of infrared rays emitted from HMD 120 . The detection result is output to the computer 200 as motion detection data.

In step S<b>1140 , processor 210 identifies the viewing direction of user 5 wearing HMD 120 based on the position and tilt included in the motion detection data of HMD 120 .

In step S1150, processor 210 executes the application program and arranges objects in virtual space 11 based on instructions included in the application program.

In step S 1160 , controller 300 detects an operation by user 5 based on the signal output from motion sensor 420 and outputs detection data representing the detected operation to computer 200 . In another aspect, manipulation of controller 300 by user 5 may be detected based on images from cameras placed around user 5 .

In step S<b>1170 , processor 210 detects an operation of controller 300 by user 5 based on the detection data acquired from controller 300 .

At step S 1180 , processor 210 generates view image data based on the operation of controller 300 by user 5 . The generated visual field image data is output to HMD 120 by communication control module 540 .

In step S<b>1190 , HMD 120 updates the field-of-view image based on the received field-of-view image data, and displays the updated field-of-view image on monitor 130 .

[Avatar object]
An avatar object according to this embodiment will be described with reference to FIGS. Hereafter, it is a figure explaining the avatar object of each user 5 of HMD set 110A, 110B. Hereinafter, the user of the HMD set 110A is referred to as the user 5A, the user of the HMD set 110B as the user 5B, the user of the HMD set 110C as the user 5C, and the user of the HMD set 110D as the user 5D. A is attached to the reference sign of each component regarding the HMD set 110A, B is attached to the reference sign of each component regarding the HMD set 110B, C is attached to the reference sign of each component regarding the HMD set 110C, and the HMD set A D is attached to the reference number of each component related to 110D. For example, HMD 120A is included in HMD set 110A.

FIG. 12A is a schematic diagram showing a situation in which each HMD 120 provides the virtual space 11 to the user 5 in the network 2. FIG. Computers 200A-200D provide users 5A-5D with virtual spaces 11A-11D via HMDs 120A-120D, respectively. In the example shown in FIG. 12A, virtual space 11A and virtual space 11B are configured with the same data. In other words, the computers 200A and 200B share the same virtual space. An avatar object 6A of the user 5A and an avatar object 6B of the user 5B exist in the virtual space 11A and the virtual space 11B. Although the avatar object 6A in the virtual space 11A and the avatar object 6B in the virtual space 11B are each wearing the HMD 120, this is for the sake of clarity of explanation, and in reality these objects are not wearing the HMDs 120. not

In one aspect, processor 210A may position virtual camera 14A capturing view image 17A of user 5A at the eye position of avatar object 6A.

FIG. 12(B) is a diagram showing a field-of-view image 17A of the user 5A in FIG. 12(A). The field-of-view image 17A is an image displayed on the monitor 130A of the HMD 120A. This field-of-view image 17A is an image generated by the virtual camera 14A. An avatar object 6B of the user 5B is displayed in the field-of-view image 17A. Although not particularly illustrated, the avatar object 6A of the user 5A is similarly displayed in the visual field image of the user 5B.

In the state of FIG. 12B, the user 5A can communicate with the user 5B through dialogue through the virtual space 11A. More specifically, the voice of user 5A acquired by microphone 170A is transmitted to HMD 120B of user 5B via server 600 and output from speaker 180B provided in HMD 120B. The voice of user 5B is transmitted to HMD 120A of user 5A via server 600, and is output from speaker 180A provided in HMD 120A.

Actions of the user 5B (actions of the HMD 120B and actions of the controller 300B) are reflected in the avatar object 6B arranged in the virtual space 11A by the processor 210A. Thereby, the user 5A can recognize the action of the user 5B through the avatar object 6B.

FIG. 13 is a sequence chart showing part of the processing executed in HMD system 100 according to the present embodiment. Although the HMD set 110D is not shown in FIG. 13, the HMD set 110D operates similarly to the HMD sets 110A, 110B, and 110C. In the following description, A is attached to the reference sign of each component regarding the HMD set 110A, B is attached to the reference sign of each component regarding the HMD set 110B, and C is attached to the reference sign of each component regarding the HMD set 110C. , and D is added to the reference numeral of each component of the HMD set 110D.

In step S1310A, the processor 210A in the HMD set 110A acquires avatar information for determining the motion of the avatar object 6A in the virtual space 11A. This avatar information includes, for example, information about the avatar such as motion information, face tracking data, and voice data. The movement information includes information indicating temporal changes in the position and tilt of the HMD 120A, information indicating hand movements of the user 5A detected by the motion sensor 420A, and the like. The face tracking data includes data specifying the position and size of each part of the user 5A's face. The face tracking data includes data indicating the movement of each organ that constitutes the face of the user 5A and line-of-sight data. The audio data includes data indicating the audio of the user 5A acquired by the microphone 170A of the HMD 120A. The avatar information may include information specifying the avatar object 6A or the user 5A associated with the avatar object 6A, information specifying the virtual space 11A in which the avatar object 6A exists, and the like. A user ID is mentioned as information which specifies the avatar object 6A and the user 5A. Room ID is mentioned as information which specifies virtual space 11A in which avatar object 6A exists. Processor 210A transmits the avatar information obtained as described above to server 600 via network 2 .

In step S1310B, processor 210B in HMD set 110B acquires avatar information for determining the motion of avatar object 6B in virtual space 11B and transmits it to server 600, as in the process in step S1310A. Similarly, in step S1310C, the processor 210C in the HMD set 110C acquires avatar information for determining the motion of the avatar object 6C in the virtual space 11C and transmits it to the server 600.

In step S1320, server 600 temporarily stores the player information received from each of HMD set 110A, HMD set 110B, and HMD set 110C. Server 600 integrates avatar information of all users (users 5A to 5C in this example) associated with common virtual space 11 based on user IDs, room IDs, and the like included in each avatar information. Then, the server 600 transmits the integrated avatar information to all users associated with the virtual space 11 at a predetermined timing. Accordingly, synchronization processing is executed. Such synchronization processing enables the HMD set 110A, the HMD set 110B, and the HMD 110C to share each other's avatar information at substantially the same timing.

Subsequently, based on the avatar information transmitted from server 600 to each HMD set 110A-110C, each HMD set 110A-110C executes the process of steps S1330A-S1330C. The processing of step S1330A corresponds to the processing of step S1180 in FIG.

In step S1330A, the processor 210A in the HMD set 110A updates information on the avatar objects 6B and 6C of the other users 5B and 5C in the virtual space 11A. Specifically, the processor 210A updates the position, orientation, etc. of the avatar object 6B in the virtual space 11 based on the motion information included in the avatar information transmitted from the HMD set 110B. For example, processor 210A updates the information (such as position and orientation) of avatar object 6B contained in the object information stored in memory module 530. FIG. Similarly, the processor 210A updates the information (position, orientation, etc.) of the avatar object 6C in the virtual space 11 based on the motion information included in the avatar information transmitted from the HMD set 110C.

In step S1330B, processor 210B in HMD set 110B updates information on avatar objects 6A and 6C of users 5A and 5C in virtual space 11B, similar to the process in step S1330A. Similarly, in step S1330C, the processor 210C in the HMD set 110C updates information on the avatar objects 6A, 6B of the users 5A, 5B in the virtual space 11C.

[Detailed module configuration]
Details of the module configuration of the computer 200 will be described with reference to FIG. FIG. 14 is a block diagram showing the detailed configuration of the modules of computer 200 according to an embodiment.

As shown in FIG. 14, the control module 510 includes a virtual camera control module 1421, a visual field determination module 1422, a reference gaze identification module 1423, a facial organ detection module 1424, a motion detection module 1425, and a virtual space definition module 1421. It comprises a module 1426 , a virtual object generation module 1427 , a manipulated object control module 1428 and an avatar control module 1429 . Rendering module 520 includes a view image generation module 1438 . The memory module 530 holds spatial information 1431 , object information 1432 , user information 1433 and face information 1434 .

The virtual camera control module 1421 arranges the virtual camera 14 in the virtual space 11 . The virtual camera control module 1421 controls the placement position of the virtual camera 14 in the virtual space 11 and the orientation (inclination) of the virtual camera 14 .

The virtual camera control module 1421 can control the orientation of the virtual camera 14, that is, the direction of the line of sight from the virtual viewpoint, according to the movement of the user's head. The movement of the user's head is the movement of the HMD 120 associated with the user, that is, the HMD 120 worn by the user, and is detected by the HMD sensor 410 . The HMD sensor 410 detects the movement direction, movement amount, movement speed, etc. of the user's head as the movement of the user's head, based on temporal changes in the position of the user's head.

The virtual camera control module 1421 can also control the direction of the line of sight from the virtual viewpoint according to the movement (position, orientation) of the controller 300 . Movement of the controller 300 is detected by the HMD sensor 410 using infrared rays emitted from an infrared LED 360, for example. The HMD sensor 410 detects the movement direction, movement amount, movement speed, etc. of the controller 300 as the movement of the controller 300 based on temporal changes in the position of the controller 300 .

The virtual camera control module 1421 normally controls the line-of-sight direction according to the movement of the user's head, but receives an input from the controller 300 to enable control of the line-of-sight direction according to the movement of the controller 300 . Then, the line-of-sight direction is controlled according to the movement of the controller 300 while this input is being accepted.

The visual field determination module 1422, as shown in FIGS. 6 and 7, determines the visual field area 15 based on the position of the virtual camera 14, that is, the virtual viewpoint and the direction of the line of sight from the virtual viewpoint. A view image generation module 1438 generates a view image 17 displayed on the monitor 130 based on the determined view area 15 .

The reference line of sight identification module 1423 identifies the line of sight of the user 5 based on the signal from the gaze sensor 140 . The facial organ detection module 1424 detects the facial organs of the user 5 (eg, mouth, eyes, eyebrows) from the facial images of the user 5 generated by the first camera 150 and the second camera 160 . The motion detection module 1425 detects the motion (shape) of each organ detected by the facial organ detection module 1424 .

The virtual space definition module 1426 defines the virtual space 11 in the HMD system 100 by generating virtual space data representing the virtual space 11 .

The virtual object generation module 1427 generates objects placed in the virtual space 11 . Objects may include, for example, landscapes including forests, mountains, etc., animals, etc. that are arranged as the story progresses in the game.

The operating object control module 1428 arranges in the virtual space 11 an operating object for receiving a user's operation in the virtual space 11 . The user operates an object placed in the virtual space 11, for example, by operating the operation object. In one aspect, the manipulation object may include, for example, a hand object corresponding to the hand of the user wearing the HMD 120, or the like. In one aspect, the manipulation object can correspond to a hand portion of an avatar object, which will be described later.

Avatar control module 1429 generates data for arranging avatar objects of users of other computers 200 connected via network 2 in virtual space 11 . In one aspect, avatar control module 1429 generates data for placing user 5's avatar object in virtual space 11 . In one aspect, avatar control module 1429 generates an avatar object that resembles user 5 based on an image containing user 5 . In another aspect, the avatar control module 1429 selects an avatar object selected by the user 5 from among a plurality of types of avatar objects (for example, an object imitating an animal or a deformed human object), and places the avatar object in the virtual space 11. Generate data for placement.

Avatar control module 1429 reflects the movement of HMD 120 detected by HMD sensor 410 on the avatar object. For example, the avatar control module 1429 detects that the HMD 120 is tilted and generates data for tilting and arranging the avatar object. In one aspect, the avatar control module 1429 reflects movements of the controller 300 on the avatar object. In this case, controller 300 includes a motion sensor, an acceleration sensor, a plurality of light emitting elements (for example, infrared LEDs), or the like for detecting movement of controller 300 . The avatar control module 1429 reflects the motion of facial organs detected by the motion detection module 1425 on the face of the avatar object placed in the virtual space 11 . In other words, the avatar control module 1429 reflects the motion of the user's face on the avatar object.

The control module 510 detects a collision when each of the objects placed in the virtual space 11 collides with another object. The control module 510 can detect, for example, the timing at which a certain object touches another object, and performs predetermined processing when the detection is made. The control module 510 can detect the timing when the objects are separated from the touching state, and performs a predetermined process when the detection is made. The control module 510 can detect that objects are touching. Specifically, when the operating object touches another object, the operating object control module 1428 detects that the operating object touches the other object, and performs predetermined processing. .

Memory module 530 holds data used by computer 200 to provide virtual space 11 to user 5 . In one aspect, memory module 530 retains spatial information 1431 , object information 1432 , user information 1433 and facial information 1434 .

The space information 1431 holds one or more templates defined for providing the virtual space 11 .

The object information 1432 holds content to be reproduced in the virtual space 11 , objects used in the content, and information (for example, position information) for arranging the objects in the virtual space 11 . The content may include, for example, games, content showing scenery similar to that in the real world, and the like.

The user information 1433 holds a program for causing the computer 200 to function as a control device of the HMD system 100, an application program using each content held in the object information 1432, and the like.

The face information 1434 holds pre-stored templates for the face feature detection module 1424 to detect the face features of the user 5 . In one aspect, face information 1434 holds mouth template 1435 , eye template 1436 , and eyebrow template 1437 . Each template may be an image corresponding to the organs that make up the face. For example, mouth template 1435 can be an image of a mouth. Each template may contain multiple images.

[360-degree video display]
In one aspect, processor 210 defines a virtual space using a 360-degree video as virtual space data. As a result, the monitor 130 of the HMD 120 is provided with a field-of-view image including a 360-degree video in the field of view. The 360-degree video may be, for example, a 360-degree video pre-stored in the storage 230, a 360-degree video downloaded from the outside, or the like. Also, the 360-degree moving image may be a 360-degree moving image captured by a 360-degree camera connected to the computer 200, or may be an animated moving image such as a background moving image of a game program.

[Control of gaze direction]
In one embodiment, the processor 210 in the computer 200 controls the line-of-sight direction according to the movement of the user's head (third mode) and the line-of-sight direction control mode according to the movement of the controller 300 . and a mode (fourth mode) for controlling the direction of the line of sight. The user can switch the control mode by operating the controller 300 .

FIG. 15 is a flow chart showing the processing performed by the processor 210 in the control mode of the line-of-sight direction according to the movement of the user's head. During this control mode, the processor 210 disables the other control mode, namely the control of the line of sight direction in response to the movement of the controller 300 . Also, the processor 210 repeatedly executes the processing shown in FIG. 15 during this control mode.

As shown in FIG. 15 , in step S1501, the processor 210 determines whether or not an input has been received from the controller 300 to enable control of the line-of-sight direction according to the movement of the controller 300 . The activation input may be any input operation, for example, an operation of pressing a button 350 provided on the controller 300, or the like. If no activation input has been received (S1501: NO), the processor 210 terminates the processing shown in FIG.

If an activation input has been received in step S1501 (S1501: YES), processor 210 switches the line-of-sight direction control mode to a control mode according to the movement of controller 300 in step S1502. At step S1503, the processor 210 disables the other control mode, which is the control according to the movement of the user's head.

At step S1504, the processor 210 determines whether or not motion of the controller 300 is detected. If the movement of controller 300 is not detected (S1504; NO), the process proceeds to step S1510. If the movement of the controller 300 is detected (S1504; YES), in step S1505 the processor 210 determines whether the operation mode of the controller 300 is the drag operation mode (first mode) or the flick operation mode (second mode). do. The drag operation mode is an operation mode in which the line-of-sight direction can be controlled based on the amount of movement of the controller 300 , and the flick operation mode is an operation mode in which the line-of-sight direction can be controlled based on the movement speed of the controller 300 . The processor 210 accepts a selection input from the user on a menu screen or the like, and presets either the drag operation mode or the flick operation mode as a prescribed operation mode, but accepts an input to change the operation mode from the user. may be switched at an arbitrary timing.

In the drag operation mode (S1505; drag), in step S1506, the processor 210 changes the direction of the line of sight from the virtual viewpoint to a direction determined according to the movement of the controller 300. FIG. For example, the processor 210 may translate the line of sight along with the virtual viewpoint in the same direction as the movement direction of the controller 300 . Also, the processor 210 may rotate the line of sight in a rotation direction determined according to the moving direction of the controller 300 with the virtual viewpoint as a base point. The direction of rotation determined according to the moving direction of the controller 300 is, for example, a counterclockwise direction from the virtual viewpoint when the moving direction of the controller 300 is rightward, and a clockwise direction of the line of sight when the controller 300 is moving leftward. It is the direction of rotation. With a simple operation, it is possible to change the direction of the line of sight with the sensation of a drag operation as if grabbing and moving the virtual space, thereby improving the virtual experience.

In one aspect, the processor 210 causes the speed of changing the direction of the line of sight to be a constant speed regardless of the speed of movement of the controller 300 . As a result, the speed of changing the field of view can be restricted to a constant value, and VR motion sickness due to the changing of the field of view can be suppressed. As the constant speed, for example, an experimentally obtained speed at which motion sickness does not occur can be selected.

When the line of sight is rotated with the virtual viewpoint as a base point in a rotation direction determined according to the movement direction of the controller 300 , the processor 210 rotates at a constant angular velocity regardless of the movement speed of the controller 300 . Processor 210 determines the amount of rotation at this time according to the amount of movement of controller 300 . For example, the greater the amount of movement of the controller 300, the greater the amount of rotation of the line of sight. The user 5 can adjust the amount of rotation by the amount of movement of the controller 300, and can perform the adjustment intuitively and easily.

In step S1507, the processor 210 generates a field of view image corresponding to the field of view determined according to the direction of the line of sight. The generated field-of-view image is displayed by HMD 120 .

16 and 17 are diagrams illustrating an example of controlling the direction of the line of sight 16 according to the movement of the controller 300. FIG. 16 is a diagram showing a state before the direction of the line of sight 16 is rotated, and FIG. 17 is a diagram showing a state after the direction of the line of sight 16 is rotated. 18 is a diagram showing a field-of-view image 1821 in the state of FIG. 16, and FIG. 19 is a diagram showing a field-of-view image 1921 in the state of FIG. The field-of-view image 1821 in FIG. 18 and the field-of-view image 1921 in FIG. 19 include a virtual hand 1831 corresponding to the user's 5 hand.

As shown in FIG. 16, an object 1645 placed in the virtual space 11 is within the field of view 15 , but an object 1643 is outside the field of view 15 . Therefore, as shown in FIG. 18, the field image 1821 includes the object 1645 but does not include the object 1643 .
After that, when the user 5 continues to input activation and moves the controller 300 in the direction of the arrow 1644 as shown in FIG. in the direction of arrow 1841 . At this time, the processor 210 arranges the open-shaped virtual hand 1831 in the virtual space 11 while there is no validation input, and arranges the virtual hand 1831 in the shape corresponding to the drag operation mode while the validation input is accepted. For example, it may be replaced with a virtual hand 1831 having a gripping shape.

Processor 210 rotates the direction of line of sight 16 in the direction indicated by arrow 1644 in FIG. When the controller 300 is rotated by the amount of rotation corresponding to the amount of movement of the controller 300, the processor 210 stops in the direction shown in FIG. Since the object 1643 is also included in the field of view area 15 after the line of sight 16 is rotated, the object 1643 is also included in the field of view image 1921 shown in FIG.

In one aspect, the processor 210 sets a threshold for the moving speed of the controller 300, and if the detected moving speed of the controller 300 is less than the threshold, rotates the controller 300 at an angular speed determined according to the moving speed; rotates at a constant angular velocity determined according to this threshold. The user 5 can adjust the amount of rotation by changing the moving speed of the controller 300, and can perform the adjustment intuitively and easily. Since the rotation speed of the field of view is fixed when the movement speed reaches a certain level, it is possible to reduce motion sickness due to changes in the field of view.

On the other hand, if the operation mode of the controller 300 is the flick operation mode (S1505: flick), in step S1508 the processor 210 rotates the line of sight in a rotation direction determined according to the moving direction of the controller 300, with the virtual viewpoint as the starting point. Let Also, the processor 210 rotates the line of sight at an angular velocity determined according to the movement speed of the controller 300 . For example, the faster the controller 300 moves, the faster the angular velocity. As a result, it is possible to operate with the feeling of a flick operation as if shaking off the virtual space, and it is possible to improve the virtual experience with an easy operation. In the case of the flick operation, the shape of the virtual hand included in the view image may be the shape of the hand with the fingers open. By making the hand shape different from that in the drag operation mode, the user 5 can easily recognize the difference in the operation mode.

In step S1509, the processor 210 generates a field-of-view image corresponding to the field of view determined according to the line-of-sight direction. In one aspect, the processor 210 generates a view image that is less visible during the rotation of the line of sight than when the line of sight is not rotated. The processor 210 can reduce the visibility by image processing the field of view image corresponding to the field of view determined by the direction of the rotated line of sight. As the image with reduced visibility, an image in which the amount of information visually recognized by the user is reduced can be used. For example, an image with a narrow field of view, for example, the periphery of which is painted black.

In step S1510, the processor 210 determines whether or not the input of enabling the direction control of the line of sight according to the movement of the controller 300 has ended. For example, when the pressing operation of the button 350 corresponding to the activation input is released, the processor 210 determines that the activation input is finished. If the validation input has not ended (step S1510; NO), the process returns to step S1504.

If the activation input has ended (S1510: YES), in step S1511, the processor 210 changes the direction of the line of sight according to the movement of the head of the user 5 from the control mode for controlling the direction of the line of sight according to the movement of the controller 300. Switch to the control mode that controls the direction of At step S<b>1512 , the processor 210 disables the direction control of the line of sight according to the movement of the controller 300 .

In the above processing, the processor 210 may define a standard direction of the line of sight such as the home position, and upon receiving a predetermined input for resetting, perform direction control to return the direction of the line of sight to the standard direction. The standard direction of the line of sight may be the direction immediately before receiving the activation input, or may be a predetermined direction. The predetermined input may be an operation of the controller 300 such as simultaneous pressing of a plurality of buttons 350 , or a predetermined movement of the head of the user 5 detected by the HMD sensor 410 . According to this control, even if the direction of the line of sight is greatly changed, it can be easily returned to the standard direction.

In one aspect, the processor 210 performs charging processing in accordance with the control of the line-of-sight direction in accordance with the movement of the controller 300 . The billing amount in the billing process may be a fixed amount, or may be an amount corresponding to the amount of change in the line-of-sight direction. For example, the processor 210 can perform pay-as-you-go billing in which the billing amount increases as the amount of rotation increased according to the movement of the controller 300 . The user 5 can have a highly immersive virtual experience with easy operation by charging.

As described above, by controlling the line-of-sight direction according to the movement of the controller 300 while the activation input is being received, the user 5 can normally change the line-of-sight direction by moving the head, and the line-of-sight direction can be changed directly behind. When it is difficult to change the direction of the line of sight by the movement of the head, such as when turning the head to the side, or when operating while lying down, the direction of the line of sight can be changed by the controller 300 by performing the activation input operation only during that time. can. It is possible to switch to the operation by the controller 300 with a simple operation only when necessary, and the virtual experience of the user 5 can be improved.

In addition, since the direction of the line of sight according to the movement of the controller 300 can be changed in the drag operation mode or the flick operation mode, the user 5 can have a virtual experience as if changing the field of view of the virtual space with the user 5's hand. You get a deep sense of immersion.

In the above-described embodiment, the processor 210 may accept the fact that the shape of the hand of the user 5 is a specific shape as the validation input regardless of the operation of the controller 300 . For example, a particular shape is a clenched hand. For example, when the HMD sensor 410 is implemented by a camera, the shape of the hand of the user 5 is detected by the processor 210 performing image analysis processing on the image of the hand of the user 5 that is captured and output by this camera.

Similarly, the processor 210 may perform line-of-sight direction control in response to the movement of the hand of the user 5 while receiving the activation input. As described above, the processor 210 detects the movement of the hand, such as the direction of movement, the amount of movement, and the movement speed of the hand, based on temporal changes in the position, shape, posture, and the like of the hand detected by image analysis processing of the hand image. To detect. The processor 210 controls the direction of the line of sight in response to the detected hand movement in a manner similar to controlling the direction of the line of sight in response to the movement of the controller 300 .

[Video playback control]
In one embodiment, processor 210 plays a video in virtual space. The moving image may be a 360-degree moving image that defines the virtual space, or may be a moving image played on an object such as a screen placed in the virtual space. While processor 210 receives an input from controller 300 to enable playback control of a moving image in accordance with the movement of controller 300 , processor 210 performs playback control of the moving image in accordance with the movement of controller 300 .

FIG. 20 is a flow chart showing a processing procedure when the processor 210 performs moving image reproduction control.
As shown in FIG. 20 , in step S2021, processor 210 determines whether or not an input for activating reproduction control according to movement of controller 300 has been received from controller 300 . The activation input may be any input operation, for example, an operation of pressing a button 350 provided on the controller 300, or the like. If the input for activation has not been received (S2021: NO), the processor 210 ends this process.

On the other hand, if an activation input has been received (S2021: YES), processor 210 determines in step S2022 whether or not movement of controller 300 has been detected. If the movement of controller 300 is not detected (S2022: NO), the process proceeds to step S2004.

If the motion of the controller 300 is detected (S2022: YES), the processor 210 performs video playback control according to the motion of the controller 300 in step S2023. In one aspect, processor 210 performs playback control corresponding to the movement direction of controller 300 . For example, the processor 210 performs reproduction control of fast-forwarding the moving image if the movement direction of the controller 300 is to the right, and rewinding of the moving image if the movement direction is to the left. Also, the processor 210 performs reproduction control according to the movement amount or movement speed of the controller 300 . For example, in the case of fast forward, the processor 210 increases the fast forward speed as the movement amount of the controller 300 increases or as the movement speed increases.

In one aspect, the processor 210 selects one of the drag operation mode in which the reproduction control is performed according to the movement amount of the controller 300 and the flick operation mode in which the reproduction control is performed according to the movement speed of the controller 300. Playback control may be performed according to the selected operating mode.
In step S2024, processor 210 generates a view image corresponding to the view defined by the line of sight. The generated field-of-view image is displayed by HMD 120 .

FIG. 21 shows an example of a field-of-view image during playback control.
A field-of-view image 2121 shown in FIG. 21 is a field-of-view image generated from a 360-degree moving image that defines the virtual space. The view image 2121 includes a virtual hand 2122 corresponding to the user's 5 hand. When the controller 300 is used to fast-forward a 360-degree moving image, the user 5 moves the controller 300 to the right while performing an input operation for enabling playback control. The processor 210 moves the virtual hand 2122 placed in the virtual space to the right in accordance with the movement of the controller 300 to the right, and also increases the frame rate of the field-of-view image output to the HMD 120 to fast-forward the 360-degree video. . Also, the processor 210 compares the amount of movement of the controller 300 with, for example, a threshold value, and divides it into three stages.

If the view image 2121 includes a fast-forward mark 2124, the user 5 can easily recognize that fast-forwarding is being performed. A mark such as the mark 2124 that indicates the content of the playback control is displayed in the view image, for example, by the processor 210 arranging the mark object in the virtual space or performing image processing on the view image.

In one aspect, processor 210 performs volume control of the moving image according to the movement of controller 300 while receiving an input from controller 300 to enable reproduction control of the moving image according to movement of controller 300 . For example, when the controller 300 is used to turn down the volume in the visual field image 2121 shown in FIG. Processor 210 moves virtual hand 2122 downward in response to downward movement of controller 300 and reduces the volume output to HMD 120 . The processor 210 also performs display control to lower the gauge indicating the volume of the volume bar 2125 included in the view image 2121 in the same manner as the mark 2124 .

At step S2025, the processor 210 determines whether or not the activation input is finished. If the validation input has not been completed (S2025: NO), the process returns to step S2022. On the other hand, if the validation input has been completed (S2025: YES), this processing ends.

As described above, while the activation input is being accepted, the processor 210 controls the reproduction of the moving image according to the movement of the controller 300, so that the user 5 can perform the reproduction operation with a simple and sensuous operation. , improving operability and virtual experience.

[Control of line-of-sight direction according to movement of information display terminal]
In one embodiment, a portable terminal such as a smart phone or a tablet terminal functions as the computer 200, and the display of the terminal functions as the monitor 130 that displays the view image. A terminal is equipped with a gyro sensor or the like, and can detect movements such as inclination and movement of the terminal. The processor 210 in the terminal controls the direction of the line of sight according to the detected movement of the terminal in the same manner as in the above-described embodiment in which the direction of the line of sight is controlled according to the movement of the head of the user 5, thereby displaying the view image. It is possible to generate

The processor 210 of the terminal normally disables the control of the line-of-sight direction according to the movement of the terminal. , to control the direction of the line of sight according to the movement of the terminal.
FIG. 22 shows the processing performed by the processor 210 of the terminal when disabling the control of the direction of gaze in response to the movement of the terminal.

As shown in FIG. 22, in step S2221, the processor 210 determines whether or not an input to enable control of the line-of-sight direction according to the movement of the terminal has been received. For example, if the display of the terminal is a touch panel, the activation input may be a touch on the display, or if a button such as a home button is provided, it may be an operation of pressing the button.

If the activation input has not been received (S2221: NO), the processor 210 ends this process.
On the other hand, if an activation input has been accepted (S2221: YES), in step S2222 the processor 210 activates the control of the line-of-sight direction according to the movement of the terminal. The processor 210 moves the line of sight in the direction in which the terminal moves in response to movement of the terminal detected by a gyro sensor or the like of the terminal, for example, movement or tilt of the terminal, and changes the direction of the line of sight in the tilted direction of the terminal. to change

At step S2223, the processor 210 determines whether or not the activation has been entered. If the validation input has not ended (S2223: NO), the process returns to step S2222. If the activation input has ended (S2223: YES), in step S2224 the processor 210 deactivates the control of the line-of-sight direction according to the movement of the terminal, and ends this process.

23 to 25 show display examples of field-of-view images on the terminal. FIG. 23 and FIG. 24 are display examples of the field-of-view image when the line-of-sight direction control performed according to the movement of the terminal is disabled. FIG. 25 is a display example of the field-of-view image when the line-of-sight direction control according to the movement of the terminal is enabled.
As shown in FIG. 23, a field image 2322 is displayed on the terminal 2321 . Since line-of-sight direction control is disabled until an input for enabling is accepted, as shown in FIG. 24, the line-of-sight direction is not changed even if the terminal 2321 is tilted. Therefore, the field of view does not tilt and the field of view image 2322 does not change.

On the other hand, when an input for activation is received, control of the line-of-sight direction according to the movement of the terminal is activated while the input is being received. As shown in FIG. 25, when the tilt of the terminal 2321 is detected, the processor 210 rotates, for example, the direction of the line of sight in the virtual space by the same tilt as the tilt of the terminal 2321 with the virtual viewpoint as the base point. Therefore, the terminal 2321 displays a view image 2521 in which the view is tilted due to the rotation of the line of sight.

As described above, the processor 210 controls the line-of-sight direction according to the movement of the terminal while receiving the activation input. can be changed. If there is no activation input, the line-of-sight direction does not change even if the terminal moves, so there is no need to worry about the line-of-sight direction changing even when the user 5 accidentally tilts the terminal. You can get rid of it.

In the above embodiments, the virtual space (VR space) in which the user is immersed by the HMD has been exemplified and explained, but a transmissive HMD may be employed as the HMD. In this case, by outputting a visual field image obtained by synthesizing a part of an image constituting a virtual space with a real space visually recognized by a user through a transmissive HMD, an augmented reality (AR) space or mixed reality ( A user may be provided with a virtual experience in MR (Mixed Reality) space. In this case, an action may be generated on the target object in the virtual space based on the movement of the user's hand instead of the operation object. Specifically, the processor may identify the coordinate information of the position of the user's hand in the real space and define the position of the target object in the virtual space in relation to the coordinate information in the real space. As a result, the processor can grasp the positional relationship between the user's hand in the real space and the target object in the virtual space, and execute processing corresponding to the above-described collision control or the like between the user's hand and the target object. . As a result, it is possible to give an effect to the target object based on the movement of the user's hand.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

[Appendix]
The technical features disclosed above can be summarized as follows.

(Appendix 1)
According to one embodiment, a program for execution by a computer is provided. The program comprises the steps of: defining a virtual space including a virtual viewpoint; detecting movement of a user's head with an associated head-mounted device; detecting movement of a controller held by the user; and receiving an input from the controller to enable control of the line-of-sight direction in accordance with the movement of the controller. a step of controlling the direction of the line of sight according to the movement of the controller while the input to enable is being received; A program for causing a computer to execute a step to be displayed on the .

(Appendix 2)
In the program, the step of controlling the direction of the line of sight according to the movement of the controller controls the direction of the line of sight in a direction determined according to the movement of the controller at a constant speed regardless of the movement speed of the controller. The program according to (Appendix 1), which is changed.

(Appendix 3)
In the program, the step of controlling the direction of the line of sight according to the movement of the controller includes moving the line of sight with the virtual viewpoint as a base point in a rotation direction determined according to the movement direction of the controller by a movement amount of the controller. The program according to (Appendix 2), rotating at a constant angular velocity regardless of the moving speed of the controller by the amount of rotation determined according to .

(Appendix 4)
In the step of controlling the direction of the line of sight in accordance with the movement of the controller, the step of controlling the direction of the line of sight in accordance with the movement of the controller includes controlling the line of sight with the virtual viewpoint as a base point in accordance with the movement direction of the controller when the movement speed of the controller is less than a threshold value. is rotated at an angular speed determined according to the movement speed by a rotation amount determined according to the movement amount of the controller in a rotation direction determined by The program according to (Appendix 1), wherein, as a base point, the rotation amount is rotated in the rotation direction at a constant angular velocity determined according to the threshold value.

(Appendix 5)
In the program, the step of controlling the direction of the line of sight according to the movement of the controller includes moving the line of sight in a rotation direction determined according to the movement direction of the controller, with the virtual viewpoint as a base point, and at the movement speed of the controller. and displaying the field image on the head-mounted device while rotating the line of sight at an angular velocity determined according to is displayed on the head-mounted device.

(Appendix 6)
The program selects from the controller one of a first mode in which the direction of the line of sight is controlled based on the amount of movement of the controller, and a second mode in which the direction of the line of sight is controlled based on the speed of movement of the controller. The step of receiving an input to select is further executed by the computer, and the step of controlling the direction of the line of sight according to the movement of the controller is based on the amount of movement of the controller when the first mode is selected. Any one of (Appendix 1) to (Appendix 5), wherein the line-of-sight direction is controlled, and when the second mode is selected, the line-of-sight direction is controlled based on the movement speed of the controller. It is a described program.

(Appendix 7)
The program controls the third mode of controlling the field of view according to the movement of the head and the field of view according to the movement of the controller in response to the start of acceptance of the enabling input. causing the computer to further execute a step of switching to a fourth mode; and a step of switching the fourth mode to the third mode in response to termination of acceptance of the enabling input; , the control of the field of view according to the movement of the controller is invalidated, and in the fourth mode, the control of the field of view according to the movement of the head is invalidated, (Appendix 1) to (Appendix 6 ) is a program according to any one of

(Appendix 8)
The program according to any one of (Appendix 1) to (Appendix 7), for causing the computer to further execute a step of performing a billing process according to the control of the line-of-sight direction according to the movement of the controller. program.

(Appendix 9)
According to one embodiment, a program for execution by a computer is provided. The program comprises the steps of: defining a virtual space including a virtual viewpoint; reproducing a moving image in the virtual space; detecting movement of a controller held by a user; receiving an input to enable control of playback of the moving image; controlling playback of the moving image in response to movement of the controller while the input to enable is being received; and displaying, on a head-mounted device associated with the user, a field-of-view image corresponding to the field of view from the head-mounted device.

(Appendix 10)
According to one embodiment, a computer-implemented method is provided. The method includes defining a virtual space including a virtual viewpoint, detecting movement of the head of a user with an associated head-mounted device, and responsive to movement of the user's head from the virtual viewpoint. detecting movement of a controller held by the user; and receiving an input from the controller to enable control of the line-of-sight direction in accordance with the movement of the controller. a step of controlling the direction of the line of sight according to the movement of the controller while the input to enable is being received; and displaying to.

(Appendix 11)
According to one embodiment, an information processing device is provided. The information processing apparatus includes a storage unit that stores the program, and a computer that executes the program, the program defining a virtual space including a virtual viewpoint in the computer; Detecting associated head movements of the user; Controlling a direction of a line of sight from the virtual viewpoint according to the user's head movements; Detecting movements of a controller held by the user. receiving from the controller an input to enable control of the line-of-sight direction in accordance with the movement of the controller; , the step of controlling the line-of-sight direction, and the step of displaying, on the head-mounted device, a field-of-view image corresponding to the field of view determined according to the line-of-sight direction.

(Appendix 12)
According to one embodiment, a program for execution by a computer is provided. The program defines a virtual space including a virtual viewpoint, detects movement of a terminal, and receives an input to enable control of a line-of-sight direction from the virtual viewpoint according to the movement of the terminal. , controlling the direction of a line of sight from the virtual viewpoint according to the movement of the terminal while the input to activate is being received; and a step of displaying on the terminal.

It should be considered that the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above description, and is intended to include all modifications within the scope and meaning of equivalents of the scope of the claims.

2 network, 5 user, 6 avatar object, 11 virtual space, 12 center, 14 virtual camera, 15 field of view, 100 HMD system, 110 HMD set, 130 monitor, 170 microphone, 180...Speaker, 190...Sensor, 200...Computer, 210...Processor, 220...Memory, 230...Storage, 240...Input/output interface, 250...Communication interface, 300...Controller, 310...Grip, 320...Frame, 340, 350 , 370, 380... Buttons, 390... Analog stick, 410... HMD sensor, 420... Motion sensor, 430... Display, 510... Control module, 520... Rendering module, 530... Memory module, 540... Communication control module, 600... Server , 610 ... processor, 620 ... memory, 630 ... storage, 640 ... input/output interface, 650 ... communication interface, 1421 ... virtual camera control module, 1422 ... visual field determination module, 1423 ... reference line of sight identification module, 1424 ... motion detection module , 1426 ... virtual space definition module, 1427 ... virtual object generation module, 1428 ... operation object control module, 1429 ... avatar control module, 1438 ... view image generation module.

Claims (6)

a definition means for defining a virtual space including a virtual viewpoint;
a moving image reproducing means for reproducing moving images in the virtual space;
motion detection means for detecting motion of a controller held by a user;
receiving means for receiving an input from the controller for activating control of reproduction of the moving image according to movement of the controller;
playback control means for controlling playback of the moving image in response to movement of the controller while the input to enable is being accepted;
display control means for displaying a view image corresponding to the view from the virtual viewpoint on a device worn by the user;
A program that causes a computer to run 2. The program according to claim 1, wherein said reproduction control means controls reproduction of said moving image such that said moving image is fast-forwarded or rewound according to the moving direction of said controller. 3. The reproduction control means according to claim 2, wherein said reproduction control means controls reproduction of said moving image such that said moving image is fast-forwarded when said controller is moved in the right direction, and said moving image is rewound when said controller is moved in the left direction. program. 2. The program according to claim 1, wherein said reproduction control means controls reproduction of said moving image according to a movement amount or movement speed of said controller. 2. The apparatus according to claim 1, wherein said reproduction control means controls the volume of said moving image according to the movement of said controller while receiving an input for enabling reproduction control of said moving image according to said movement of said controller. program. a definition means for defining a virtual space including a virtual viewpoint;
a moving image reproducing means for reproducing moving images in the virtual space;
motion detection means for detecting motion of a controller held by a user;
receiving means for receiving an input from the controller for activating control of reproduction of the moving image according to movement of the controller;
playback control means for controlling playback of the moving image in response to movement of the controller while the input to enable is being accepted;
display control means for displaying a field-of-view image corresponding to the field of view from the virtual viewpoint on a device worn by the user;
system.

Images