JP2023096844A - Program, information processing method, and information processing apparatus - Google Patents

Abstract

To improve usability when providing a panoramic image to a user.SOLUTION: A program 231 is run on a computer 200 comprising a processor 210. The program 231 causes the processor 210 to perform a step of generating a presentation image to be presented to a user on the basis of a virtual camera arranged inside a celestial sphere to which an image is projected, a step of outputting the presentation image to a display device, a step of acquiring movement information about movement of at least a part of user's body detected by a sensor unit, and a step of changing a physical relationship between the virtual camera and the celestial sphere or a view angle of the virtual camera on the basis of the movement information.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to programs, information processing methods, and information processing apparatuses.

2. Description of the Related Art Conventionally, there is known a technique for providing a user with a virtual space such as a VR (Virtual Reality) space using equipment such as a head-mounted device (hereinafter also referred to as "HMD"). For example, Patent Literature 1 discloses that an inertial sensor in the HMD is used to detect the motion of the HMD, and based on the detected motion, the action (including movement, etc.) of the character in the virtual reality scene is determined. It is

Further, Patent Literature 2 discloses that, in an HMD including a display unit that displays a superimposed image to be superimposed on the user's actual field of view, the display area of the superimposed image is reduced according to the posture of the HMD. Further, in Patent Document 3, an image of a virtual space is displayed on the HMD, and when an input device for inputting an operation by the user exists at a predetermined position in the real space, part of the image displayed on the HMD is magnified, and the magnified image is superimposed and displayed on the image in the virtual space.

JP 2018-508805 A JP 2016-212599 A JP 2019-46291 A

By the way, there is known a technique for providing a user with an image based on a virtual camera placed inside a celestial sphere on which an image is projected. Such an image is also called a panorama image, and can give the user an immersive feeling as if he or she were there. However, in conventional panoramic images, although the user can change the line-of-sight direction, the viewpoint magnification is often fixed and cannot be changed. I couldn't. A conventional panoramic image is projected onto a large celestial sphere whose radius is set to infinity. Since the positional relationship of the virtual camera is not changed, the scale of the image is not changed based on the user's motion information, and there is room for improvement in terms of usability.

Incidentally, Patent Documents 1 to 3 do not show any concept of changing the scale of the panoramic image based on the motion information of the user. Also, it is not disclosed to change the positional relationship of the virtual cameras displayed based on the user's motion information.

One aspect of the present disclosure aims to improve usability when providing a panoramic image to a user.

According to one embodiment shown in the present disclosure,
A program executed on a computer with a processor,
The program causes the processor to:
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
a step of acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
A program is provided.

According to an embodiment shown in the present disclosure, usability can be improved when a panoramic image is provided to a user.

1 is a schematic representation of the configuration of an HMD system according to an embodiment; FIG. It is a block diagram showing an example of the hardware constitutions of the computer according to an embodiment. FIG. 4 is a diagram conceptually representing a uvw viewing coordinate system set in an HMD according to an embodiment; 4 is a flowchart illustrating an example of operational processing according to an embodiment; FIG. 4 is a schematic diagram showing an example of enlargement and reduction of a presentation image according to an embodiment; FIG. 5 is a schematic diagram showing an example of changing the positional relationship between a virtual camera and a celestial sphere according to an embodiment; FIG. 5 is a schematic diagram showing an example of changing the positional relationship between a virtual camera and a celestial sphere according to an embodiment; FIG. 5 is a schematic diagram showing an example of changing the viewing angle of a virtual camera according to an embodiment; FIG. 4 is a schematic diagram showing an example of the relationship between virtual space and real space according to an embodiment;

Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. In one or more of the embodiments presented in this disclosure, elements of each embodiment may be combined with each other and the combined result shall also form part of the embodiments presented in this disclosure.

An example of providing a user with a panoramic image using an HMD system will be described below, but the present disclosure is not limited to this example. The present disclosure can be applied without any particular limitation as long as it is a system that provides panoramic images to users. In the following, a case where a panoramic image is provided to a user in a VR space is taken as an example, but the present disclosure applies not only to VR but also to AR (Augmented Reality), MR (Mixed Reality), and SR (Substitutional Reality). It can be applied to so-called XR technology in general.

In this specification, a “panoramic image” refers to a presentation image presented to the user based on, for example, a virtual camera arranged inside the celestial sphere on which the image is projected. "Panorama images" include so-called 360-degree images, 180-degree images, full-dome images, and half-dome images. A “panoramic image” may be an image developed within a predetermined angular range other than 360 degrees or 180 degrees. Also, the "panorama image" may be a moving image or a still image.

[Configuration of HMD system]
A configuration of an HMD (Head-Mounted Device) system 100 will be described with reference to FIG. FIG. 1 is a diagram showing a schematic configuration of an HMD system 100 according to this embodiment. The HMD system 100 is provided as a system for home use or as a system for business use.

HMD system 100 includes server 600 , HMD sets 110 A, 110 B, 110 C, and 110 D, external device 700 , and network 2 . Each of HMD sets 110A, 110B, 110C, and 110D is configured to communicate with server 600 and external device 700 via network 2 . Hereinafter, the HMD sets 110A, 110B, 110C, and 110D are collectively referred to as the HMD set 110 as well. The number of HMD sets 110 configuring the HMD system 100 is not limited to four, and may be three or less or five or more. HMD set 110 includes HMD 120 , computer 200 , HMD sensor 410 , display 430 and controller 300 . HMD 120 includes monitor 130 , gaze sensor 140 , first camera 150 , second camera 160 , microphone 170 , and speaker 180 . Controller 300 may include motion sensor 420 .

In one aspect, computer 200 is connectable to network 2 such as the Internet, and is capable of communicating with server 600 and other computers connected to network 2 . Other computers include, for example, computers of other HMD sets 110 and external devices 700 . In another aspect, HMD 120 may include sensor 190 instead of HMD sensor 410 .

The HMD 120 can be worn on the head of the user 5 and provide the user 5 with a virtual space during operation. More specifically, HMD 120 displays a right-eye image and a left-eye image on monitor 130 . When each eye of the user 5 views the respective image, the user 5 can perceive the image as a three-dimensional image based on the parallax of both eyes. The HMD 120 may include both a so-called head-mounted display having a monitor and a head-mounted device on which a smartphone or other terminal having a monitor can be attached. Also, as the HMD 120, VR glasses, smart glasses, AR glasses, MR glasses, contact lens displays, or the like may be used.

The monitor 130 is implemented as, for example, a non-transmissive display device. In one aspect, the monitor 130 is arranged on the main body of the HMD 120 so as to be positioned in front of both eyes of the user 5 . Therefore, the user 5 can immerse himself in the virtual space by viewing the three-dimensional image displayed on the monitor 130 . In one aspect, the virtual space includes images of, for example, a background, objects that the user 5 can manipulate, and menus that the user 5 can select. In one aspect, the monitor 130 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smart phone or other information display terminal.

In another aspect, monitor 130 may be implemented as a transmissive display. In this case, the HMD 120 may be an open type, such as a glasses type, instead of a closed type that covers the eyes of the user 5 as shown in FIG. Transmissive monitor 130 may be temporarily configurable as a non-transmissive display by adjusting its transmittance. The monitor 130 may include a configuration for simultaneously displaying a portion of the image forming the virtual space and the real space. For example, the monitor 130 may display an image of the real space captured by a camera mounted on the HMD 120, or may make the real space visible by setting a partial transmittance high.

In one aspect, monitor 130 may include a sub-monitor for displaying images for the right eye and a sub-monitor for displaying images for the left eye. In another aspect, monitor 130 may be configured to display the image for the right eye and the image for the left eye as one. In this case, monitor 130 includes a high speed shutter. The high speed shutter operates to alternately display the right eye image and the left eye image so that the image is perceived by only one eye.

In one aspect, the HMD 120 includes multiple light sources (not shown). Each light source is implemented by, for example, an LED (Light Emitting Diode) that emits infrared rays. HMD sensor 410 has a position tracking function for detecting movement of HMD 120 . More specifically, HMD sensor 410 reads a plurality of infrared rays emitted by HMD 120 and detects the position and tilt of HMD 120 in the physical space.

In another aspect, HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the HMD 120 by executing image analysis processing using the image information of the HMD 120 output from the camera.

In another aspect, HMD 120 may include sensor 190 instead of HMD sensor 410 or in addition to HMD sensor 410 as a position detector. HMD 120 can detect the position and tilt of HMD 120 using sensor 190 . For example, if sensor 190 is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, HMD 120 can detect its own position and tilt using any of these sensors instead of HMD sensor 410 . As an example, when the sensor 190 is an angular velocity sensor, the angular velocity sensor temporally detects angular velocities around three axes of the HMD 120 in real space. The HMD 120 calculates temporal changes in angles around the three axes of the HMD 120 based on the angular velocities, and further calculates the tilt of the HMD 120 based on the temporal changes in the angles.

Gaze sensor 140 detects the directions in which the user's 5 right and left eyes are directed. That is, the gaze sensor 140 detects the line of sight of the user 5 . Detection of the line-of-sight direction is achieved by, for example, a known eye-tracking function. Gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, gaze sensor 140 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 140 may be, for example, a sensor that irradiates the right and left eyes of the user 5 with infrared light and receives reflected light from the cornea and iris of the irradiated light, thereby detecting the rotation angle of each eyeball. . The gaze sensor 140 can detect the line of sight of the user 5 based on each detected rotation angle.

The first camera 150 photographs the lower part of the user's 5 face. More specifically, first camera 150 captures the nose, mouth, and the like of user 5 . The second camera 160 photographs the eyes, eyebrows, etc. of the user 5 . The housing of the HMD 120 on the side of the user 5 is defined as the inside of the HMD 120 , and the housing of the HMD 120 on the side opposite to the user 5 is defined as the outside of the HMD 120 . In one aspect, first camera 150 may be positioned outside HMD 120 and second camera 160 may be positioned inside HMD 120 . Images generated by first camera 150 and second camera 160 are input to computer 200 . In another aspect, the first camera 150 and the second camera 160 may be realized as one camera, and the face of the user 5 may be photographed with this one camera.

The microphone 170 converts the speech of the user 5 into an audio signal (electrical signal) and outputs it to the computer 200 . The speaker 180 converts the audio signal into audio and outputs it to the user 5 . In another aspect, HMD 120 may include earphones instead of speaker 180 .

The controller 300 is connected to the computer 200 by wire or wirelessly. The controller 300 accepts command input from the user 5 to the computer 200 . In one aspect, controller 300 is configured to be grippable by user 5 . In another aspect, controller 300 is configured to be attachable to part of user's 5 body or clothing. In yet another aspect, controller 300 may be configured to output at least one of vibration, sound, and light based on signals transmitted from computer 200 . In yet another aspect, controller 300 receives an operation from user 5 for controlling the position and movement of an object placed in the virtual space.

In one aspect, controller 300 includes multiple light sources. Each light source is implemented, for example, by an LED that emits infrared rays. The HMD sensor 410 has a position tracking function. In this case, the HMD sensor 410 reads multiple infrared rays emitted by the controller 300 and detects the position and tilt of the controller 300 in the physical space. In another aspect, HMD sensor 410 may be implemented by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the controller 300 by executing image analysis processing using the image information of the controller 300 output from the camera.

Motion sensor 420 , in one aspect, is attached to the hand of user 5 to detect movement of the hand of user 5 . For example, the motion sensor 420 detects hand rotation speed, number of rotations, and the like. The detected signal is sent to computer 200 . Motion sensor 420 is provided in controller 300, for example. In one aspect, motion sensor 420 is provided, for example, in controller 300 configured to be grippable by user 5 . In another aspect, for safety in the real space, the controller 300 is attached to an object such as a glove that is attached to the hand of the user 5 so as not to fly off easily. In yet another aspect, sensors not worn by user 5 may detect movement of user's 5 hand. For example, a signal from a camera that takes an image of user 5 may be input to computer 200 as a signal representing the motion of user 5 . Motion sensor 420 and computer 200 are, for example, wirelessly connected to each other. In the case of wireless communication, the form of communication is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

Display 430 displays an image similar to the image displayed on monitor 130 . This allows users other than the user 5 wearing the HMD 120 to view the same image as that of the user 5 . The image displayed on display 430 does not have to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 430 include a liquid crystal display and an organic EL monitor.

Server 600 may transmit programs to computer 200 . In another aspect, server 600 may communicate with other computers 200 to provide virtual reality to HMDs 120 used by other users. For example, when a plurality of users play a participatory game in an amusement facility, each computer 200 communicates a signal based on each user's action with the other computers 200 via the server 600 so that a plurality of users can participate in the same virtual space. users to enjoy common games. Each computer 200 may communicate signals based on each user's actions with other computers 200 without going through the server 600 .

The external device 700 may be any device as long as it can communicate with the computer 200 . The external device 700 may be, for example, a device capable of communicating with the computer 200 via the network 2, or may be a device capable of directly communicating with the computer 200 through short-range wireless communication or wired connection. Examples of the external device 700 include smart devices, PCs (Personal Computers), and peripheral devices of the computer 200, but are not limited to these.

[Computer hardware configuration]
A computer 200 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of computer 200 according to this embodiment. The computer 200 includes a processor 210, a memory 220, a storage 230, an input/output interface 240, and a communication interface 250 as main components. Each component is connected to bus 260 respectively.

Processor 210 executes a series of instructions contained in a program stored in memory 220 or storage 230 based on a signal given to computer 200 or based on the establishment of a predetermined condition. In one aspect, processor 210 is implemented as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), MPU (Micro Processor Unit), FPGA (Field-Programmable Gate Array), or other device.

Memory 220 temporarily stores programs and data. The program is loaded from storage 230, for example. Data includes data input to computer 200 and data generated by processor 210 . In one aspect, memory 220 is implemented as random access memory (RAM) or other volatile memory.

Storage 230 permanently holds programs and data. The storage 230 is implemented as, for example, a ROM (Read-Only Memory), hard disk device, flash memory, or other non-volatile storage device. The programs stored in the storage 230 include programs for providing a virtual space in the HMD system 100, simulation programs, game programs, user authentication programs, and programs for realizing communication with other computers 200. The data stored in the storage 230 includes data and objects for defining the virtual space.

In another aspect, storage 230 may be implemented as a removable storage device such as a memory card. In still another aspect, instead of storage 230 built into computer 200, a configuration using programs and data stored in an external storage device may be used. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used, such as an amusement facility, it is possible to collectively update programs and data.

Input/output interface 240 communicates signals between HMD 120 , HMD sensor 410 , motion sensor 420 and display 430 . Monitor 130 , gaze sensor 140 , first camera 150 , second camera 160 , microphone 170 and speaker 180 included in HMD 120 can communicate with computer 200 via input/output interface 240 of HMD 120 . In one aspect, the input/output interface 240 is implemented using a USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or other terminals. Input/output interface 240 is not limited to the one described above.

In some aspects, input/output interface 240 may also communicate with controller 300 . For example, input/output interface 240 receives signals output from controller 300 and motion sensor 420 . In another aspect, input/output interface 240 sends instructions output from processor 210 to controller 300 . The command instructs the controller 300 to vibrate, output sound, emit light, or the like. Upon receiving the command, the controller 300 performs any one of vibration, sound output, or light emission according to the command.

The communication interface 250 is connected to the network 2 and communicates with other computers connected to the network 2 (for example, the server 600). In one aspect, the communication interface 250 is implemented as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication) or other wireless communication interface. be done. Communication interface 250 is not limited to the one described above.

In one aspect, processor 210 accesses storage 230, loads one or more programs stored in storage 230 into memory 220, and executes the sequences of instructions contained in the programs. The one or more programs may include an operating system of computer 200, an application program for providing virtual space, game software executable in virtual space, and the like. Processor 210 sends a signal for providing virtual space to HMD 120 via input/output interface 240 . HMD 120 displays an image on monitor 130 based on the signal. In this embodiment, the processor 210 can function as a generation unit 211 , an output unit 212 , an acquisition unit 213 , and a change unit 214 by reading a program 231 stored in the storage 230 . The function of each of these units will be detailed in later paragraphs.

In the example shown in FIG. 2, computer 200 is provided outside HMD 120, but computer 200 may be built into HMD 120 in another aspect. As an example, a portable information communication terminal (for example, a smart phone) including monitor 130 may function as computer 200 .

The computer 200 may have a configuration that is commonly used by a plurality of HMDs 120 . According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so each user can enjoy the same application as other users in the same virtual space.

In one embodiment, in the HMD system 100, a real coordinate system, which is a coordinate system in real space, is set in advance. The real coordinate system has three reference directions (axes) parallel to the vertical direction in the real space, the horizontal direction perpendicular to the vertical direction, and the front-rear direction perpendicular to both the vertical and horizontal directions. The horizontal direction, vertical direction (vertical direction), and front-rear direction in the real coordinate system are defined as x-axis, y-axis, and z-axis, respectively. More specifically, in the real coordinate system, the x-axis is parallel to the horizontal direction of the real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the physical space.

In one aspect, HMD sensor 410 includes an infrared sensor. The presence of HMD 120 is detected when the infrared sensor detects infrared rays emitted from each light source of HMD 120 . The HMD sensor 410 further detects the position and inclination (orientation) of the HMD 120 in the physical space according to the movement of the user 5 wearing the HMD 120, based on the value of each point (each coordinate value in the real coordinate system). do. More specifically, the HMD sensor 410 can detect temporal changes in the position and tilt of the HMD 120 using each value detected over time.

Each tilt of HMD 120 detected by HMD sensor 410 corresponds to each tilt of HMD 120 around three axes in the real coordinate system. The HMD sensor 410 sets the uvw visual field coordinate system to the HMD 120 based on the tilt of the HMD 120 in the real coordinate system. The uvw visual field coordinate system set in the HMD 120 corresponds to the viewpoint coordinate system when the user 5 wearing the HMD 120 sees an object in virtual space.

[uvw visual field coordinate system]
The uvw visual field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw viewing coordinate system set in the HMD 120 according to one embodiment. HMD sensor 410 detects the position and tilt of HMD 120 in the real coordinate system when HMD 120 is activated. Processor 210 sets the uvw viewing coordinate system to HMD 120 based on the detected values.

As shown in FIG. 3 , the HMD 120 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 5 wearing the HMD 120 . More specifically, the HMD 120 rotates the horizontal direction, the vertical direction, and the front-rear direction (x-axis, y-axis, and z-axis) defining the real coordinate system by tilts around each axis of the HMD 120 within the real coordinate system. The three directions newly obtained by tilting around the axes are set as the pitch axis (u-axis), yaw-axis (v-axis), and roll-axis (w-axis) of the uvw visual field coordinate system in the HMD 120 .

In one aspect, when user 5 wearing HMD 120 is standing upright and looking straight ahead, processor 210 sets HMD 120 to a uvw viewing coordinate system that is parallel to the real coordinate system. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the real coordinate system are the pitch axis (u-axis) and yaw-axis (v-axis) of the uvw visual field coordinate system in the HMD 120. , and the roll axis (w-axis).

After the uvw visual field coordinate system is set on the HMD 120 , the HMD sensor 410 can detect the tilt of the HMD 120 in the set uvw visual field coordinate system based on the movement of the HMD 120 . In this case, the HMD sensor 410 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 120 in the uvw visual field coordinate system as the tilt of the HMD 120 . The pitch angle (θu) represents the tilt angle of the HMD 120 around the pitch axis in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 120 around the yaw axis in the uvw visual field coordinate system. A roll angle (θw) represents the tilt angle of the HMD 120 around the roll axis in the uvw visual field coordinate system.

The HMD sensor 410 sets the uvw visual field coordinate system in the HMD 120 after the HMD 120 moves based on the detected inclination of the HMD 120 to the HMD 120 . The relationship between the HMD 120 and the uvw visual field coordinate system of the HMD 120 is always constant regardless of the position and tilt of the HMD 120 . When the position and tilt of the HMD 120 change, the position and tilt of the uvw visual field coordinate system of the HMD 120 in the real coordinate system change in conjunction with the change in the position and tilt.

In one aspect, HMD sensor 410 detects HMD 120 based on the infrared light intensity and the relative positional relationship between a plurality of points (for example, the distance between points) obtained based on the output from the infrared sensor. position in the physical space may be specified as a relative position with respect to the HMD sensor 410 . Processor 210 may determine the origin of the uvw visual field coordinate system of HMD 120 in physical space (real coordinate system) based on the specified relative position.

[Processor function]
As shown in FIG. 2 , in this embodiment, the processor 210 can function as a generation unit 211 , an output unit 212 , an acquisition unit 213 , and a change unit 214 by reading a program 231 . Each of these functions will be described below.

The generation unit 211 generates a presentation image to be presented to the user 5 based on the virtual camera arranged inside the celestial sphere on which the image is projected. The presentation image can also be changed according to the change in the direction of the virtual camera, the positional relationship between the virtual camera and the celestial sphere, or the viewing angle of the virtual camera. The presentation image may be changed, for example, by drawing the image to be output to the HMD 120 at that time in real time in accordance with the change in the direction of the virtual camera or the like. Alternatively, the presentation image may be changed, for example, by drawing an image outside the field of view of the virtual camera in advance, and changing the range of the image to be output to the HMD 120 at that time in accordance with the change in the direction of the virtual camera, etc. good. An image projected onto the celestial sphere can be generated based on data acquired from the server 600, the external device 700, or the storage 230, for example. Note that the celestial sphere is not a large one whose radius is set to infinity, but has a radius of a predetermined length.

The output unit 212 outputs the presentation image generated by the generation unit 211 to the HMD 120, which is a display device. A display device for outputting a presentation image is not limited to the HMD 120, and may be another display device. The presentation image output to the display device is provided to the user 5 as a panoramic image.

Acquisition unit 213 acquires motion information regarding movement of at least part of the body of user 5 detected by the sensor unit. The sensor unit is not particularly limited as long as it can detect movement of at least part of the body of the user 5 . The sensor unit is configured by one or more of the gaze sensor 140, the sensor 190, the HMD sensor 410, and the motion sensor 420, for example. Note that at least a part of the body of the user 5 includes, for example, one or more of the head, hands, fingers, and eyes of the user 5, and particularly if it is a part whose movement can be grasped by the sensor unit. Not limited.

The motion information may include, for example, tilt information regarding the tilt of at least part of the body of the user 5 with respect to a predetermined reference plane. Examples of the predetermined reference plane include a plane containing any two of the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the uvw visual field coordinate system, A plane including the pitch axis (u-axis) and the roll axis (w-axis) is preferable, but not limited to this. The motion information may also include, for example, first motion information regarding motion of at least part of the body of the user 5 in a direction parallel to the reference plane described above.

The changing unit 214 changes the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information acquired by the acquiring unit 213 . By this change, the image generated by the generation unit 211 is enlarged or reduced. The changing unit 214 changes the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera, for example, based on one or more of the tilt information and the first motion information.

In one aspect, the changing unit 214 moves the position of the virtual camera within the celestial sphere. In this case, it is preferable not to change the radius of the celestial sphere, but it may be changed. Also, in a certain aspect, the changing unit 214 changes the size (radius) of the celestial sphere. In this case, it is preferable not to change the position of the virtual camera, but it may be changed. When the changing unit 214 changes the viewing angle of the virtual camera, it is preferable not to change the position of the virtual camera and the size of the celestial sphere, but they may be changed. Note that the changing unit 214 can change the orientation of the virtual camera based on the motion information acquired by the acquiring unit 213 .

The change in the positional relationship or viewing angle as described above by the change unit 214 is preferably executed when the motion information satisfies a predetermined condition. An example of the predetermined condition is that at least part of the body of the user 5 moves beyond a first region set around at least part of the body. The first area is an area provided for the purpose of suppressing the above change due to a small action not intended to enlarge or reduce the presentation image. The size of the first region is not particularly limited as long as the above purpose can be achieved, and can be appropriately set according to the part of the body whose motion is to be detected by the sensor unit. The size of the first region may have a preset value based on the age, sex, height, etc. of the user 5, or may be arbitrarily set by the user 5. FIG. When the movement of the head of the user 5 is to be detected, the first area may be, for example, a spherical or circular area centered on the head of the user 5 and having a radius of 5 to 150 cm. This is because the range differs depending on whether the user is sitting or standing. It is conceivable to set the upper limit to 50 cm for the sitting position and 150 cm for the standing position. It is also possible to set the range according to the scale of the space where the user is.

Also, the predetermined condition may be, for example, that the amount of change in motion is equal to or greater than a predetermined threshold. The amount of change in motion is, for example, the amount of movement of at least part of the body of the user 5 per unit time. This condition is also set for the purpose of suppressing the above change due to a small action not intended to enlarge or reduce the presentation image. Therefore, the magnitude of the predetermined threshold value is not particularly limited as long as it is within a range where the above object can be achieved, and can be appropriately set according to the part of the body targeted for motion detection by the sensor unit. The size of the predetermined threshold value may have a preset value based on the age, sex, height, etc. of the user 5, or may be arbitrarily set by the user 5. FIG. When the movement of the head of the user 5 is to be detected, the predetermined threshold value may range from 0.1 to 5 to 50 cm per second, for example. Further, in addition to or instead of the amount of change in motion, the predetermined condition may be that the speed or acceleration of motion is equal to or greater than a predetermined threshold.

Also, the predetermined condition may be that the magnitude of the tilt indicated by the tilt information included in the motion information is within a predetermined range. That is, if the magnitude of the tilt indicated by the tilt information is outside the predetermined range, the change in positional relationship or viewing angle as described above may not be executed. Specifically, when the magnitude of the tilt indicated by the tilt information is within a predetermined range, the changing unit 214 moves at least part of the body of the user 5 in the direction parallel to the reference plane (first The positional relationship or the viewing angle may be changed based on the motion information). This condition is set for the purpose of preventing the above change from being made during an operation that does not seem to be intended for enlarging or reducing the presentation image. Actions that are not intended to enlarge or reduce the presentation image include, for example, actions such as turning the face straight up or straight down for stretching. The predetermined range is not particularly limited as long as the above object can be achieved, and can be appropriately set according to the part of the body whose motion is to be detected by the sensor unit.

[Operation processing]
Next, operation processing by the HMD system 100 according to this embodiment will be described. A case will be described below where motion information regarding the movement of the head of the user 5 is acquired by the sensor unit, but the present disclosure is not limited to this example. Further, the order of each process constituting the flowcharts described in this specification may be random as long as there is no contradiction or inconsistency in the contents of the process, and the processes may be executed in parallel.

FIG. 4 is a flowchart illustrating an example of operational processing according to an embodiment. First, in step S1, processor 210 generates a virtual space. A virtual space is generated based on data acquired from the storage 230, the server 600, or the external device 700, for example. The virtual space includes a virtual camera and a celestial sphere having a predetermined length and radius. The virtual space may contain other virtual objects.

At step S2, the processor 210 projects the image onto the celestial sphere. In step S3, the processor 210 generates a presentation image (panorama image) to be presented to the user 5 based on the virtual camera arranged inside the celestial sphere on which the image is projected. Next, in step S4, processor 210 outputs the presentation image generated in step S3 to HMD 120, which is a display device.

In step S5, the processor 210 acquires motion information regarding the movement of the head of the user 5 detected by the sensor unit. Step S5 can be repeatedly performed at predetermined time intervals. If the motion information acquired in step S5 does not satisfy the predetermined condition (No in step S6), the process of step S7, which will be described later, is not executed. The predetermined condition in step S5 is preferably one or more of the conditions described above.

If the motion information acquired in step S5 satisfies a predetermined condition (Yes in step S6), in step S7 processor 210 determines the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information. change and exit. The presentation image is enlarged or reduced by the processing in step S7.

Hereinafter, some processes shown in the flowchart of FIG. 4 will be described more specifically with reference to FIGS. 5 to 9. FIG. FIG. 5 is a schematic diagram showing an example of enlargement and reduction of a presentation image 500 according to an embodiment. A presentation image 500 in FIG. 5 is an image output to the HMD 120 in step S4 in FIG. In the example of FIG. 5 , a presentation image 500 includes a character 501 and an object 502 . Presented images 500A and 500B are obtained by enlarging or reducing presenting image 500 .

A presentation image 500A is obtained by enlarging the presentation image 500 as a result of executing the process of step S7 based on the motion information A acquired in step S5. Character 501 and object 502 in presentation image 500A are enlarged more than character 501 and object 502 in presentation image 500 . As a result of the enlargement, the range included in the presentation image 500A among the images projected onto the celestial sphere is narrower than the presentation image 500. FIG. The motion information A is not particularly limited, but includes, for example, information indicating that the user 5 has moved the head toward the current viewing direction (hereinafter also referred to as "forward").

A presentation image 500B is obtained by reducing the presentation image 500 as a result of executing the process of step S7 based on the motion information B acquired in step S5. The character 501 and the object 502 in the presentation image 500B are smaller than the character 501 and the object 502 in the presentation image 500. FIG. In addition, as a result of the reduction, the range included in the presentation image 500B among the images projected on the celestial sphere was wider than the presentation image 500, and the presentation image 500B was not included in the presentation image 500. Object 503 is included. The motion information B is not particularly limited, but includes, for example, information indicating that the user 5 has moved the head to the side opposite to the current line of sight direction (hereinafter also referred to as "backward side").

6 to 8 relate to the process of step S7 in FIG. FIG. 6 is a schematic diagram showing an example of changing the positional relationship between the virtual camera 602 and the celestial sphere 601 according to an embodiment. Specifically, FIG. 6 shows an example in which the presentation image 500 is enlarged or reduced by moving the position of the virtual camera 602 within the celestial sphere 601 in step S7. In the example of FIG. 6, the radius r1 of the celestial sphere 601 and the viewing angle θ1 of the virtual camera 602 are not changed. may be configured to change at least one of

A state 610 shown in FIG. 6 shows an example of the positional relationship between the celestial sphere 601 and the virtual camera 602 in step S4. In state 610 , virtual camera 602 is positioned at position P 1 , which is the center of celestial sphere 601 . The image output to the HMD 120 in state 610 is, for example, the presentation image 500 shown in FIG.

A state 610A is a state after the process of step S7 is executed based on the motion information A acquired in the state 610. FIG. In state 610A, virtual camera 602 is positioned at position P2. Position P2 is on the front side of position P1. That is, in step S7, the virtual camera 602 is moved forward and the positional relationship with the celestial sphere 601 is changed. In other words, the distance between the virtual camera 602 and the celestial sphere 601 is reduced on the front side. Therefore, the image output to the HMD 120 in state 610A is an enlarged version of the image output in state 610, and is, for example, presentation image 500A shown in FIG.

A state 610B is a state after the process of step S7 is executed based on the motion information B acquired in the state 610. FIG. In state 610B, virtual camera 602 is positioned at position P3. Position P3 is on the rear side of position P1. That is, in step S7, the virtual camera 602 is moved backward to change the positional relationship with the celestial sphere 601. FIG. In other words, the distance between the virtual camera 602 and the celestial sphere 601 is increased on the front side. Therefore, the image output to the HMD 120 in state 610B is a reduced version of the image output in state 610, and is, for example, presentation image 500B shown in FIG.

FIG. 7 is a schematic diagram showing an example of changing the positional relationship between virtual camera 602 and celestial sphere 601 according to an embodiment. Specifically, FIG. 7 shows an example of enlarging or reducing the presentation image 500 by changing the size of the radius r1 of the celestial sphere 601 in step S7. In the example of FIG. 7, the position P1 and the viewing angle θ1 of the virtual camera 602 are not changed. may be configured to change

The state 610 shown in FIG. 7 is the same as the state 610 shown in FIG. 6, so the description is omitted. A state 710A is a state after the processing of step S7 is executed based on the motion information A acquired in the state 610. FIG. Celestial sphere 601A in state 710A has radius r2 that is smaller than radius r1 of celestial sphere 601. FIG. That is, in step S7, the celestial sphere 601 is reduced to a celestial sphere 601A. As a result, the distance between the virtual camera 602 and the celestial sphere 601A is shorter than the distance between the virtual camera 602 and the celestial sphere 601 on the front side. Therefore, the image output to the HMD 120 in state 710A is an enlarged version of the image output in state 610, and is, for example, presentation image 500A shown in FIG.

A state 710B is a state after the process of step S7 is executed based on the motion information B acquired in the state 610. FIG. Celestial sphere 601B in state 710B has a radius r3 that is greater than radius r1 of celestial sphere 601 . That is, in step S7, the celestial sphere 601 is enlarged to become the celestial sphere 601B. As a result, the distance between the virtual camera 602 and the celestial sphere 601B is longer than the distance between the virtual camera 602 and the celestial sphere 601 on the front side. Therefore, the image output to the HMD 120 in state 710B is a reduced version of the image output in state 610, and is, for example, presentation image 500B shown in FIG.

FIG. 8 is a schematic diagram showing an example of changing the positional relationship between the virtual camera and the celestial sphere according to an embodiment. Specifically, FIG. 8 shows an example in which the presentation image 500 is enlarged or reduced by changing the viewing angle θ1 of the virtual camera 602 in step S7. In the example of FIG. 8, the radius r1 of the celestial sphere 601 and the position P1 of the virtual camera 602 are not changed. It may be configured to change one.

The state 610 shown in FIG. 8 is the same as the state 610 shown in FIG. 6, so description thereof will be omitted. A state 810A is a state after the processing of step S7 is executed based on the motion information A acquired in the state 610. FIG. Viewing angle θ 2 of virtual camera 602 in state 810A is smaller than viewing angle θ 1 of virtual camera 602 in state 610 . That is, in step S7, the viewing angle of the virtual camera 602 is changed so as to be smaller than the viewing angle θ1. As a result, the range of the celestial sphere 601 included in the field of view of the virtual camera 602 is narrowed, and the narrow range is enlarged and provided to the user 5 . That is, the image output to HMD 120 in state 810A is an enlarged version of the image output in state 610, and is, for example, presentation image 500A shown in FIG.

A state 810B is a state after the processing of step S7 is executed based on the motion information B acquired in the state 610. FIG. Viewing angle θ 3 of virtual camera 602 in state 810B is greater than viewing angle θ 1 of virtual camera 602 in state 610 . That is, in step S7, the viewing angle of the virtual camera 602 is changed to be larger than the viewing angle θ1. As a result, the range of the celestial sphere 601 included in the field of view of the virtual camera 602 is widened, and the wide range is reduced and provided to the user 5 . That is, the image output to the HMD 120 in state 810B is a reduced version of the image output in state 610, and is, for example, presentation image 500B shown in FIG.

FIG. 9 relates to the processing from step S5 to step S7 in FIG. FIG. 9 is a schematic diagram showing an example of the relationship between virtual space 901A and real space 901B according to an embodiment. In the example of FIG. 9, virtual camera 602 exists at position P1 and user 5 exists at position Q1. The sensor unit detects, for example, the movement of the head of the user 5 in the physical space 901B and transmits it to the computer 200 as movement information. The processor 210 changes, for example, one or more of the position and viewing angle of the virtual camera 602 in the virtual space 901A and the radius r1 of the celestial sphere 601 when the motion information satisfies a predetermined condition.

In the physical space 901B, a boundary line 904 is set around the position of the head of the user 5 (position Q1 in FIG. 9). The inside of the boundary line 904 is set as the first area. A boundary line 604 in the virtual space 901 A corresponds to the boundary line 904 . When the position of the head of the user 5 moves beyond the boundary line 904, it is determined in step S6 that the predetermined condition is satisfied, and the steps described above, such as moving the virtual camera 602 beyond the boundary line 604, for example, are performed. The process of S7 is executed. On the other hand, when the position of the head of the user 5 does not move beyond the boundary line 904, it is not determined in step S6 that the predetermined condition is satisfied, and the position of the virtual camera 602, for example, is not changed.

Note that in the example of FIG. 9, the boundary line 904 is, for example, the outer edge of a sphere centered at the position of the head of the user 5 and having a radius R5. Boundary line 904 may be, for example, the outer edge of a circle of radius R5 on the reference plane. The boundary line 904 may be at the same position until, for example, an action is taken by the user 5 to reset its position. Also, the boundary line 904 may be set around the position of the head of the user 5 at predetermined time intervals or timing such as when the movement of the head of the user 5 stops, for example. Boundary 604 is, for example, the outer edge of a sphere or circle centered at the position of virtual camera 602 and having radius r5. The position of boundary line 604 is changed according to boundary line 904 .

A boundary line 603 is set in the virtual space 901A. The boundary line 603 is provided, for example, inside the inner edge of the celestial sphere 601, and is the outer edge of a sphere whose center is the position P1 and whose radius is r4 (<r1). An area inside the boundary line 603 is set as a movable area in which the virtual camera 602 can move. A boundary line 903 in the virtual space 901 A corresponds to the boundary line 603 , but even if the user 5 moves beyond the boundary line 903 , the virtual camera 602 does not move outside the boundary line 603 . Note that the ratio (r4/R4) between the radius r4 of the boundary line 603 and the radius R4 of the boundary line 903 is a scale indicating how much the virtual camera 602 moves with respect to the distance the user 5 has moved. The ratio of the radius r4 to the radius R4 is not particularly limited, and can be appropriately set according to the type of image to be projected, user's selection, and the like.

Here, if step S7 includes a process of changing the radius r1 of the celestial sphere 601, it is preferable that the minimum and maximum values of the radius r1 are set. That is, for example, even if the radius r1 of the celestial sphere 601 is set to decrease as the user 5 moves forward, after the radius r1 reaches the minimum value, the user 5 moves further forward. It is preferred that no change in radius is made when advancing to . Similarly, for example, even if the radius r1 of the celestial sphere 601 is set to increase as the user 5 moves backward, after the radius r1 reaches the maximum value, the user 5 moves further backward. Even so, preferably no change in radius is made. The minimum and maximum values of the radius r1 are not particularly limited, and can be appropriately set according to the type of image to be projected, user's selection, and the like.

Similarly, if step S7 includes processing for changing the viewing angle θ1 of the virtual camera 602, it is preferable that the minimum and maximum values of the viewing angle θ1 are set. That is, for example, even if the viewing angle θ1 is set to decrease as the user 5 moves forward, after the viewing angle θ1 reaches the minimum value, the user 5 moves further forward. It is preferable that no change in viewing angle is made even if it advances. Similarly, for example, even if the viewing angle θ1 of the celestial sphere 601 is set to increase toward the rear side, after the viewing angle θ1 reaches the maximum value, the user 5 may move further toward the rear side. It is preferable that the viewing angle is not changed even if the viewing angle is advanced to . The minimum and maximum values of the viewing angle θ1 are not particularly limited, and can be appropriately set according to the type of image to be projected, user's selection, and the like.

The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit the interpretation of the present invention. It goes without saying that the present invention can be modified and improved without departing from its spirit, and that equivalents thereof are included in the present invention.

[Additional notes]
The contents of the present disclosure are listed below.

(Item 1)
A program executed on a computer with a processor,
The program causes the processor to:
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
a step of acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
program.
This improves usability when providing a panoramic image to the user. Specifically, the user can enlarge or reduce the panoramic image by moving at least a part of the body. Moreover, it becomes possible to provide a new user experience by these.

(Item 2)
the changing step includes moving the position of the virtual camera within the celestial sphere;
The program of item 1.
As a result, the position of the virtual camera moves according to the user's motion, so the user can intuitively enlarge or reduce the panorama image.

(Item 3)
a movable area in which the virtual camera can move is set inside the inner edge of the celestial sphere;
A program according to item 2.
This prevents the virtual camera from moving beyond the celestial sphere, and prevents the user from seeing the panoramic image or from seeing the panoramic image because it is too enlarged.

(Item 4)
the modifying step includes modifying the size of the celestial sphere;
The program of item 1.
As a result, the size of the celestial sphere changes according to the user's motion, so the user can intuitively enlarge or reduce the panorama image.

(Item 5)
The modifying step is performed when at least part of the body moves beyond a first region set around at least part of the body.
The program according to any one of items 1 to 4.
As a result, it is possible to prevent the panorama image from being enlarged or reduced by a small operation that is not intended to enlarge or reduce the panoramic image, and as a result, usability can be further improved.

(Item 6)
The changing step is executed when the change amount of the movement is equal to or greater than a predetermined threshold,
The program according to any one of items 1 to 4.
As a result, it is possible to prevent the panorama image from being enlarged or reduced by a small operation that is not intended to enlarge or reduce the panoramic image, and as a result, usability can be further improved.

(Item 7)
the motion information includes tilt information about the tilt of at least part of the user's body with respect to a predetermined reference plane;
The modifying step includes modifying the positional relationship or the viewing angle based on the movement in a direction parallel to the reference plane;
the changing step is not performed if the magnitude of the slope is outside a predetermined range;
The program according to any one of items 1 to 6.
As a result, it is possible to prevent the panorama image from being enlarged or reduced by an operation that is not intended to enlarge or reduce the panoramic image, and as a result, usability can be further improved.

(Item 8)
The display device is a head-mounted device,
The detecting step includes detecting movement of the user's head by the sensor unit,
The program according to any one of items 1 to 7.
This aspect is a preferred application example of the present disclosure.

(Item 9)
An information processing method executed in a computer having a processor,
The information processing method includes:
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
a step of acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
Information processing methods.
This improves usability when providing a panoramic image to the user. Specifically, by moving at least a part of the user's body, the user can view the presented image in an enlarged or reduced size. Moreover, it becomes possible to provide a new user experience by these.

(Item 10)
An information processing device comprising a processor,
The processor
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
Acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
Information processing equipment.
This improves usability when providing a panoramic image to the user. Specifically, by moving at least a part of the user's body, the user can view the presented image in an enlarged or reduced size. Moreover, it becomes possible to provide a new user experience by these.

100: HMD system 110, 110A, 110B, 110C, 110D: HMD set 120: HMD
200: Computer 210: Processor 220: Memory 230: Storage 600: Server 700: External device

Claims (10)

A program executed on a computer with a processor,
The program causes the processor to:
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
a step of acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
program. the changing step includes moving the position of the virtual camera within the celestial sphere;
A program according to claim 1. a movable area in which the virtual camera can move is set inside the inner edge of the celestial sphere;
3. A program according to claim 2. the modifying step includes modifying the size of the celestial sphere;
A program according to claim 1. The modifying step is performed when at least part of the body moves beyond a first region set around at least part of the body.
A program according to any one of claims 1 to 4. The changing step is executed when the change amount of the movement is equal to or greater than a predetermined threshold,
A program according to any one of claims 1 to 4. the motion information includes tilt information about the tilt of at least part of the user's body with respect to a predetermined reference plane;
The modifying step includes modifying the positional relationship or the viewing angle based on the movement in a direction parallel to the reference plane;
the changing step is not performed if the magnitude of the slope is outside a predetermined range;
A program according to any one of claims 1 to 6. The display device is a head-mounted device,
The sensor unit is capable of detecting movement of the user's head,
A program according to any one of claims 1 to 7. An information processing method executed in a computer having a processor,
The information processing method includes:
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
a step of acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
Information processing methods. An information processing device comprising a processor,
The processor
generating a presentation image to be presented to a user based on a virtual camera placed inside the celestial sphere on which the image is projected;
outputting the presentation image to a display device;
Acquiring motion information regarding movement of at least part of the user's body detected by a sensor unit;
changing the positional relationship between the virtual camera and the celestial sphere or the viewing angle of the virtual camera based on the motion information;
Information processing equipment.

Images