JP7030726B2 - Programs and information processing devices run on computers to provide virtual reality - Google Patents

Description

The present disclosure relates to the provision of video content in virtual reality space, and more specifically to the control of shooting of video content in virtual reality space.

A technique for providing virtual reality using a head-mounted device is known. For example, Japanese Patent No. 6124985 (Patent Document 1) states, "A video content distribution system capable of improving user satisfaction with a video content distribution service when the user views the same video content from different viewpoints. And the content management server ”(see [Summary]).

Japanese Patent No. 6124985

When video content such as video content is presented in virtual space, the user may wish to shoot, as in real space. Therefore, there is a need for technology for shooting video content in virtual space.

The present disclosure has been made to solve the above-mentioned problems, and an object in a certain aspect is to provide a technique capable of easily realizing shooting of moving image content to be reproduced in a virtual space.

According to certain embodiments, a program running on a computer is provided to provide virtual reality. This program gives the computer a step to define the virtual space provided by the headmount device connected to the computer, a step to reproduce the image in the virtual space, and a step to detect that the image is taken in the virtual space. In response to the detection, the step of stopping the reproduction of the image is executed.

In a certain aspect, the user can easily shoot the moving image content being played in the virtual space.

The above and other objects, features, aspects and advantages of the invention will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

It is a figure which shows the outline of the structure of the HMD system 100 according to a certain embodiment. It is a block diagram which shows an example of the hardware composition of the computer 200 which follows one aspect. It is a figure which conceptually represents the uvw field coordinate system set in the HMD 110 according to a certain embodiment. It is a figure which conceptually represents one aspect which expresses the virtual space 2 according to a certain embodiment. It is a figure which showed the head of the user 190 who wears an HMD 110 according to a certain embodiment from the top. It is a figure which shows the YZ cross section which looked at the view area 23 from the X direction in the virtual space 2. It is a figure which shows the XZ cross section which looked at the view area 23 from the Y direction in the virtual space 2. It is a figure which shows the schematic structure of the controller 160 according to a certain embodiment. FIG. 3 is a block diagram showing a computer 200 according to an embodiment as a module configuration. It is a sequence chart which shows a part of the processing performed in the HMD system 100 according to a certain embodiment. It is a block diagram which shows the structure of a shooting control module 270. It is a flowchart which shows a part of the processing performed by the processor 10 of the computer 200 according to a certain embodiment. It is a figure which shows one aspect of storing data in the memory module 240 according to a certain embodiment. It is a figure which shows the structure of the photographing data 1400 according to a certain embodiment. It is a figure which shows the correspondence between the image displayed on a monitor 112, and the arrangement of each object in virtual space 2 at the time of the display. It is a flowchart which shows a part of the process which a processor 10 of a computer 200 executes. It is a figure which shows the correspondence between the image displayed on a monitor 112, and the arrangement of each object in virtual space 2 at the time of the display.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated.

Hereinafter, embodiments will be described with reference to the drawings. In the following description, the same parts and components are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of these will not be repeated.

[HMD system configuration]
The configuration of the HMD (Head-Mounted Device) system 100 will be described with reference to FIG. FIG. 1 is a diagram illustrating an outline of a configuration of an HMD system 100 according to an embodiment. In certain aspects, the HMD system 100 is provided as a home system or a business system.

The HMD system 100 includes an HMD 110, an HMD sensor 120, and a controller 16.
0 and a computer 200 are provided. The HMD 110 includes a monitor 112 and a gaze sensor 1.
40, a speaker 115, and a microphone 119 are included. The controller 160 may include a motion sensor 130.

In one aspect, the computer 200 is the Internet or other network 19
It is possible to connect to the server 150 and other computers connected to the network 19. In another aspect, the HMD 110 may include the sensor 114 instead of the HMD sensor 120.

The server 150 includes a processor 151, a memory 152, and a communication interface 153.
And include. The server 150 is realized by a computer having a well-known configuration. Processor 151 executes the instruction. The memory 152 is a RAM (Random Access Memory).
Other volatile memories. The communication interface 153 communicates with the computer 200, the user terminals 201A, 201N, and the like. The storage 154 holds data received by the server 150 from the computer 200 or other external device or data generated by the processor 151. The storage 154 may be, for example, a hard disk or SSD (So).
lid State Disc) Can be realized by other non-volatile storage devices.

The HMD 110 is attached to the head of the user 190, and the virtual space 2 is used by the user 190 during operation.
Can be provided to. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the monitor 112, respectively. When each eye of the user 190 visually recognizes the respective image, the user 190 can recognize the image as a three-dimensional image based on the parallax of both eyes.

The monitor 112 is realized, for example, as a non-transparent display device. In one aspect, the monitor 112 is arranged in the body of the HMD 110 so as to be located in front of both eyes of the user 190. Therefore, the user 190 can immerse himself / herself in the virtual space 2 when he / she visually recognizes the three-dimensional image displayed on the monitor 112. In certain embodiments, the virtual space 2 includes, for example, a background, an object operable by the user 190, and an image of a menu selectable by the user 190. If the configuration is such that a plurality of computers 200 can experience a virtual experience in one virtual space 2 by passing a signal based on the operation of each user, the avatar object corresponding to each user is presented in the virtual space 2. Will be done.

The object is a virtual object existing in the virtual space 2. In one aspect, the object may be an avatar object corresponding to the user, virtual accessories and clothing worn by the avatar object, a virtual panel imitating a panel showing information about the user, a virtual letter imitating a letter, and a post. Includes imitated virtual posts. Further, the avatar object is a character that symbolizes the user 190 in the virtual space 2, and includes, for example, a human type, an animal type, a robot type, and the like. Objects come in many shapes.
The user 190 selects a favorite object from predetermined objects in the virtual space 2
You may present it to the virtual space 2 or you may present the object you created to the virtual space 2.

In certain embodiments, the monitor 112 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor included in a so-called smartphone or other information display terminal.

In certain aspects, the monitor 112 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In other aspects, monitor 112
May be configured to display the image for the right eye and the image for the left eye as a unit. In this case, the monitor 112 includes a high speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

The gaze sensor 140 detects the direction (line-of-sight direction) in which the line of sight of the right eye and the left eye of the user 190 is directed. Detection in that direction is achieved, for example, by a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In certain aspects, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the angle of rotation of each eyeball by receiving the reflected light from the cornea and the iris with respect to the irradiation light. .. The gaze sensor 140 can detect the line-of-sight direction of the user 190 based on each detected rotation angle.

The speaker 115 outputs the voice (utterance) corresponding to the voice data received from the computer 200 to the outside. The microphone 119 outputs voice data corresponding to the utterance of the user 190 to the computer 200. The user 190 can use the microphone 119 to speak to another user, while the speaker 115 can hear the voice (utterance) of the other user.

More specifically, when the user 190 speaks into the microphone 119, the user 19
The voice data corresponding to the utterance of 0 is input to the computer 200. Computer 200
Outputs the voice data to the server 150 via the network 19. Server 15
0 outputs the voice data received from the computer 200 to another computer 200 via the network 19. The other computer 200 outputs the voice data received from the server 150 to the speaker 115 of the HMD 110 worn by another user. As a result, other users can hear the voice of the user 190 through the speaker 115 of the HMD 110. Similarly, utterances from other users are output from the speaker 115 of the HMD 110 worn by the user 190.

The computer 200 monitors an image that moves another avatar object corresponding to the other user according to the voice data received from the computer 200 of the other user.
Display on 12. For example, in a certain situation, the computer 200 displays an image that moves the mouth of another avatar object on the monitor 112 as if it were a virtual space 2.
The virtual space 2 is expressed as if the avatar objects are talking to each other. By transmitting and receiving voice data between the plurality of computers 200 in this way, a conversation (chat) between a plurality of users is realized in one virtual space 2.

The HMD sensor 120 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. The HMD sensor 120 uses this function to detect the position and tilt of the HMD 110 in the real space.

In another aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and tilt of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include the sensor 114 as the position detector instead of the HMD sensor 120. The HMD 110 uses the sensor 114 and the HMD 11
The position and tilt of 0 itself can be detected. For example, if the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 may be H.
Instead of the MD sensor 120, any of these sensors can be used to detect its position and tilt. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in real space over time. HMD11
0 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle.

Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance. Further, the visual field image may include a configuration for presenting the real space as a part of the image constituting the virtual space 2. For example, an image taken by a camera mounted on the HMD 110 may be superimposed on a part of the field image and displayed, or by setting a high transmittance of a part of the transmissive display device, the field image may be displayed. The real space may be visible from a part.

The server 150 may send the program to the computer 200. In other aspects,
The server 150 may communicate with another computer 200 to provide virtual reality to the HMD 110 used by another user. For example, in an amusement facility, when a plurality of users play a participatory game, each computer 200 communicates a signal based on the operation of each user with another computer 200, and the plurality of users are common in the same virtual space 2. Allows you to enjoy the game. Further, as described above, the plurality of computers 200 transmit and receive signals based on the actions of each user, so that a plurality of users can enjoy a conversation in one virtual space 2.

The controller 160 receives an instruction input from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be grippable by the user 190. In another aspect, the controller 160 is configured to be wearable on a part of the body or clothing of the user 190. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on the signal sent from the computer 200. In another aspect, the controller 160 accepts operations given by the user 190 to control the position and movement of objects placed in the space that provides virtual reality.

The motion sensor 130 is attached to the hand of the user 190 in a certain aspect.
Detects the movement of the user 190's hand. For example, the motion sensor 130 detects the rotation speed, the number of rotations, and the like of the hand. Data representing the detection result of the hand movement of the user 190 obtained by the motion sensor 130 (hereinafter, also referred to as detection data) is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove-shaped controller 160. In one embodiment, for safety in real space, controller 160
Is preferably attached to something that does not easily fly by being attached to the hand of the user 190, such as a glove type. In another aspect, a sensor not attached to the user 190 may detect the movement of the user 190's hand. For example, the signal of the camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190.
The motion sensor 130 and the computer 200 are connected to each other by wire or wirelessly. In the case of wireless communication, the communication mode is not particularly limited, and for example, Bluetooth (
Registered trademark) Other known communication methods are used.

In another aspect, the HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 2.

In still other aspects, the HMD system 100 may include a communication circuit for connecting to the Internet or a telephone function for connecting to a telephone line.

[Computer hardware configuration]
The computer 200 according to the present embodiment will be described with reference to FIG. Figure 2 shows
It is a block diagram which shows an example of the hardware composition of the computer 200 which follows one aspect. The computer 200 includes a processor 10, a memory 11, a storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on the signal given to the computer 200 or based on the condition that a predetermined condition is satisfied. In one aspect, the processor 10
Is a CPU (Central Processing Unit), MPU (Micro Processor Unit), FP
It is realized as a GA (Field-Programmable Gate Array) and other devices.

The memory 11 temporarily stores programs and data. The program is loaded from storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In a certain situation, the memory 11
Is realized as RAM (Random Access Memory) or other volatile memory.

Storage 12 permanently retains programs and data. Storage 12
For example, it is realized as a ROM (Read-Only Memory), a hard disk device, a flash memory, or other non-volatile storage device. The program stored in the storage 12 is H.
The MD system 100 includes a program for providing the virtual space 2, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data and objects for defining the virtual space 2.

In another aspect, the storage 12 may be realized as a detachable storage device such as a memory card. In still other aspects, configurations may be used that use programs and data stored in an external storage device instead of the storage 12 built into the computer 200. With such a configuration, it becomes possible to collectively update programs and data in a situation where a plurality of HMD systems 100 are used, for example, in an amusement facility.

In certain embodiments, the input / output interface 13 communicates a signal with the HMD 110, the HMD sensor 120, or the motion sensor 130. In a certain aspect, the input / output interface 13 is a USB (Universal Serial Bus) interface, D.
VI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multi)
media Interface) Realized using other terminals. Input / output interface 1
3 is not limited to the above.

In certain embodiments, the input / output interface 13 further comprises a controller 16.
Can communicate with 0. For example, the input / output interface 13 receives an input of a signal output from the motion sensor 130. In another aspect, the input / output interface 13 is
The instruction output from the processor 10 is sent to the controller 160. The command is vibration,
Instruct the controller 160 to output audio, emit light, and the like. Upon receiving the command, the controller 160 executes either vibration, audio output, or light emission in response to the command.

The communication interface 14 is connected to the network 19 and is connected to another computer (for example, a server 150, another user's computer 2) connected to the network 19.
Communicate with (00, etc.). In one aspect, the communication interface 14 may be, for example, L.
AN (Local Area Network) or other wired communication interface, or WiFi
(Wireless Fidelity), Bluetooth®, NFC (Near Field Com)
munication) Realized as another wireless communication interface. The communication interface 14 is not limited to the above.

In one aspect, the processor 10 accesses the storage 12 and the storage 12
One or more programs stored in the memory 11 are loaded into the memory 11 and a series of instructions included in the program are executed. The one or more programs may include an operating system of the computer 200, an application program for providing the virtual space 2, game software that can be executed in the virtual space 2 using the controller 160, and the like. The processor 10 sends a signal for providing the virtual space 2 to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 112 based on the signal.

In the example shown in FIG. 2, the computer 200 is configured to be provided outside the HMD 110, but in other aspects, the computer 200 may be built in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including a monitor 112 may function as a computer 200.

Further, the computer 200 may have a configuration commonly used for a plurality of HMD 110s. According to such a configuration, for example, the same virtual space 2 can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space 2.

In one embodiment, the HMD system 100 has a preset global coordinate system. The global coordinate system is parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-back direction orthogonal to both the vertical direction and the horizontal direction, respectively.
It has one reference direction (axis). In this embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction of the real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the real space.

In one aspect, the HMD sensor 120 includes an infrared sensor. Infrared sensor is H
When the infrared rays emitted from each light source of the MD110 are detected, the presence of the HMD110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120
Each value detected over time can be used to detect temporal changes in the position and tilt of the HMD 110.

The global coordinate system is parallel to the coordinate system in real space. Therefore, the HMD sensor 120
Each tilt of the HMD 110 detected by means of each tilt around the three axes of the HMD 110 in the global coordinate system. The HMD sensor 120 is an HMD in the global coordinate system.
Based on the tilt of 110, the uvw field coordinate system is set to HMD110. The uvw field-of-view coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 sees an object in the virtual space 2.

[Uvw field coordinate system]
The uvw field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw field coordinate system set in the HMD 110 according to an embodiment. HMD sensor 12
0 detects the position and tilt of the HMD 110 in the global coordinate system when the HMD 110 is activated. Processor 10 HMDs the uvw field coordinate system based on the detected values.
Set to 110.

As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 190 wearing the HMD 110. More specifically, HMD1
Reference numeral 10 is to tilt the horizontal direction, the vertical direction, and the front-back direction (x-axis, y-axis, z-axis) that define the global coordinate system around each axis by the inclination around each axis of the HMD 110 in the global coordinate system. Three new directions obtained by uvw in HMD110
It is set as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the visual field coordinate system.

In a certain aspect, when the user 190 wearing the HMD 110 stands upright and is visually recognizing the front, the processor 10 HMDs the uvw field coordinate system parallel to the global coordinate system.
Set to 110. In this case, the horizontal direction (x-axis) and the vertical direction (x-axis) in the global coordinate system (
The y-axis) and the anteroposterior direction (z-axis) coincide with the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw field coordinate system in the HMD 110.

After the uvw field coordinate system is set to HMD 110, the HMD sensor 120 will be HMD 11
Based on the movement of 0, the inclination (change amount of inclination) of the HMD 110 in the set uvw visual field coordinate system can be detected. In this case, the HMD sensor 120 is set as the inclination of the HMD 110.
The pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw field coordinate system are detected. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

The HMD sensor 120 determines the HMD 110 based on the detected tilt angle of the HMD 110.
The uvw field-of-view coordinate system in the HMD 110 after the movement is set to the HMD 110. HM
The relationship between the D110 and the uvw field coordinate system of the HMD110 is always constant regardless of the position and tilt of the HMD110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

In one aspect, the HMD sensor 120 is based on the infrared light intensity obtained based on the output from the infrared sensor and the relative positional relationship between the points (eg, the distance between the points). The position of the above in the real space may be specified as a relative position with respect to the HMD sensor 120. Further, the processor 10 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

[Virtual space]
The virtual space 2 will be further described with reference to FIG. FIG. 4 is a diagram conceptually representing one aspect of expressing the virtual space 2 according to a certain embodiment. The virtual space 2 is 360 in the center 21.
It has an all-sky spherical structure that covers the entire direction. In FIG. 4, the celestial sphere in the upper half of the virtual space 2 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 2. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system defined in the virtual space 2. The computer 200 associates each partial image constituting the contents (still image, moving image, etc.) expandable in the virtual space 2 with each corresponding mesh in the virtual space 2, and the virtual space image visible to the user 190. The virtual space 2 in which 22 is expanded is provided to the user 190.

In a certain aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the XYZ coordinate system are defined as the X-axis, the Y-axis, and the Z-axis, respectively. Therefore, X in the XYZ coordinate system
The axis (horizontal direction) is parallel to the x-axis of the global coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the global coordinate system, and the Z-axis (front-back direction) of the XYZ coordinate system is. It is parallel to the z-axis of the global coordinate system.

When the HMD 110 is started, that is, in the initial state of the HMD 110, the virtual camera 1 is
It is arranged in the center 21 of the virtual space 2. The virtual camera 1 moves in the virtual space 2 in the same manner as the movement of the HMD 110 in the real space. As a result, HMD1 in the real space
The changes in position and orientation of 10 are similarly reproduced in virtual space 2.

As in the case of the HMD 110, the virtual camera 1 is defined with the uvw field-of-view coordinate system. The uvw field-of-view coordinate system of the virtual camera in the virtual space 2 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, H
When the tilt of the MD 110 changes, the tilt of the virtual camera 1 also changes accordingly. again,
The virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space.

Since the orientation of the virtual camera 1 is determined according to the position and tilt of the virtual camera 1, the user 1
When the 90 visually recognizes the virtual space image 22, the reference line of sight (reference line of sight 5) is the virtual camera 1.
It depends on the direction of. The processor 10 of the computer 200 defines the view area 23 in the virtual space 2 based on the reference line of sight 5. The view area 23 is H in the virtual space 2.
It corresponds to the field of view of the user 190 wearing the MD 110.

The line-of-sight direction of the user 190 detected by the gaze sensor 140 is the direction in the viewpoint coordinate system when the user 190 visually recognizes the object. The uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 112. Also, u of the virtual camera 1
The vw visual field coordinate system is linked to the uvw visual field coordinate system of the HMD 110. Therefore, the HMD system 100 according to a certain aspect is the user 190 detected by the gaze sensor 140.
The line-of-sight direction can be regarded as the line-of-sight direction of the user 190 in the uvw field-of-view coordinate system of the virtual camera 1.

[User's line of sight]
The determination of the line-of-sight direction of the user 190 will be described with reference to FIG. FIG. 5 is a top view of the head of a user 190 wearing an HMD 110 according to an embodiment.

In one aspect, the gaze sensor 140 detects each line of sight of the user 190's right and left eyes. In one aspect, if the user 190 is looking closer, the gaze sensor 140 will
The lines of sight R1 and L1 are detected. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 identifies the gaze point N1 which is the intersection of the lines of sight R1 and L1 based on the detected values. On the other hand, when the computer 200 receives the detected values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 specifies the line-of-sight direction N0 of the user 190 based on the position of the specified gazing point N1. For example, the computer 200 detects the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the extending direction of the straight line passing through the gazing point N1 as the line-of-sight direction N0. The line-of-sight direction N0 is the direction in which the user 190 actually directs the line of sight with both eyes. Further, the line-of-sight direction N0 corresponds to the direction in which the user 190 actually directs the line of sight to the field-of-sight area 23.

[Visibility area]
The view area 23 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing a YZ cross section of the view area 23 as viewed from the X direction in the virtual space 2. FIG. 7 is a diagram showing an XZ cross section of the view area 23 as viewed from the Y direction in the virtual space 2.

As shown in FIG. 6, the field of view region 23 in the YZ cross section includes the region 24. Area 24
Is defined by the reference line of sight 5 of the virtual camera 1 and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α around the reference line of sight 5 in the virtual space 2 as a region 24.

As shown in FIG. 7, the field of view 23 in the XZ cross section includes the region 25. Area 25
Is defined by the reference line of sight 5 and the XZ cross section of the virtual space 2. The processor 10 defines a range including the azimuth angle β centered on the reference line of sight 5 in the virtual space 2 as a region 25.

In one aspect, the HMD system 100 provides the user 190 with the virtual space 2 by displaying a field of view image on the monitor 112 based on a signal from the computer 200. The visual field image corresponds to a portion of the virtual space image 22 that is superimposed on the visual field region 23.
When the user 190 moves the HMD 110 attached to the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the view area 23 in the virtual space 2 changes. This will result in
The field-of-view image displayed on the monitor 112 is updated to an image superimposed on the field-of-view area 23 in the direction in which the user 190 faces in the virtual space 2 among the virtual space images 22. User 190
The desired direction in the virtual space 2 can be visually recognized.

While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the HMD system 10
0 can give the user 190 a high sense of immersion in the virtual space 2.

In a certain aspect, the processor 10 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. in this case,
The processor 10 is an HM based on the position and orientation of the virtual camera 1 in the virtual space 2.
The image area projected on the monitor 112 of the D110 (that is, the view area 23 in the virtual space 2) is specified.

According to certain embodiments, the virtual camera 1 preferably includes two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Further, it is preferable that an appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In the present embodiment, the virtual camera 1 includes two virtual cameras, and the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD 110. Assuming that it is configured as such, the technical idea according to the present disclosure will be exemplified.

[controller]
An example of the controller 160 will be described with reference to FIG. FIG. 8 is a diagram showing a schematic configuration of a controller 160 according to an embodiment.

As shown in FIG. 8A, the controller 160 may include a right controller 800 and a left controller (not shown) in certain aspects. Right controller 800
Is operated by the right hand of the user 190. The left controller is operated by the left hand of the user 190. In one aspect, the right controller 800 and the left controller are symmetrically configured as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller 800 and the left hand holding the left controller. In another aspect, the controller 160 may be an integrated controller that accepts the operation of both hands. Hereinafter, the right controller 800 will be described.

The right controller 800 includes a grip 30, a frame 31, and a top surface 32. The grip 30 is configured to be gripped by the right hand of the user 190. For example, the grip 30 may be held by the palm of the user 190's right hand and three fingers (middle finger, ring finger, little finger).

The grip 30 includes buttons 33 and 34 and a motion sensor 130. Button 33
Is arranged on the side surface of the grip 30 and accepts an operation by the middle finger of the right hand. The button 34 is arranged on the front surface of the grip 30 and accepts an operation by the index finger of the right hand. In one aspect, the buttons 33, 34 are configured as trigger type buttons. The motion sensor 130 is built in the housing of the grip 30. If the operation of the user 190 can be detected from around the user 190 by a camera or other device, the grip 30 may be used.
The motion sensor 130 may not be provided.

The frame 31 includes a plurality of infrared LEDs 35 arranged along its circumferential direction. The infrared LED 35 emits infrared rays as the program progresses while the program using the controller 160 is being executed. The infrared rays emitted from the infrared LED 35 can be used to detect each position and posture (tilt, orientation) of the right controller 800 and the left controller. In the example shown in FIG. 8, infrared LEDs 35 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. An array with one column or three or more columns may be used.

The top surface 32 includes buttons 36 and 37 and an analog stick 38. Button 36,
37 is configured as a push-type button. Buttons 36 and 37 accept operations by the thumb of the user 190's right hand. The analog stick 38 accepts an operation in any direction 360 degrees from the initial position (neutral position) in a certain aspect. The operation is
For example, it includes an operation for moving an object arranged in the virtual space 2.

In one aspect, the right controller 800 and the left controller are infrared LEDs 35.
Includes batteries for driving other components. Batteries include, but are not limited to, rechargeable, button type, dry cell type and the like. In other aspects, the right controller 800 and the left controller may be connected to, for example, the USB interface of the computer 200. In this case, the right controller 800 and the left controller do not require batteries.

FIG. 8B shows an example of a hand object 810 arranged in the virtual space 2 corresponding to the right hand of the user 190 holding the right controller 800. For example, user 19
The yaw, roll, and pitch directions are defined for the hand object 810 corresponding to the right hand of 0. For example, if the input operation is performed on the button 34 of the right controller 800, the index finger of the hand object 810 is grasped, and if the input operation is not performed on the button 34, the segmentation diagram ( As shown in B), the index finger of the hand object 810 may be extended. For example, in the hand object 810, when the thumb and index finger are extended, the direction in which the thumb is extended is the yaw direction, and the direction in which the index finger is extended is the roll direction, the yaw direction axis, and the direction perpendicular to the plane defined by the roll direction axis. The direction is defined by the hand object 810 as the pitch direction.

[HMD control device]
The control device of the HMD 110 will be described with reference to FIG. In certain embodiments, the control device is implemented by a computer 200 having a well-known configuration. FIG. 9 is a block diagram showing a computer 200 according to an embodiment as a module configuration.

As shown in FIG. 9, the computer 200 includes a display control module 220, a virtual space control module 230, a voice control module 225, and a memory module 240.
Communication control module 250, playback control module 260, and shooting control module 270
And prepare.

The display control module 220 is a virtual camera control module 22 as a sub module.
1, a view area determination module 222, a view image generation module 223, and a reference line-of-sight identification module 224 are included.

The virtual space control module 230 is a virtual space definition module 2 as a sub-module.
31, virtual object generation module 232, and hand object control module 23
3 and is included.

In one embodiment, the display control module 220, the virtual space control module 230
, And the voice control module 225 is realized by the processor 10. In other embodiments, the plurality of processors 10 may operate as the display control module 220, the virtual space control module 230, and the voice control module 225. The memory module 240 is realized by the memory 11 or the storage 12. Communication control module 2
50 is realized by the communication interface 14.

In one aspect, the display control module 220 controls the image display on the monitor 112 of the HMD 110. The virtual camera control module 221 has a virtual camera 1 in the virtual space 2.
Is arranged to control the behavior, orientation, etc. of the virtual camera 1. The view area determination module 222 defines the view area 23 according to the orientation of the head of the user 190 wearing the HMD 110.
The field image generation module 223 generates data (also referred to as field image data) of the field image displayed on the monitor 112 based on the determined field area 23. Further, the view image generation module 223 generates the view image data based on the data received from the virtual space control module 230. The view image data generated by the view image generation module 223 is output to the HMD 110 by the communication control module 250. The reference line-of-sight identification module 224 identifies the line-of-sight of the user 190 based on the signal from the gaze sensor 140.

The virtual space control module 230 controls the virtual space 2 provided to the user 190.
The virtual space definition module 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2.

The virtual object generation module 232 generates data of an object arranged in the virtual space 2. Objects can include, for example, other avatar objects, virtual panels, virtual letters, and virtual posts. The data generated by the virtual object generation module 232 is output to the field of view image generation module 223.

The hand object control module 233 arranges the hand object in the virtual space 2. The hand object corresponds, for example, to the right or left hand of the user 190 holding the controller 160. In a certain aspect, the hand object control module 233 generates data for arranging the hand object corresponding to the right hand or the left hand in the virtual space 2. Further, the hand object control module 233 generates data for moving the hand object in response to the operation of the controller 160 by the user 190. The data generated by the hand object control module 233 is output to the field of view image generation module 223.

In other aspects, if the movement of a part of the body of the user 190 (eg, the movement of the left hand, right hand, left foot, right foot, head, etc.) is associated with the controller 160, the virtual space control module 230 will be the user 190. Generate data for arranging a partial object corresponding to a part of the body in the virtual space 2. When the user 190 operates the controller 160 using a part of the body, the virtual space control module 230 generates data for moving the partial object. These data are output to the field of view image generation module 223.

When the voice control module 225 detects an utterance using the microphone 119 of the user 190 from the HMD 110, the voice control module 225 identifies the computer 200 to which the voice data corresponding to the utterance is transmitted. The voice data is the computer 2 specified by the voice control module 225.
It is sent to 00. When the voice control module 225 receives voice data from another user's computer 200 via the network 19, the voice control module 225 outputs voice (utterance) corresponding to the voice data from the speaker 115.

The memory module 240 holds data used by the computer 200 to provide the virtual space 2 to the user 190. Memory module 240 in one aspect
Holds spatial information 241 and object information 242, user information 243, content data 244, and history data 245.

Spatial information 241 holds one or more templates defined to provide virtual space 2.

The object information 242 holds the content to be reproduced in the virtual space 2 and the information for arranging the object used in the content. The content is
For example, it may include games, content representing landscapes similar to the real world, and the like. Moreover,
The object information 242 includes data for arranging a hand object corresponding to the hand of the user 190 who operates the controller 160 in the virtual space 2, data for arranging the avatar object of each user in the virtual space 2, and a virtual panel. Includes data for arranging other objects such as in virtual space 2.

The user information 243 holds a program for operating the computer 200 as a control device of the HMD system 100, an application program using each content held in the object information 242, and the like. The data and programs stored in the memory module 240 are input by the user 190 of the HMD 110. Alternatively, the processor 10 is a computer operated by a business operator that provides the content (
For example, a program or data is downloaded from the server 150), and the downloaded program or data is stored in the memory module 240.

The content data 244 is data for expanding the content in the virtual space 2.
This data is created by the content provider. In another aspect, the content data 244 may be provided by a general user pre-registered in the community providing the content. In this case, the general user may receive points or other rewards based on the viewing of the content posted by himself / herself and based on a predetermined standard.

The history data 245 includes a history of reproduction, shooting, and other operations performed in the virtual space 2.

The communication control module 250 may communicate with the server 150 and other information communication devices via the network 19.

The reproduction control module 260 controls reproduction in the virtual space 2 using the content data 244. The shooting control module 270 controls shooting by the user 190 in the virtual space 2.

In a certain aspect, the imaging control module 270 detects that imaging is performed in the virtual space 2, and stops reproduction of the image in response to the detection. In a certain aspect, the shooting control module 270 detects that shooting is performed in the virtual space 2, and in response to the detection, presents a user interface object for controlling the reproduction of the image to the virtual space 2.

In a certain aspect, the shooting control module 270 detects that shooting is performed based on the fact that the camera object is presented in the virtual space 2. In a certain aspect, the reproduction control module 260 stops the reproduction of the image reproduced in the direction taken by the camera object in the virtual space 2. In another aspect, the reproduction control module 260 may stop reproduction of an image included in the shooting range of the camera object.

In one aspect, the imaging control module 270 presents a camera object in a manner that allows the user 190 to take a selfie. For example, the imaging control module 270 presents a camera object so that the in-camera faces the user 190.

In a certain aspect, the shooting control module 270 presents the avatar object corresponding to the user 190 and the image in the background of the avatar object on the preview screen of the camera object. The reproduction control module 260 stops the reproduction of the image displayed on the preview screen.

In a certain aspect, the photographing control module 270 detects the operation of the user 190. For example, the photographing control module 270 detects that imaging is performed based on the detected motion being a predetermined motion as an motion suggesting shooting.

In a certain aspect, the shooting control module 270 presents a camera object having a preview screen to the virtual space 2, and presents an avatar object corresponding to the user 190 to the virtual space 2. The shooting control module 270 detects that shooting is performed based on the line of sight of the avatar object facing the preview screen.

In one aspect, the imaging control module 270 presents the camera object to the virtual space 2. The virtual object generation module 232 generates data for presenting a limb object corresponding to any of the user's limbs to the virtual space 2. The view image generation module 223 presents a limb object to the virtual space 2 based on the data. The limb object is, for example, a right hand object, a left hand object, a right foot object, and a left foot object. The processor 10 moves the limb object based on the movement of the user 190. The shooting control module 270 detects that shooting is performed based on the fact that the limb object is in the vicinity of the camera object.

In one aspect, the reproduction control module 260 presents the reproduced image by going back a predetermined time. In one aspect, the reproduction control module 260 records a place in the virtual space 2 where reproduction is stopped.

In certain aspects, the display control module 220 and the virtual space control module 230 may be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

The processing in the computer 200 is realized by the hardware and the software executed by the processor 10. Such software may be pre-stored in a hard disk or other memory module 240. Also, the software
It may be stored in a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other networks. Such software is read from the data recording medium by an optical disk drive or other data reader, or downloaded from the server 150 or other computer via the communication control module 250, and then temporarily stored in the storage module. .. The software is processor 1
Read from the storage module by 0 and stored in RAM in the form of an executable program. Processor 10 executes the program.

The hardware that constitutes the computer 200 is general. Therefore, it can be said that the most essential part of the present embodiment is the program stored in the computer 200. Since the operation of the hardware of the computer 200 is well known, the detailed description will not be repeated.

The data recording medium is not limited to CD-ROM, FD (Flexible Disk), and hard disk, but magnetic tape, cassette tape, and optical disk (MO (Magnetic Optical)).
Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC (Integr
ated Circuit) Card (including memory card), optical card, mask ROM, EPROM
(Electronically Programmable Read-Only Memory), EEPROM (Electronicall)
y Erasable Programmable Read-Only Memory), a non-volatile data recording medium such as a semiconductor memory such as a flash ROM that carries a fixed program may be used.

The program referred to here may include not only a program that can be directly executed by the processor 10, but also a program in the form of a source program, a compressed program, an encrypted program, and the like.

[Control structure of HMD system]
The control structure of the HMD system 100 will be described with reference to FIG. FIG. 10 is a sequence chart showing a part of the processing performed in the HMD system 100 according to an embodiment.

As shown in FIG. 10, in step S1010, the processor 10 of the computer 200 specifies the virtual space image data as the virtual space definition module 231 and defines the virtual space 2.

In step S1020, the processor 10 initializes the virtual camera 1. for example,
The processor 10 arranges the virtual camera 1 at a predetermined center point in the virtual space 2 in the work area of the memory, and directs the line of sight of the virtual camera 1 in the direction in which the user 190 is facing.

In step S1030, the processor 10 is set as the field of view image generation module 223.
Generates field of view image data for displaying the initial field of view image. The generated view image data is output to the HMD 110 by the communication control module 250.

In step S1032, the monitor 112 of the HMD 110 displays the view image based on the view image data received from the computer 200. The user 190 wearing the HMD 110 can recognize the virtual space 2 when he / she visually recognizes the visual field image.

In step S1034, the HMD sensor 120 detects the position and tilt of the HMD 110 based on the plurality of infrared rays emitted from the HMD 110. The detection result is output to the computer 200 as motion detection data.

In step S1040, the processor 10 identifies the viewing direction of the user 190 wearing the HMD 110 based on the position and inclination included in the motion detection data of the HMD 110.

In step S1050, the processor 10 executes the application program.
The object is presented in the virtual space 2 based on the instruction included in the application program. The object presented at this time includes other avatar objects.

In step S1060, the controller 160 detects the operation of the user 190 based on the signal output from the motion sensor 130, and outputs the detection data representing the detected operation to the computer 200. In another aspect, the operation of the controller 160 by the user 190 may be detected based on the image from the camera arranged around the user 190.

In step S1065, the processor 10 detects the operation of the controller 160 by the user 190 based on the detection data acquired from the controller 160.

In step S1070, the processor 10 generates visual field image data for presenting the hand object to the virtual space 2.

In step S1080, the processor 10 is the controller 160 by the user 190.
Generates field image data based on the operation of. The generated view image data is output to the HMD 110 by the communication control module 250.

In step S1092, the HMD 110 updates the view image based on the received view image data, and displays the updated view image on the monitor 112.

[Technical Thought]
Next, the technical idea relating to the present disclosure will be described. According to one embodiment, the technical idea is summarized as follows. The computer 200 can control the reproduction of the background moving image while the user 190 wearing the HMD 110 determines the shooting angle in the virtual space 2, or from the determination of the angle to the shooting. The control of the playback of the background video at the time of shooting includes, for example, stopping the playback of the background video, rewinding the background video, and slightly advancing the playback of the background video (fast forward). Further, regarding the control of the reproduction of the background moving image, the reproduction of the entire background moving image may be stopped. Alternatively, the reproduction of a part of the background that is the shooting range may be stopped. In yet another aspect, a still image may be displayed on the preview screen of the smartphone object arranged in the virtual space for self-shooting.

As a method for controlling the reproduction of the background moving image, for example, it is conceivable that the computer 200 first buffers the image data in the buffer area for a certain period of time while reproducing the background moving image. The following can be considered as the timing for reading the buffering data.

-When the user 190 wearing the HMD 110 calls the self-shooting camera object from a menu or the like when holding the self-shooting camera and causes the avatar to hold the camera object, the computer 200 displays the buffered data. Reproduce.

-When holding a selfie camera and trying to take a picture At this time, the line of sight of the user 190 is directed to the preview part of the selfie camera in the virtual space 2. In the virtual space 2, a hand object corresponding to the hand of the user 190 is presented near the user interface for shooting (for example, a shooting button).

In the present embodiment, the user 190 can more finely control the reproduction timing of the background moving image included in the self-portrait captured image, so that the reproduction timing reflects the preference of the user 190. The computer 200 saves the captured image in association with the reproduction timing.

Further, the angle information at the time of shooting is associated with the ID of the background moving image and is associated with the memory module 2.
Accumulate in 40. The angle information indicates from which position in the virtual space 2 to which direction and when the image was taken. With the angle information, it is possible to recommend (recommend) the angle to other users for the position taken by many users. For example, whether or not the user has taken a picture based on the recommendation may be based on the user's preference. Therefore, by accumulating the places where the reproduction of the moving image content is stopped, it is possible to grasp the user's preference for the moving image content. For example, when the background moving image stopped for shooting represents a landscape, it can be inferred that the user's preference is naturally directed. Alternatively, if the background video is in front of a restaurant, it can be inferred that the user's preference is directed to food. Therefore, it is possible to realize advertisement distribution based on the preference.

The details of the photographing control module 270 will be described with reference to FIG. 11. FIG. 11 is a block diagram showing the configuration of the photographing control module 270. The shooting control module 270 includes a camera object presentation module 1110, a shooting detection module 1120, an interest object extraction module 1130, a viewing history recording module 1140, and a shooting information generation module 1150.

The camera object presentation module 1110 presents the camera object to the virtual space 2. For example, the camera object presentation module 1110 presents a camera object based on an operation by the user 190 (for example, an operation of the controller 160, a gaze of an icon presented at the end of the virtual space 2, etc.). In another aspect, the camera object presentation module 1110 may present a camera object based on the arrival of a point pre-designated by the provider of the content as a shooting point in the content being played. This point can be defined as, for example, the elapsed time from the beginning of the content.

The shooting detection module 1120 detects that shooting is performed by the user 190 in the virtual space 2. Shooting detection, for example, when the user 190 presses the shutter button of the camera object, or when a predetermined action is executed to instruct shooting, such as when a predetermined voice is emitted. Module 1120 detects that shooting is taking place.

In another aspect, the shooting detection module 1120 is directed to the virtual camera 1 (reference line of sight).
Detects that shooting is performed based on the fact that is colliding with the object for preview display and that the eye tracking result detects that the user 190's line of sight is directed at the object for preview display. do. The line of sight of the user 190, that is, the object to which the user 190 is paying attention, can be determined by the direction of the virtual camera 1 and the result of eye tracking. According to this configuration, when a user 190 sees an object for preview display in the virtual space 2 and an operation similar to the operation in the real space is detected, it is detected that shooting is performed, so that the virtual reality does not feel strange. You can immerse yourself in.

In another aspect, the imaging detection module 1120 detects that imaging is performed based on the movement of any of the limbs (limbs) of the user 190. For example, the virtual object generation module 232 generates data for displaying a camera object and data for presenting a limb object corresponding to a limb (hand) in a virtual space, and the view image generation module 223 generates data for them. Based on the data of, the camera object and the limb object are presented in the virtual space 2. The hand object control module 233 moves the limb object according to the detection result of the movement of the limb (hand). For example, when a limb object is present in the vicinity of the camera object, the shooting detection module 1120 detects that “shooting is performed”.

The interest object extraction module 1130 extracts objects that the user 190 is interested in. For example, the interest object extraction module 1130 is interested in an object contained in an area where the gaze time is longer than a predetermined time based on the time when the user 190 is gaze at the virtual space 2. Extract as an object.

The viewing history recording module 1140 records the history of the content viewed by the user 190. For example, the viewing history recording module 1140 has a content identification number and
The memory module 2 describes the identification number of the user 190, the point (time) at which the viewing of the content is started and ended, and the operation (shooting, recommendation operation, etc.) performed during the viewing of the content.
Record at 40. In another aspect, the viewing history recording module 1140 transmits the viewing history to the server 150.

The shooting information generation module 1150 generates shooting information and records the shooting information in the memory module 240. The shooting information includes the identification information of the photographer (user 190 of the HMD 110), the point (time) of the content in which the shooting was performed, and the shooting date and time.

[Control structure]
The control structure of the computer 200 will be described with reference to FIG. FIG. 12 is a flowchart showing a part of the processing executed by the processor 10 of the computer 200 according to an embodiment.

In step S1210, the processor 10 defines the virtual space 2 provided by the headmount device. User 190 can experience virtual reality through virtual space 2.

In step S1220, the processor 10 displays a list of playable moving image contents in the virtual space. For example, the processor 10 presents a list of video content based on the actions taken by the user 190. The operation is the virtual space 2 using the controller 160.
Including the operation of the user 190, the utterance of the user 190, and the like.

In step S1230, the processor 10 reads out the moving image content selected from the list. The moving image content is, for example, 360-degree moving image content, but is not limited to this. Video content may be provided by a content provider or by a user registered in the community as a collection of members who provide the content. In this case, the member who provided the content viewed by many users may receive points or other rewards based on predetermined criteria from the community.

In step S1240, the processor 10 reproduces the moving image content in the virtual space 2. The user 190 watches the moving image content via the monitor 112 of the HMD 110. The user 190 may find a scene desired to be shot while viewing the video content. Alternatively, if it has already been viewed by another user, a point (scene) recommended as a shooting point by the other user may arrive. Furthermore, the video content may be marked with a shooting point in advance as a recommended scene by the creator of the video content. In such a case, the user 190 calls the camera object for shooting in the virtual space 2.

In step S1250, the processor 10 detects that shooting is performed in the virtual space 2. For example, the processor 10 detects that shooting is performed when the camera object is presented in the virtual space 2. In another aspect, the processor 10 detects that the photographing is performed based on the detected operation being a predetermined operation as an operation suggesting the photographing.

In step S1260, the processor 10 stops playing the moving image content in response to the detection of the shooting. For example, the processor 10 stops the reproduction of the entire moving image content expanded in the virtual space 2. In another aspect, the processor 10 may stop the reproduction of the moving image content developed in the virtual space 2 only in the area including the shooting range and a certain margin. In this case, the processor 10 has a memory area in the memory 11 for drawing the moving image content in an area not including the area, and a memory area for drawing the moving image content to be shot in the area as a still image. Separately, the drawing process is executed.

In step S1270, the processor 10 executes shooting in the virtual space. More specifically, the processor 10 sets the shooting mode by the camera object to the self-shooting mode based on the command of the self-shooting mode by the user 190. When the user 190 adjusts the angle in the virtual space 2, the angle is temporarily fixed, and the avatar object of the user 190 is arranged between the camera object and the background to be photographed. The processor 10 executes the image pickup process in the state of the arrangement, and saves the image of the area to be photographed in the memory 11.

In step S1280, the processor 10 records shooting information (photographer (= user 190 of HMD110), content type stamp (point), shooting date and time, etc.).

In step S1290, the processor 10 generates viewing history data and transmits the viewing history data to the server. After that, the processor 10 ends the process.

In the above process, regarding the reproduction of 360-degree video content, for example, the processor 10
Pastes a panoramic video inside a celestial sphere that can be defined by virtual space 2. The processor 10 stores the data of the panoramic moving image in the memory 11 and other buffer areas, and sequentially reproduces the panoramic moving image based on the data. Even after the panoramic moving image is played back, the data is held in the buffer area for a predetermined period of time. When the user 190 performs a predetermined gesture, or when a camera object or another predetermined object as a trigger is arranged in the virtual space 2, the processor 10 buffers the data used for the past reproduction in the buffer area. By reading from, the moving image can be presented to the user 190 as if the background had stopped.

[data structure]
The data structure of the memory module 240 will be described with reference to FIG. FIG. 13 is a diagram showing an aspect of data storage in the memory module 240 according to an embodiment. The memory module 240 holds content data 244 and history data 245 in addition to spatial information 241 and object information 242 and user information 243.

The content data 244 includes the content ID 1310, the content data 1311, the download date / time 1312, and the management data 1313. Content ID 1310
Identify the content. The content data 1311 specifies the actual data of the content. The download date and time 1312 represents the date and time when the content was downloaded from the server 150. The management data 1313 represents other data associated with the content. The management data 1313 includes, for example, the number of times of reproduction and the reproducible time given in advance by the provider of the content.

The history data 245 holds a record of the history in which the content is played. More specifically, the history data 245 includes a content ID 1310, a reproduction date and time 1321, an operation record 1322, a reproduction start point 1323, a reproduction end point 1324, and a shooting point 1325.

The reproduction date and time 1321 represents the date and time when the content was reproduced.
The operation record 1322 identifies the operation performed by the user 190 when the content is reproduced. The operation may include playback, content shooting, fast-forwarding, reverse-forwarding, and the like.

The reproduction start point 1323 represents a place where the reproduction of the content by the user 190 is started. In one aspect, the reproduction start point 1323 is indicated, for example, using time data assigned to the content. In the time data, for example, the beginning of the content is set to 0. When the content includes a plurality of chapters, the beginning of the chapter may be set to 0 in addition to the chapter number.

The reproduction end point 1324 represents a place where the reproduction of the content by the user 190 is completed. The reproduction end point 1324 is also indicated by using the time data in the same manner as the reproduction start point 1323.

The shooting point 1325 specifies the shooting location when shooting is performed in the virtual space during the reproduction of the content. In certain aspects, this location is also specified using time data, similar to the reproduction start point 1323 and the reproduction end point 1324. For example, the shooting point 1326 includes time data and a direction (= shooting direction) in which the virtual camera 1 is facing. The shooting direction may include a subject of interest to the user 190. therefore,
By collecting data in the direction in which the virtual camera 1 is facing, it becomes easy to identify interesting contents of the user 190.

When the computer 200 is shared by a plurality of different users in another aspect, each history record constituting the history data 245 may further include each user ID.

With reference to FIG. 14, the shooting data 140 sent from the computer 200 to the server 150.
0 will be described. FIG. 14 is a diagram showing the configuration of shooting data 1400 according to a certain embodiment. The shooting data 1400 includes a header 1410, a user ID 1420, a content ID 1430, a shooting point 1440, and a shooting date and time 1450.

The header 1410 includes a source, a destination, a transmission date and time, and the like of the shooting data 1400. User I
The D1420 identifies the sender of the shooting data 1400, that is, the user 190 who generated the shooting data 1400 by shooting in the virtual space 2. Content ID1
430 identifies the content played by the user identified by the user ID 1420 (eg, user 190). The shooting point 1440 is the content ID 143.
For the content specified by 0, the place where the shooting was performed in the virtual space 2 is specified. This location is specified by the position information in the virtual space 2 (see FIG. 4). The shooting date and time 1450 represents the date and time when the shooting was performed at the place specified by the shooting point 1440. This date and time may be the date and time in the real space or the date and time in the virtual space 2.

With reference to FIG. 15, the transition of the arrangement of the image displayed on the monitor 112 and each object in the virtual space 2 will be described. FIG. 15 is a diagram showing the correspondence between the image displayed on the monitor 112 and the arrangement of each object in the virtual space 2 when the display is performed.

In one aspect, the computer 200 plays the content selected by the user 190. When the playback of the content starts, the content is displayed on the monitor 112. As shown in the state (A), the monitor 112 displays the object 1501 and the object 1502. At this time, the arrangement of each object in the virtual space 2 is shown as the state (B).

As the playback of the content progresses, the object 1502 moves, and flies in front of the object 1501, the monitor 112 displays an image as shown in the state (C). At this time, the object 1501 and the object 1502 are in the state (D) in the virtual space 2.
It is in the positional relationship shown in.

When the user 190 performs a predetermined operation to shoot the content, a camera object 1510 representing a camera for shooting in the virtual space 2 is presented in the virtual space 2. For example, as shown in state (E), the camera object 1510 is presented in the center of the monitor 112. At this time, as shown in the state (F), the camera object 1510 is arranged between the virtual camera 1 and the object 1502.

When the user 190 selects the selfie mode in the virtual space 2, the avatar object 1520 corresponding to the user 190 is arranged as shown in the state (G). The positional relationship in the virtual space 2 at this time is the camera object 15 as shown in the state (H).
An avatar object 1520 is arranged between the 10 and the object 1502. The distance between the avatar object 1520 and the camera object 1510 is a predetermined distance as a shooting condition in the self-shooting mode. In another aspect, the preview screen is displayed, and the user 190 may be able to adjust his / her own distance while checking the preview screen. After that, when the user 190 performs a predetermined operation for shooting, or when a predetermined time elapses for automatic shooting, shooting in the virtual space 2 is executed.

The mode of presenting the 360-degree moving image content is not limited to the mode shown in FIG.
For example, a celestial sphere object is defined for the virtual space 2, and a panoramic moving image is pasted inside the celestial sphere. The virtual camera 1 arranged inside the celestial sphere photographs the inside of the celestial sphere according to its orientation and outputs it to the HMD 110 as 360 moving image contents. The direction of the line of sight of the virtual camera 1 is controlled according to the output from the gyro sensor attached by the HMD 110.

A control structure of the computer 200 according to another aspect will be described with reference to FIG.
FIG. 16 is a flowchart showing a part of the processing executed by the processor 10 of the computer 200. The same step numbers are assigned to the same processes as described above. Therefore, the description of the same process is not repeated. When the process shown in FIG. 16 is executed, a user interface object for controlling the reproduction of the content in the virtual space 2 is presented.

In step S1610, the processor 10 presents a user interface object for controlling the reproduction of the moving image content. For example, the processor 10 has arrived at a scene pre-designated by the content provider based on the behavior of the user 190, at predetermined time intervals, or as a scene for presenting a user interface object. Based on, the user interface object is presented in the virtual space 2.

In step S1620, the processor 10 detects the input of an instruction to the user interface object. For example, a user 190 wearing the controller 160 in his hand.
When a player performs a valid operation in the virtual space 2, the operation is detected as an instruction to the user interface object.

In step S1630, the processor 10 controls the reproduction of the moving image content in response to the input instruction. For example, the processor 10 pauses and ends playback of the content and fast forwards the content forward or backward.

In step S1640, the processor 10 detects the end of shooting. For example, when the user 190 finishes the shooting operation, the processor 10 determines that the shooting is finished.

In step S1650, processor 10 hides the user interface object. After that, the process ends.

With reference to FIG. 17, the transition of the arrangement of the image displayed on the monitor 112 and each object in the virtual space 2 will be described. FIG. 17 is a diagram showing the correspondence between the image displayed on the monitor 112 and the arrangement of each object in the virtual space 2 when the display is performed.

In one aspect, the computer 200 plays the content selected by the user 190. When the playback of the content starts, the content is displayed on the monitor 112. As shown in the state (A), the monitor 112 displays the object 1501 and the object 1502. At this time, the arrangement of each object in the virtual space 2 is shown as the state (B).

As the content playback progresses, the object 1502 moves and flies in front of the object 1501. When the user 190 makes a gesture to form a frame with the thumb and index finger of both hands, the signal corresponding to the gesture is sent to the controller 160 attached to both hands.
The computer 200 identifies the gesture based on the signal and determines what kind of gesture was performed. This gesture is presented as a left-handed object 1710 and a right-handed object 1711 in virtual space 2. For example, the monitor 112 displays an image as shown in the state (C). At this time, the object 1501, the object 1502, the left-handed object 1710, and the right-handed object 1711 are in the positional relationship shown in the state (D) in the virtual space 2.

Then, as shown in the state (E), the computer 200 presents a user interface object 1720 for controlling the reproduction of the content in the virtual space 2. The user interface object 1720 includes a switch similar to a switch that accepts a command for controlling the reproduction of a moving image in the real space. At this time, as shown in the state (F), the user interface object 1720 is arranged between the objects 1501 and the object 1502 and the virtual camera 1.

In this way, when the user 190 tries to shoot the moving image content developed in the virtual space 2, the reproduction of the moving image content is temporarily stopped, so that a clear image can be taken.

The technical features disclosed above are summarized, for example, as follows.
(Structure 1) According to a certain embodiment, a program executed by the computer 200 for providing virtual reality is provided. This program detects to the computer 200 that a step of defining a virtual space 2 provided by the HMD 110 connected to the computer 200, a step of reproducing an image in the virtual space 2, and shooting in the virtual space 2 are performed. The step and the step of stopping the reproduction of the image in response to the detection are executed.
(Structure 2) According to an embodiment, the program causes the computer 200 to further perform a step of presenting the camera object 1510 to the virtual space 2. The detection step includes a step of detecting that shooting is performed based on the fact that the camera object 1510 is presented in the virtual space 2.
(Structure 3) According to a certain embodiment, the step of stopping the reproduction of the image includes the step of stopping the reproduction of the image being reproduced in the direction taken by the camera object 1510 in the virtual space 2.
(Structure 4) According to a certain embodiment, stopping the reproduction of an image includes a step of stopping the reproduction of an image included in the shooting range of the camera object 1510.
(Structure 5) According to an embodiment, the step of presenting the camera object 1510 is such that the user 190 of the HMD 110 can take a selfie.
Includes steps to present.
(Structure 6) According to an embodiment, the program is on the computer 200 and the user 190.
The avatar object corresponding to and the image in the background of the avatar object
Further execution of the step presented on the preview screen of the camera object 1510.
The step of stopping the reproduction of the image includes the step of stopping the reproduction of the image displayed on the preview screen.
(Structure 7) According to an embodiment, the program is on the computer 200, HMD110.
Further execute the step of detecting the operation of the user 190. The detection step includes a step of detecting that shooting is performed based on the detected motion being a predetermined motion as an motion suggesting shooting.
(Structure 8) According to an embodiment, the program presents the computer 200 with a step of presenting the camera object 1510 to the virtual space 2, an avatar object corresponding to the user 190, and an image in the background of the avatar object. Camera object 1
Further execution of the step presented on the preview screen of 510. The step of detecting that shooting is performed includes a step of detecting that shooting is performed when the line of sight of the avatar object is facing the preview screen.
(Structure 9) According to a certain embodiment, the step of detecting the movement includes the step of detecting the movement of any of the limbs of the user 190. The step of detecting that an imaging is performed includes a step of detecting that an imaging is performed based on the movement of any of the limbs.
(Structure 10) According to an embodiment, the program causes the computer 200 to further perform a step of presenting the reproduced image by going back a predetermined time.
(Structure 11) According to a certain embodiment, the program is installed in the computer 200 in the virtual space 2.
Further execute the step of recording the place where the reproduction is stopped in.
(Structure 12) According to an embodiment, the program causes the computer 200 to further perform a step of presenting a user interface object 1720 for controlling the reproduction of the image in response to the detection that an image is being taken.
(Structure 13) According to a certain embodiment, an information processing apparatus including a memory for storing the program described in any of the above and a memory for executing the program is provided.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 virtual camera, 2 virtual space, 5 reference line of sight, 10,151 processor, 11,1
52 memory, 12,154 storage, 13 input / output interface, 14 communication interface, 15 bus, 19 network, 21 center, 22 virtual space image, 23 view area, 24,25 area, 30 grip, 31 frame, 32 top surface,
33, 34, 36, 37 buttons, 38 analog sticks, 100 systems, 11
2 monitors, 114,120 sensors, 115 speakers, 119 microphones, 130 motion sensors, 140 gaze sensors, 150 servers, 153 communication interfaces, 160 controllers, 190 users, 200 computers, 220 display control modules, 221 virtual camera control modules, 222 Visibility area determination module, 22
3 View image generation module, 224 reference line-of-sight identification module, 225 voice control module, 230 virtual space control module, 231 virtual space definition module, 232
Virtual object generation module, 233 hand object control module, 240 memory module.

Claims (12)

A program that is run on a computer to provide virtual reality to the user, said program to the computer.
Steps to define virtual space and
The step of reproducing an image in the virtual space and
The step of detecting that shooting is performed in the virtual space,
In response to the detection, the step of stopping the reproduction of the image and
The step of presenting the camera object in the virtual space,
The step of presenting the avatar object corresponding to the user and the image in the background of the avatar object to the object displaying the preview screen included in the camera object is executed .
The step of detecting that the shooting is performed is a program including a step of detecting that the shooting is performed when the line of sight of the avatar object is directed to the preview screen. The program according to claim 1, wherein the detection step includes a step of detecting that the shooting is performed based on the camera object being presented in the virtual space. The program according to claim 2, wherein the step of stopping the reproduction of the image includes stopping the reproduction of the image reproduced in the direction taken by the camera object in the virtual space. The program according to claim 3, wherein stopping the reproduction of the image includes stopping the reproduction of the image included in the shooting range of the camera object. The program according to any one of claims 2 to 4, wherein the step of presenting the camera object includes presenting the camera object in a manner capable of taking a selfie by the user. The program according to claim 5, wherein the step of stopping the reproduction of the image includes stopping the reproduction of the image displayed on the preview screen. The program causes the computer to perform further steps to detect the user's actions.
The step according to claim 1, wherein the detection step includes a step of detecting that the imaging is performed based on the detected motion being a predetermined motion as an motion suggesting the imaging. program. The step of detecting the movement includes the step of detecting the movement of any of the limbs of the user.
The program according to claim 7, wherein the step of detecting that the imaging is performed includes a step of detecting that the imaging is performed based on the movement of any one of the limbs. The program according to any one of claims 1 to 8, wherein the program further causes the computer to perform a step of presenting a reproduced image by going back a predetermined time. The program according to any one of claims 1 to 9, wherein the program further causes the computer to perform a step of recording the place where the reproduction is stopped in the virtual space. 13. Program. A memory in which the program according to any one of claims 1 to 11 is stored, and
An information processing device including a memory for executing the program.

Images