JP7030075B2 - Programs, information processing devices, and information processing methods - Google Patents

Description

The present disclosure relates to programs, information processing devices, and information processing methods.

Patent Document 1 discloses that when the panoramic moving image to be displayed is switched, the line-of-sight direction with respect to the panoramic moving image before switching is maintained for the panoramic moving image after switching.

Japanese Patent No. 6147966

In the method of Patent Document 1, the line-of-sight direction before switching the moving image scene is maintained even after switching the moving image scene. Therefore, the line-of-sight direction in the panoramic moving image after switching is not always oriented in a preferable direction, and there is room for improvement in improving the virtual experience of the user.

It is an object of the present disclosure to provide a program, an information processing device, and an information processing method capable of improving a user's virtual experience.

According to one aspect of the present disclosure.
A program for providing a virtual experience to the user via an image display device associated with the user's head
A step of acquiring video content composed of a plurality of scenes including at least the first scene and the second scene, and
The steps to define a virtual space to provide the virtual experience,
Based on the moving image content, the step of adapting the first scene to the virtual space and playing it back,
A step of controlling the field of view from a virtual viewpoint in the virtual space according to the posture of the head,
The step of specifying the reference direction of the second scene and
A step of switching the playback scene of the moving image content from the first scene to the second scene,
In response to the switching to the second scene, the computer is made to execute the step of associating the reference direction with the direction of the virtual viewpoint and starting the reproduction of the second scene.

According to the present disclosure, it is possible to improve the virtual experience of the user.

It is a schematic diagram which shows the virtual space distribution system which concerns on embodiment of this invention (hereinafter, simply referred to as this embodiment). It is a schematic diagram which shows the user terminal. It is a figure which shows the hardware configuration of a control device. It is a figure which shows the head of the user who wears an HMD. It is an xyz space diagram which shows an example of a virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which illustrates the method of determining the line-of-sight direction of a user. It is a block diagram which shows the functional structure of a control circuit part. It is a figure which shows an example of the structure of the moving image content displayed on an HMD. It is a figure which shows the table which defines the initial direction set in each chapter of a moving image content. It is a figure which shows the hardware configuration of the server shown in FIG. It is a sequence diagram which shows the flow of the process which the virtual space distribution system provides a virtual space to a user. It is a sequence diagram which shows the flow of the process which the virtual space distribution system plays back the moving image content selected by the user through the platform in virtual space in virtual space. It is a figure explaining the process of reproduction of the moving image content in a virtual space. It is a figure explaining the process of reproduction of the moving image content in a virtual space. It is a figure explaining the process of reproduction of the moving image content in a virtual space. It is a figure explaining the process of reproduction of the moving image content in a virtual space.

[Explanation of Embodiments Shown in the Present Disclosure]
An outline of the embodiments shown in the present disclosure will be described.
(1) A program for providing a virtual experience to the user via an image display device associated with the user's head.
A step of acquiring video content composed of a plurality of scenes including at least the first scene and the second scene, and
The steps to define a virtual space to provide the virtual experience,
Based on the moving image content, the step of adapting the first scene to the virtual space and playing it back,
A step of controlling the field of view from a virtual viewpoint in the virtual space according to the posture of the head,
The step of specifying the reference direction of the second scene and
A step of switching the playback scene of the moving image content from the first scene to the second scene,
A program for causing a computer to perform a step of associating the reference direction with the direction of the virtual viewpoint and starting playback of the second scene in response to switching to the second scene.

According to the above program, the virtual experience of the user can be improved. In particular, when switching the playback scene of the video content from the first scene to the second scene, the user is newly satisfied by starting the playback of the second scene by associating the reference direction with the direction of the virtual viewpoint. It is possible to provide a high degree of virtual experience.

(2) The video content is a 360-degree video content.
A plurality of the reference directions are set in the second scene.
Further including a step of selecting a first reference direction to be adopted at the start of reproduction of the second scene from the plurality of reference directions.
In the specifying step, the first reference direction is specified as the reference direction of the second scene.
In the step of starting the reproduction, the item (1) in which the reproduction of the second scene is started by associating the first reference direction with the direction of the virtual viewpoint in response to the switching to the second scene. ).

According to the above program, by setting a specific direction in the 360-degree video as the first reference direction, the second scene can be played back with the virtual viewpoint facing in an appropriate direction according to the content of the video content. Being able to get started can further enhance the virtual experience.

(3) The program according to item (1) or (2), wherein the reference direction is the first viewing direction set by the provider of the moving image content.

According to the above program, the reproduction of the second scene can be started in a state where the virtual line of sight is directed in the direction intended by the provider of the moving image content, so that the virtual experience can be further improved.

(4) The program according to any one of items (1) to (3), wherein the reference direction is a second viewing direction set by the user.

According to the above program, by selecting a desired direction as a reference direction by the user, it is possible to start playback of the second scene with the virtual viewpoint facing in a direction suitable for the user's preference, so that the video content can be started. It is possible to improve the satisfaction of the user who watches the program.

(5) The program according to item (4), wherein the second viewing direction can be set as the reference direction for each scene.

According to the above program, the user's satisfaction can be further improved.

(6) The program according to any one of items (1) to (5), wherein the reference direction is a third viewing direction facing the object designated by the user.

According to the above program, the reproduction of the second scene can be started with the virtual viewpoint facing the direction of the object selected by the user, so that the satisfaction of the user can be improved.

(7) The reference direction is a third viewing direction facing the object specified by the user.
The program according to item (2), wherein in the selection step, the third viewing direction is selected as the first reference direction for each scene by selecting the object.

According to the above program, the user's satisfaction can be further improved.

(8) In the step of starting the reproduction, the orientation of the virtual viewpoint on the horizontal plane in the depth direction and the orientation of the second scene on the horizontal plane in the reference direction in response to the switching to the second scene. The program according to any one of items (1) to (7), which starts the reproduction of the second scene in combination with the above.

According to the above program, the orientation of the virtual viewpoint and the orientation in the reference direction can be easily matched.

(9) An information processing device for providing a virtual experience to the user via an image display device associated with the user's head.
By controlling the processor included in the information processing device
A step of acquiring video content composed of a plurality of scenes including at least the first scene and the second scene, and
The steps to define a virtual space to provide the virtual experience,
Based on the moving image content, the step of adapting the first scene to the virtual space and playing it back,
A step of controlling the field of view from a virtual viewpoint in the virtual space according to the posture of the head,
The step of specifying the reference direction of the second scene and
A step of switching the playback scene of the moving image content from the first scene to the second scene,
An information processing apparatus for executing a step of associating the reference direction with the direction of the virtual viewpoint and starting reproduction of the second scene in response to switching to the second scene.

According to the information processing device, the virtual experience of the user can be improved.

(10) An information processing method executed by a computer to provide a virtual experience to the user via an image display device associated with the user's head.
A step of acquiring video content composed of a plurality of scenes including at least the first scene and the second scene, and
The steps to define a virtual space to provide the virtual experience,
Based on the moving image content, the step of adapting the first scene to the virtual space and playing it back,
A step of controlling the field of view from a virtual viewpoint in the virtual space according to the posture of the head,
The step of specifying the reference direction of the second scene and
A step of switching the playback scene of the moving image content from the first scene to the second scene,
An information processing method including a step of associating the reference direction with the direction of the virtual viewpoint in response to switching to the second scene and starting reproduction of the second scene.

According to the above information processing method, the virtual experience of the user can be improved.

[Details of Embodiments Shown in the Present Disclosure]
Hereinafter, embodiments shown in the present disclosure will be described with reference to the drawings. For convenience of explanation, the description of the element having the same reference number as the element already described in the description of the present embodiment will not be repeated.

First, the outline of the configuration of the virtual space distribution system 100 (information processing system) will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter, simply referred to as a distribution system 100). As shown in FIG. 1, the distribution system 100 includes a user terminal 1A operated by a user A equipped with a head-mounted device (HMD) 110, a user terminal 1B operated by a user B equipped with the HMD 110, and a server 2. And prepare. The user terminals 1A and 1B are communicably connected to the server 2 via a communication network 3 such as the Internet. Hereinafter, for convenience of explanation, the user terminals 1A and 1B may be collectively referred to as the user terminal 1. Users A and B may be collectively referred to as user U. In this embodiment, it is assumed that the user terminals 1A and 1B have the same configuration. In the present embodiment, the virtual space includes a VR (Virtual Reality) space.

Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic view showing a user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mount device (HMD) 110 mounted on the head of the user U, a headphone 116, a microphone 118, a position sensor 130, an external controller 320, and a control circuit. A unit 120 is provided.

The HMD 110 includes a display unit 112, an HMD sensor 114, and a gaze sensor 140. The display unit 112 includes a non-transparent display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. As a result, the user U can immerse himself in the virtual space by seeing only the view image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the left eye of the user U, and a right-eye display unit configured to provide an image to the right eye of the user U. May be good. Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance.

The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (angular velocity sensor, gyro sensor, etc.), and detects various movements (tilts, etc.) of the HMD 110 mounted on the head of the user U. be able to.

The gaze sensor 140 has an eye tracking function for detecting the line of sight of the user U. The gaze sensor 140 may include, for example, a gaze sensor for the right eye and a gaze sensor for the left eye. The gaze sensor for the right eye acquires information on the angle of rotation of the eyeball of the right eye by irradiating the user U's right eye with, for example, infrared light and detecting the reflected light reflected from the right eye (particularly the cornea or iris). You may. On the other hand, the gaze sensor for the left eye irradiates the left eye of the user U with infrared light, for example, and detects the reflected light reflected from the left eye (particularly the cornea or the iris) to obtain information on the angle of rotation of the eyeball of the left eye. May be obtained.

The headphones 116 are attached to the left ear and the right ear of the user U, respectively. The headphone 116 is configured to receive voice data (electrical signal) from the control circuit unit 120 and output voice based on the received voice data. The microphone 118 is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Further, the microphone 118 is configured to transmit voice data to the control circuit unit 120.

The position sensor 130 is configured, for example, by a position tracking camera and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is wirelessly or wiredly connected to the control circuit unit 120 so as to detect information regarding the positions, inclinations, or emission intensities of a plurality of detection points (not shown) provided in the HMD 110. There is. Further, the position sensor 130 is configured to detect information regarding the positions, tilts and / or emission intensities of a plurality of detection points (not shown) provided on the external controller 320. The detection point is, for example, a light emitting unit that emits infrared rays or visible light. Further, the position sensor 130 may include an infrared sensor and a plurality of optical cameras.

The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, and the hand of the user U in this embodiment), so that the movement of the virtual hand displayed in the virtual space is displayed. Used to control. The external controller 320 is a right-handed external controller 320R operated by the right hand of the user U (hereinafter, simply referred to as a controller 320R) and a left-handed external controller 320L operated by the left hand of the user U (hereinafter, simply referred to as a controller 320L). ) And. The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the fingers of the right hand. The virtual right hand existing in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the fingers of the left hand. The virtual left hand existing in the virtual space moves according to the movement of the controller 320L.

The control circuit unit 120 is, for example, a computer configured to control the HMD 110. The control circuit unit 120 identifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and based on the specified position information, determines the position of the virtual camera in the virtual space and the HMD 110 in the real space. The position of the attached user U can be accurately associated. Further, the control circuit unit 120 identifies the operation of the external controller 320 based on the information acquired from the sensor built in the position sensor 130 and / or the external controller 320, and determines the operation of the specified external controller 320. Based on this, it is possible to accurately associate the operation of the virtual hand displayed in the virtual space with the operation of the external controller 320 in the real space.

Next, the hardware configuration of the control circuit unit 120 will be described with reference to FIG. FIG. 3 is a diagram showing a hardware configuration of the control circuit unit 120. As shown in FIG. 3, the control circuit unit 120 includes a memory 121, a processor 122, a storage medium 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The memory 121, the processor 122, the storage medium 123, the I / O interface 124, and the communication interface 125 are communicably connected to each other via the bus 126.

The control circuit unit 120 may be configured as a personal computer, a smartphone, a phablet, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. Further, a part of the functions of the control circuit unit 120 may be mounted on the HMD 110, and the remaining functions of the control circuit unit 120 may be mounted on another device separate from the HMD 110.

The memory 121 is, for example, a ROM (Read Only) in which various programs and the like are stored.
It is composed of a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by the Memory) and the processor 122 are stored. The processor 122 is, for example, a CPU (Central Processing).
Unit), MPU (Micro Processing Unit) and / or GPU (Graphics Processing Unit), which is a program specified from various programs embedded in the ROM, is expanded on the RAM, and various processes are performed in cooperation with the RAM. It is configured to run.

The processor 122 may control various operations of the control circuit unit 120 by expanding the control program on the RAM and executing the control program in cooperation with the RAM. The processor 122 displays the field of view image on the display unit 112 of the HMD 110 based on the field of view image data. As a result, the user U can immerse himself in the virtual space.

The storage medium (storage) 123 is, for example, a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage medium 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage medium 123 may store data related to the user U authentication program and various images and objects (for example, an avatar). Further, the storage medium 123 may be constructed with a database including a table for managing various data.

The I / O interface 124 is configured to connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, and the microphone 118 to the control circuit unit 120 so as to be communicable. It is composed of a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control circuit unit 120 may be wirelessly connected to the HMD 110, the position sensor 130, the external controller 320, the headphones 116, and the microphone 118, respectively.

The communication interface 125 is configured to connect the control circuit unit 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as a server 2 via the communication network 3 and various processing circuits for wireless connection, and is for communicating via the communication network 3. It is configured to meet the communication standards of.

Next, a method of acquiring information regarding the position and inclination of the HMD 110 will be described with reference to FIG. FIG. 4 is a diagram showing the head of the user U wearing the HMD 110. Information regarding the position and tilt of the HMD 110 linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 4, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands upright is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the v-axis and the direction orthogonal to the w-axis are defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 has an angle around each uvw axis (that is, a yaw angle indicating rotation about the v axis, a pitch angle indicating rotation about the u axis, and a w axis as the center. The tilt determined by the roll angle indicating rotation) is detected. The control circuit unit 120 determines the angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

Next, with reference to FIG. 5, an outline of the virtual space 5 provided to the user U wearing the HMD 110 will be described. FIG. 5 is an xyz space diagram showing an example of the virtual space 5. As shown in FIG. 5, the virtual space 5 has an all-sky spherical structure that covers the entire center 51 in the 360 ° direction. FIG. 5 illustrates only the celestial sphere in the upper half of the entire virtual space 5. A plurality of substantially square or substantially rectangular meshes are associated with the virtual space 5. The position of each mesh in the virtual space 5 is predetermined as the coordinates in the space coordinate system (XYZ coordinate system) defined in the virtual space 5. The control circuit unit 120 associates each partial image constituting the contents (still image, moving image, etc.) expandable in the virtual space 5 with each corresponding mesh in the virtual space 5, so that the virtual space image can be visually recognized by the user. The virtual space 5 in which the 52 is expanded is provided to the user.

The virtual space 5 defines an XYZ space coordinate system with the center 51 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the XYZ coordinate system are the X-axis, the Y-axis, and the Z-axis, respectively. That is, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the global coordinate system, and the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the global coordinate system. The Z-axis (front-back direction) is parallel to the z-axis of the global coordinate system.

At the time of starting the HMD 110 (initial state), for example, the virtual camera 300 is arranged at the center 51 of the virtual space 5. The virtual camera 300 moves in the virtual space 5 in the same manner as the movement of the HMD 110 in the real space. As a result, changes in the position and orientation of the HMD 110 in the real space are similarly reproduced in the virtual space 5.

Similar to the HMD 110, the virtual camera 300 is defined with an uvw field-of-view coordinate system. The uvw field-of-view coordinate system of the virtual camera 300 in the virtual space 5 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the tilt of the HMD 110 changes, the tilt of the virtual camera 300 also changes accordingly. The virtual camera 300 can also move in the virtual space 5 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

The orientation of the virtual camera 300 in the virtual space 5 is determined according to the position and inclination of the virtual camera 300 in the virtual space 5. The orientation of the virtual camera 300 (the orientation of the virtual viewpoint in the virtual space 5) determines the line of sight (reference line of sight L) as a reference when the user U visually recognizes the virtual space image 52 developed in the virtual space 5. The field of view region 53 in the virtual space 5 is determined based on the reference line of sight L. The field of view 53 is an area of the virtual space 5 corresponding to the field of view of the user U wearing the HMD 110.

The field of view 53 of the virtual camera 300 is a first region 53a set as an angle range of a polar angle α centered on the reference line of sight L in the xy plane shown in the state (a) of FIG. 6, and FIG. In the xz plane shown in the state (b) of the above, there is a second region 53b set as an angle range of the azimuth angle β about the reference line of sight L. The processor 122 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the information regarding the specified line of sight of the user U and the position and inclination of the HMD 110, the processor 122 identifies the line of sight of the user U. The orientation of the virtual camera 300 (reference line of sight L of the virtual camera 300) may be determined.

The control circuit unit 120 provides the user U with the virtual space 5 by displaying the view image 54, which is a portion of the virtual space image 52 superimposed on the view area 53, on the display unit 112 of the HMD 110. When the user U moves the HMD 110, the virtual camera 300 also moves in conjunction with the movement, and as a result, the position of the view area 53 in the virtual space 5 changes. As a result, the view image 54 displayed on the display unit 112 is updated to an image superimposed on the virtual space image 52 where the user U faces (that is, the view area 53) in the virtual space 5. Therefore, the user U can visually recognize a desired portion in the virtual space 5.

Next, a method of determining the line-of-sight direction of the user will be described with reference to FIG. 7.
The line-of-sight direction of the user U detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user U visually recognizes an object. As described above, the uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user U visually recognizes the display unit 112. Further, the uvw field-of-view coordinate system of the virtual camera 300 is linked to the uvw field-of-view coordinate system of the HMD 110.

FIG. 7 is a diagram illustrating a method of determining the line-of-sight direction of the user U. As shown in FIG. 7, the gaze sensor 140 detects the line of sight of the user U's right eye and left eye. When the user U is looking near, the gaze sensor 140 detects the line of sight R1 and L1 of the user U. When the user is looking far away, the gaze sensor 140 identifies the lines of sight R2 and L2 having a smaller angle with the roll direction (w) of the HMD 110 than the user's lines of sight R1 and L1. The gaze sensor 140 transmits the detected value to the control circuit unit 120.

When the line-of-sight lines R1 and L1 are received as the line-of-sight detection values, the control circuit unit 120 identifies the gazing point N1 which is the intersection of the two. On the other hand, when the lines of sight R2 and L2 are also received, the gazing point (not shown), which is the intersection of the two, is specified. The control circuit unit 120 detects the line-of-sight direction NO of the user U based on the specified gazing point N1. The control circuit unit 120 detects, for example, the extending direction of the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user U and the gazing point N1 as the line-of-sight direction NO. The line-of-sight direction NO is the direction in which the user U actually directs the line of sight with both eyes.

Next, the functional configuration of the control circuit unit 120 will be described with reference to FIG. FIG. 8 is a block diagram showing a functional configuration of the control circuit unit 120. As shown in FIG. 8, the control circuit unit 120 provides the virtual space 5 provided to the user U by using various data received from the HMD sensor 114, the position sensor 130, the gaze sensor 140, and the external controller 320. In addition to controlling, the image display on the display unit 112 of the HMD 110 is controlled. The control circuit unit 120 includes a detection unit 210, a display control unit 220, a virtual space control unit 230, a storage unit 240, and a communication unit 250. The control circuit unit 120 functions as a detection unit 210, a display control unit 220, a virtual space control unit 230, a storage unit 240, and a communication unit 250 by the cooperation of each hardware shown in FIG. The functions of the detection unit 210, the display control unit 220, and the virtual space control unit 230 can be realized mainly by the cooperation of the memory 121, the processor 122, and the I / O interface 124. The function of the storage unit 240 can be realized mainly by the cooperation of the memory 121 and the storage medium 123. The function of the communication unit 250 may be realized mainly by the cooperation of the processor 122 and the communication interface 125. The communication unit 250 transmits / receives data to / from the server 2 via the network 3.

The detection unit 210 receives the detection value from various sensors (HMD sensor 114 or the like) connected to the control circuit unit 120. In addition, if necessary, a predetermined process using the received detection value is executed. The detection unit 210 includes an HMD detection unit 211, a line-of-sight detection unit 212, and an operation reception unit 213. The HMD detection unit 211 receives the detection values from the HMD sensor 114 and the position sensor 130, respectively. The line-of-sight detection unit 212 receives the detected value from the gaze sensor 140. The operation receiving unit 213 receives the operation by receiving the command transmitted in response to the user's operation to the external controller 320.

The display control unit 220 controls the image display on the display unit 112 of the HMD 110. The display control unit 220 includes a virtual camera control unit 221, a view area determination unit 222, and a view image generation unit 223. As shown in FIG. 5, the virtual camera control unit 221 arranges the virtual camera 300 in the virtual space 5 and controls the behavior of the virtual camera 300 in the virtual space 5. The field of view area determination unit 222 determines the field of view area 53. The visual field image generation unit 223 generates a visual field image 54 to be displayed on the display unit 112 based on the determined visual field region 53.

The virtual space control unit 230 controls the virtual space 5 provided to the user U. The virtual space control unit 230 includes a virtual space regulation unit 231, a line-of-sight management unit 232, a content identification unit 233, and a content management unit 234. The virtual space defining unit 231 defines the virtual space 5 in the distribution system 100 by generating virtual space data representing the virtual space 5 provided to the user U. The line-of-sight management unit 232 manages the reference line-of-sight L of the user U in the virtual space 5. The content specifying unit 233 specifies the content to be reproduced in the virtual space 5. The content management unit 234 specifies an initial direction, which is the front direction of the content. Further, the content management unit 234 determines the switching of the chapters of the content.

The storage unit 240 stores various data used by the control circuit unit 120 to provide the virtual space 5 to the user U. The storage unit 240 includes a template storage unit 241 and a content storage unit 242. The template storage unit 241 stores various template data. The content storage unit 242 stores data of various contents.

The template data is data representing the template of the virtual space 5. The template data has spatial structure data that defines the spatial structure of the virtual space 5. The spatial structure data is, for example, data that defines the spatial structure of a 360 ° spherical surface centered on the center 51. The template data further includes data that defines the XYZ coordinate system of the virtual space 5. The template data further has coordinate data for specifying the position of each mesh constituting the celestial sphere in the XYZ coordinate system. Further, the template data further has a flag indicating whether or not the object can be arranged in the virtual space 5.

The content is content that can be reproduced in the virtual space 5. Examples of content include platform content and viewing content.

The platform content is content related to an environment (platform) for the user U to select the viewing content that the user U wants to view in the virtual space 5. When this platform content is played back in the virtual space 5, a platform for content selection is provided to the user U. Platform content has at least a background image and data defining the object.

The viewing content is, for example, still image content or video content. The still image content has a background image. The moving image content has at least an image (still image) of each frame. The moving image content may further have audio data.

The moving image content is, for example, content generated by an omnidirectional camera. The omnidirectional camera is a camera that can generate images in all directions at once by taking pictures in all directions of the real space centered on the lens of the omnidirectional camera at once. Each image constituting the moving image content obtained by the spherical camera is distorted, but when the moving image content is reproduced in the virtual space 5, the distortion of each image is canceled by the lens constituting the display unit 112 of the HMD 110. It will be solved by that. Therefore, when playing back the moving image content, the user U can visually recognize a natural image without distortion in the virtual space 5.

In the viewing content, an initial direction (front direction), which is the direction of the image to be shown to the user U in the initial state (at the time of activation) of the HMD 110, is predetermined. The initial direction defined for the moving image content generated by the spherical camera usually corresponds to a predetermined shooting direction defined for the spherical camera used for shooting the moving image content. The initial direction may be changed to a direction different from the shooting direction. Specifically, the moving image content may be defined with the direction deviated from the shooting direction as the initial direction by appropriately editing the obtained moving image content after shooting with the spherical camera.

FIG. 9 is a diagram illustrating the configuration of moving image content C as an example of viewing content that can be viewed by user U in the virtual space 5. As shown in FIG. 9, the moving image content C includes a plurality of chapters (an example of a scene) of chapters C1 to Cn. The moving image content C is configured by connecting chapters C1 to Cn. That is, the moving image content C is divided into a plurality of chapters from chapters C1 to chapters Cn that are continuous in time.

The content storage unit 242 may store an information table associated with each moving image content. FIG. 10 is a diagram showing an information table associated with the moving image content C stored in the content storage unit 242. As shown in FIG. 10, the information table includes information in the initial direction (front direction) θ1 to θn predetermined for each chapter C1 to Cn of the moving image content C. The initial directions θ1 to θn are directions (an example of a reference direction) of an image shown to the user U wearing the HMD 110. The initial directions θ1 to θn may be directions that define the center (center coordinates) of the view area 53 displayed on the HMD 110. Further, the initial directions θ1 to θn may be directions that define an angular range in the x-axis direction, the y-axis direction, and the z-axis direction of the visual field region 53 displayed on the HMD 110.

Further, a plurality of initial directions θ1 to θn may be specified for each chapter C1 to Cn. For example, as shown in FIG. 10, the information table may store the default initial directions α1 to αn (an example of the first viewing direction) defined by the provider of the moving image content C in association with the initial directions θ1 to θn. .. The default initial directions α1 to αn usually correspond to a predetermined shooting direction defined in the spherical camera used for shooting the moving image content C. The default initial directions α1 to αn may be changed to a direction different from the shooting direction. Specifically, the directions deviated from the shooting direction may be defined as the default initial directions α1 to αn by appropriately editing the obtained moving image content after shooting with the spherical camera.

Further, the information table may store the initial directions θ1 to θn in association with the user-designated directions β1 to βn (an example of the second viewing direction) arbitrarily designated by the user U who views the moving image content C. The user designation directions β1 to βn can be arbitrarily designated by the user U by using the content selection platform provided to the user U.

As shown in FIG. 10, the default initial directions α1 to αn and the user-designated directions β1 to βn may be defined by the coordinate values (x coordinate, y coordinate, z coordinate) of the XYZ coordinate system of the virtual space 5. Further, the default initial directions α1 to αn and the user-designated directions β1 to βn may be defined based on the latitude information and the longitude information of the virtual space 5.

Further, the information table may store an object (selected object) selected by the user U among various objects included in the moving image content C in association with the initial directions θ1 to θn. Also, which of the default initial directions α1 to αn, the user-specified directions β1 to βn, and the selected objects stored in the information table should be specified as the initial directions θ1 to θn of the chapters C1 to Cn (priority of each item). ) Can also be included in the information table of the moving image content C. For example, if the information table contains user-specified directions β1 to βn and / or selected objects in addition to the default initial directions α1 to αn, user-specified directions β1 to βn and / or selected objects are included in the information table rather than the default initial directions α1 to αn. May be prioritized.

Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 11 is a diagram showing a hardware configuration of the server 2. As shown in FIG. 11, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are connected to each other so as to be able to communicate with each other via the bus 24. The control unit 23 includes a memory and a processor, and the memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

The storage unit (storage) 22 is, for example, a large-capacity HDD or the like. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, and a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 22 may store user management information for managing each user and data related to various images and objects (for example, an avatar). The communication interface 21 is configured to connect the server 2 to the communication network 3.

Next, with reference to FIG. 12, a processing method in which the distribution system 100 provides the virtual space 5 to the user U will be described.

The virtual space 5 is basically provided to the user by the cooperation of the HMD 110 and the control circuit unit 120. When the process shown in FIG. 12 is started, first, in step S1, the virtual space defining unit 231 defines the virtual space 5 by generating virtual space data representing the virtual space 5 provided to the user U. .. The procedure for generating virtual space data is as follows. First, the virtual space defining unit 231 defines the prototype of the virtual space 5 by acquiring the template data of the virtual space 5 from the template storage unit 241. The virtual space defining unit 231 further acquires the content to be reproduced in the virtual space 5 from the content storage unit 242. The virtual space defining unit 231 generates virtual space data that defines the virtual space 5 by matching the acquired contents with the acquired template data. In the virtual space data, the virtual space defining unit 231 appropriately associates each partial image constituting the background image included in the content with the management data of each mesh constituting the celestial sphere of the virtual space 5. It is preferable that the virtual space defining unit 231 associates each partial image with each mesh so that the initial direction defined in the content matches the Z direction in the XYZ coordinate system of the virtual space 5.

The virtual space defining unit 231 further adds management data of each object included in the content to the virtual space data, if necessary. At that time, the virtual space defining unit 231 sets the coordinates representing the position where the corresponding object is arranged in the virtual space 5 in each management data. As a result, each object is placed at the position of the coordinate in the virtual space 5.

After that, when the HMD 110 is activated by the user U, in step S2, the HMD sensor 114 detects the position and inclination of the HMD 110 in the initial state. Next, in step S3, the HMD sensor 114 outputs the detected value to the control circuit unit 120. Next, the HMD detection unit 211 of the control circuit unit 120 receives this detection value, and in step S4, the virtual camera control unit 221 initializes the virtual camera 300 in the virtual space 5.

The procedure for initializing the virtual camera 300 is as follows. First, the virtual camera control unit 221 arranges the virtual camera 300 at an initial position (center 51 or the like in FIG. 5) in the virtual space 5. Next, the virtual camera control unit 221 sets the orientation of the virtual camera 300 in the virtual space 5. At that time, the virtual camera control unit 221 specifies the uvw field-of-view coordinate system of the HMD 110 in the initial state based on the detection value from the HMD sensor 114, and the virtual camera 300 sets the uvw field-of-view coordinate system that matches the uvw field-of-view coordinate system of the HMD 110. By setting to, the orientation of the virtual camera 300 may be set. When the virtual camera control unit 221 sets the uvw field coordinate system in the virtual camera 300, the virtual camera control unit 221 adapts the roll direction (w axis) of the virtual camera 300 to the Z direction (Z axis) of the XYZ coordinate system. Specifically, the virtual camera control unit 221 matches the direction obtained by projecting the roll direction (depth direction) of the virtual camera 300 onto the xz plane with the Z direction of the XYZ coordinate system, and the virtual camera with respect to the xz plane. The roll direction tilt of 300 is matched with the roll direction tilt of the HMD 110 with respect to the horizontal plane. By such a matching process, the roll direction of the virtual camera 300 in the initial state is matched to the initial direction of the content, so that the horizontal direction that the user initially faces after the start of playback of the content matches the initial direction of the content. Can be made to.

When the initialization process of the virtual camera 300 is completed, in step S5, the view area determination unit 222 determines the view area 53 in the virtual space 5 based on the uvw field of view coordinate system of the virtual camera 300. Specifically, the view area determination unit 222 specifies the roll direction (w axis) of the uvw field of view coordinate system of the virtual camera 300 as the reference line of sight L of the user U, and determines the view area 53 based on the reference line of sight L. .. Next, in step S6, the view image generation unit 223 is projected onto the view area 53 in the virtual space 5 out of the entire virtual space image 52 developed in the virtual space 5 by processing the virtual space data. A field image 54 corresponding to a portion is generated (rendered). Next, in step S7, the view image generation unit 223 outputs the generated view image 54 to the HMD 110 as an initial view image.

Next, in step S8, the HMD 110 displays the received initial view image on the display unit 112. As a result, the user U visually recognizes the initial view image.
Then, in step S9, the HMD sensor 114 detects the current position and inclination of the HMD 110, and in step S10, outputs these detected values to the control circuit unit 120.

After receiving each detection value by the HMD detection unit 211 of the control circuit unit 120, in step S11, the virtual camera control unit 221 specifies the viewing direction of the HMD 110 based on the detection value. Specifically, the virtual camera control unit 221 specifies the current uvw field-of-view coordinate system in the HMD 110 based on the detected values of the position and tilt of the HMD 110, and the roll direction (w-axis) of the uvw field-of-view coordinate system in the XYZ coordinate system. Is specified as the viewing direction of the HMD 110.

Next, in step S12, the virtual camera control unit 221 specifies the viewing direction of the specified HMD 110 as the reference line of sight L of the user U in the virtual space 5. Next, in step S13, the virtual camera control unit 221 controls the virtual camera 300 based on the specified reference line of sight L. If the position (starting point) and direction of the reference line of sight L are the same as the initial state of the virtual camera 300, the virtual camera control unit 221 maintains the position and direction of the virtual camera 300 as it is. On the other hand, if the position (starting point) and / or direction of the reference line of sight L has changed from the initial state of the virtual camera 300, the virtual camera control unit 221 determines the position and / or inclination of the virtual camera 300 in the virtual space 5. , Change to the position and / or inclination according to the reference line of sight L after the change. Further, the virtual camera control unit 221 resets the uvw field-of-view coordinate system for the controlled virtual camera 300.

Next, in step S14, the view area determination unit 222 determines the view area 53 in the virtual space 5 based on the reference line of sight L specified in step S12. After that, in step S15, the view image generation unit 223 projects (superimposes) on the view area 53 in the virtual space 5 out of the entire virtual space image 52 developed in the virtual space 5 by processing the virtual space data. The view image 54, which is the part to be formed, is generated (rendered). After that, in step S16, the view image generation unit 223 outputs the generated view image 54 to the HMD 110 as a view image for update (updated view image). Then, in step S17, the HMD 110 updates the view image 54 by displaying the received updated view image 54 on the display unit 112. By such processing, when the user U moves the HMD 110, the view image 54 displayed on the display unit 112 of the HMD 110 is updated in conjunction with the movement of the HMD 110.

The processes of steps S9 to S16 shown in FIG. 12 may be executed for each frame (still image constituting a moving image). For example, when the frame rate of the moving image is 90 fps, the processes of steps S9 to S16 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). In this way, since the processes of steps S9 to S16 are repeatedly executed at predetermined intervals, the field of view of the virtual camera 300 is updated according to the operation of the HMD 110, and the field of view image 54 displayed on the display unit 112 of the HMD 110. Is updated.

The order of processing specified in each step shown in FIG. 12 is merely an example, and the order of these steps can be changed as appropriate. In the example shown in FIG. 12, the control circuit unit 120 generates virtual space data so that the initial direction defined in the content matches the Z direction in the XYZ coordinate system of the virtual space 5, and then the position of the HMD 110. And tilt are detected, and based on this detected value, the virtual space data is processed so as to match the roll direction (w axis) of the virtual camera 300 with the Z direction (Z axis) of the XYZ coordinate system to the view area 53. A method of generating a view image 54, which is a projected portion, is adopted, but the present invention is not limited to this example. As another example, the control circuit unit 120 specifies the roll direction (w-axis) of the virtual camera 300 based on the position and inclination of the HMD 110 detected by the HMD sensor 114, and the horizontal plane in the roll direction of the virtual camera 300. The XYZ coordinate system of the virtual space 5 is set so that the direction on the horizontal plane of the initial direction of the content matches the direction on the top, and the virtual space data (that is, the view image data for displaying the view image 54 on the HMD 110) is displayed. It may be generated and the generated view image data may be output to the HMD 110 as the view image 54. When such a method is adopted, virtual space data (view image data) is generated so as to match the initial direction of the content with the reference line of sight L, so that the content is similar to the example shown in FIG. The direction in which the user first faces at the start of playback can be matched with the initial direction of the content.

Next, with reference to FIG. 13, a method of reproducing the moving image content according to the present embodiment will be described. FIG. 13 is a sequence diagram showing a flow of processing for reproducing the moving image content C selected by the user U using the platform in the virtual space 5 in the distribution system 100 according to the present embodiment.

When the process shown in FIG. 13 is started, first, in step S21, the virtual space defining unit 231 of the control circuit unit 120 generates virtual space data representing the virtual space 5 for providing the platform. That is, the virtual space defining unit 231 defines the virtual space 5 for providing the platform. Specifically, the virtual space defining unit 231 acquires a summary image (thumbnail) of a certain number of viewing contents (for example, a plurality of video contents) stored in the content storage unit 242, and obtains each thumbnail. Associate with the management data of any object in the virtual space data. As a result, a thumbnail is associated with each object arranged in the virtual space 5. Each viewing content in which the corresponding thumbnail is associated with the object is a candidate (candidate video content) of the moving image content that can be selected by the user U to be played back in the virtual space 5. The user U can select the candidate moving image content corresponding to the thumbnail through the selection of the thumbnail.

Next, in step S22, the visual field image generation unit 223 generates the visual field image 54 from the virtual space data of the platform based on the visual field region 53 determined by the visual field region determination unit 222. Next, in step S23, the view image generation unit 223 outputs the generated view image 54 to the HMD 110.

Next, in step S24, the HMD 110 displays the received view image 54 on the display unit 112. After the view image 54 of the platform is displayed on the display unit 112, in step S25, the HMD sensor 114 and / or the position sensor 130 detects the orientation of the HMD 110. Then, in step S26, the HMD 110 transmits each detection value detected by the HMD sensor 114 and / or the position sensor 130 to the control circuit unit 120.

Next, in step S27, the line-of-sight detection unit 212 of the control circuit unit 120 identifies the reference line-of-sight L of the virtual camera 300 by using the detected values of the received HMD sensor 114 and / or the position sensor 130.

Next, in step S28, the line-of-sight management unit 232 of the control circuit unit 120 sets the specific thumbnail included in the view image 54 for a specified time based on the specified reference line-of-sight L and each thumbnail included in the view area 53. As described above, it is determined whether or not the reference line of sight L is hit.

In step S28, when it is determined that the reference line of sight L does not hit the specific thumbnail included in the view image 54 for a predetermined time or longer (NO in step S28), the line of sight management unit 232 returns the process to step S27.
On the other hand, when it is determined in step S28 that the reference line of sight L hits the specific thumbnail included in the view image 54 for a specified time or longer (YES in step S28), in step S29, the content specifying unit 233 is the reference for the specified time or longer. The moving image content corresponding to the thumbnail determined to be hit by the line of sight L is specified. That is, the content specifying unit 233 identifies the candidate moving image content selected by the user on the platform as the moving image content to be played back in the virtual space 5. In this example, the user U selects a thumbnail corresponding to the video content C composed of chapters C1 to Cn shown in FIG. 9, and the content specifying unit 233 specifies the video content C as the content to be played in the virtual space 5. And.
The gaze sensor 140 may detect the user's line of sight, and the detected value may be used to specify the moving image content corresponding to the thumbnail that is determined to have been hit by the user's line of sight for a specified time or longer.

Next, in step S30, the virtual space defining unit 231 defines the virtual space 5 for reproducing the moving image content by generating the virtual space data for reproducing the specified moving image content C. Specifically, first, the virtual space defining unit 231 acquires the template data of the virtual space 5 corresponding to the moving image content C from the template storage unit 241. Then, the virtual space defining unit 231 acquires the moving image content C specified by the content specifying unit 233 from the content storage unit 242. The virtual space defining unit 231 generates virtual space data that defines the virtual space 5 for playing the moving image content C by matching the acquired moving image content C with the acquired template data. In the virtual space data, the virtual space defining unit 231 appropriately associates the management data of each mesh constituting the celestial sphere of the virtual space 5 with each partial image constituting the image of the first frame included in the moving image content C. In this way, virtual space data for reproducing the specified moving image content C is generated.

After generating the virtual space data representing the virtual space 5 for reproducing the moving image content, in step S31, the view area determination unit 222 determines the view area 53 based on the reference line of sight L of the virtual camera 300.

Next, in step S32, the content management unit 234 specifies the initial direction of the moving image content C (that is, the initial direction θ1 of the chapter C1) based on the information table (see FIG. 10) stored in the content storage unit 242. do. The initial direction θ1 of the chapter C1 may be defined by, for example, the default initial direction α1 (αx1, αy1, αz1). Further, when the user-designated direction β2 (βx1, βy1, βz1) is set for the chapter C1, the initial direction θ1 may be defined by the user-designated direction β1.

Next, in step S33, the visual field image generation unit 223 is based on the visual field region 53 determined in step S31 and the initial direction θ1 of the chapter C1 of the moving image content C determined in step S32, and is the virtual space of the moving image content C. The visual field image 54 is generated from the data. Specifically, the field of view image generation unit 223 generates the field of view image 54 by associating the field of view region 53 (reference line of sight L) with the initial direction θ1. After that, in step S34, the view image generation unit 223 outputs the generated view image 54 to the HMD 110.

In step S35, the HMD 110 starts playing the chapter C1 (an example of the first scene) of the moving image content C by displaying the received view image 54 on the display unit 112. As shown in FIG. 14, when the reproduction of the video content chapter C1 is started in the HMD 110, the view is in a state where the reference line of sight L of the virtual camera 300 coincides with the initial direction θ1 of the video content chapter C1 in the virtual space 5. The image 54 is displayed. As a result, the user U starts to visually recognize the visual field image 54 of the chapter C1 with the initial direction θ1 of the chapter C1 as the front. In this example, as shown in FIG. 14, the initial direction θ1 is specified by the default initial direction α1 defined by the provider of the moving image content C.

As shown in FIG. 15, when the user U moves the HMD 110, the reference line of sight L moves in the virtual space 5 in conjunction with the movement, and the view image 54 is updated. Therefore, by appropriately moving the HMD 110, the user U can visually recognize a partial image (field of view image 54) at a desired position in the spherical image of each frame constituting the chapter C1 of the moving image content C. That is, when the user U moves the HMD 110 after the reproduction of the chapter C1 is started, the reference line of sight L of the virtual camera 300 and the initial direction θ1 of the chapter C1 do not match.

Next, in step S36, the content management unit 234 specifies the initial direction θ2 of the chapter C2 (an example of the second scene) which is the next chapter of the chapter C1 during playback of the moving image content C. The content management unit 234 specifies the initial direction θ2 of the chapter C2 (an example of the second scene) with reference to the information table CT of the moving image content C stored in the content storage unit 242. The initial direction θ2 of the chapter C2 may be defined by, for example, the default initial direction α2 (αx2, αy2, αz2). Further, when the user-designated direction β2 (βx1, βy2, βz2) is set for the chapter C2, the initial direction θ2 may be defined by the user-designated direction β2. Further, as shown in FIG. 10, when the selection object Ob1 is set for the chapter C2, the initial direction θ2 may be defined in the direction facing the selection object Ob1.

Next, in step S37, the content management unit 234 determines whether or not the reproduction of the chapter C1 is completed. The content management unit 234 is based on, for example, information on the time of each chapter C1 to Cn of the moving image content C stored in the content storage unit 242 of the storage unit 240, and the elapsed time from the start of reproduction of each chapter C1 to Cn. Therefore, it can be determined whether or not the reproduction of each chapter C1 to Cn is completed.

When it is determined by the content management unit 234 that the reproduction of the chapter C1 has been completed (YES in step S37), in step S38, the field of view area determination unit 222 has the reference line of sight of the virtual camera 300 at the end of the reproduction of the chapter C1. The field of view 53 is determined based on L.

Next, in step S39, the visual field image generation unit 223 is based on the visual field region 53 determined in step S38 and the initial direction θ2 of the chapter C2 of the moving image content C determined in step S36, and is the virtual space of the moving image content C. A visual field image 54 of chapter C2 is generated from the data. Specifically, the visual field image generation unit 223 generates the visual field image 54 of the chapter C2 so that the initial direction θ2 coincides with the visual field region 53 (reference line of sight L). The view image generation unit 223 generates the view image 54 so that, for example, the orientation of the reference line of sight L in the depth direction coincides with the orientation of the chapter C2 in the depth direction of the initial direction θ2. As described above, the virtual camera control unit 221 matches the direction obtained by projecting the roll direction of the virtual camera 300 on the xz plane with the Z direction of the XYZ coordinate system, and rolls the virtual camera 300 with respect to the xz plane. The inclination in the direction matches the inclination in the roll direction of the HMD 110 with respect to the horizontal plane. By such matching processing, the roll direction (reference line of sight L) of the virtual camera 300 and the initial direction θ2 of the chapter C2 are associated with each other. It can be matched with the initial direction θ2 of the chapter C2.
Next, in step S40, the view image generation unit 223 outputs the generated view image 54 of the chapter C2 to the HMD 110.

In step S41, the HMD 110 starts the reproduction of the chapter C2 of the moving image content C in the virtual space 5 by displaying the received view image 54 on the display unit 112. As shown in FIG. 16, when the reproduction of the chapter C2 is started in step S41, the initial direction θ2 of the chapter C2 coincides with the reference line of sight L of the virtual camera 300. That is, the reproduction of the chapter C2 is started in a state where the initial direction θ2 of the chapter C2 coincides with the direction of the reference line of sight L at the end of the reproduction of the chapter C1. As a result, the user U can start viewing the view image 54 from the initial direction θ2 of the view image 54 of the chapter C2. After that, the same processing as the processing from step S36 to step S41 is repeated for the reproduction processing after the chapter C3 of the moving image content C.

The order of processing defined in each step shown in FIG. 13 is merely an example, and the order of these steps can be changed as appropriate. In the example shown in FIG. 13, the control circuit unit 120 generates virtual space data for content reproduction by aligning the initial direction defined in the moving image content C with the Z direction in the XYZ coordinate system of the virtual space 5. Therefore, the view area 53 of each chapter C1 to Cn is determined based on the reference line of sight L of the virtual camera 300 at the start of reproduction of each chapter C1 to Cn, and each chapter C1 is set in the determined view area 53 (reference line of sight L). A method is adopted in which virtual space data is processed so as to match the initial directions θ1 to θn of Cn to generate a view image 54 of each chapter C1 to Cn, but the present invention is not limited to this example. As another example, the content storage unit 242 of the control circuit unit 120 stores content data for each chapter of the moving image content C, and the control circuit unit 120 first stores the content data at the start of reproduction of the chapters C1 to Cn. The reference line of sight L of the virtual camera 300 is specified, and the XYZ coordinate system of the virtual space 5 is set so that the initial directions θ1 to θn of the chapters C1 to Cn match the specified reference line of sight L, and each chapter C1 The virtual space data (view image data) of ~ Cn may be generated, and the view image data may be output to the HMD 110 as the view image 54. In this way, by regenerating the view image data for each chapter based on the reference line of sight L at the time of switching chapters and the initial direction of each chapter C1 to Cn, each chapter C1 is similarly generated as in the example shown in FIG. The horizontal direction in which the user U first faces after the start of reproduction of ~ Cn can be matched with the initial directions θ1 to θn of each chapter C1 to Cn.

As described above, in the present embodiment, when the control circuit unit 120 provides the virtual experience to the user U via the HMD 110, the moving image content composed of a plurality of chapters including at least chapters C1 and C2. A virtual space 5 for acquiring C and providing a virtual experience to the user U is defined, and based on the acquired video content C, the chapter C1 is adapted to the virtual space 5 and played back, and the user U wearing the HMD 110 is equipped with the chapter C1. It is configured to control the view area 53 in the virtual space 5 according to the posture of the head. Further, the control circuit unit 120 identifies the initial direction θ2 of the chapter C2, switches the reproduction scene of the moving image content C from the chapter C1 to the chapter C2, and the direction of the virtual viewpoint according to the reproduction scene being switched to the chapter C2. It is configured to start the reproduction of the chapter C2 by associating the initial direction θ2 of the chapter C2 with (reference line of sight L). That is, for example, when the front direction of the chapter C2 is defined as the initial direction θ2 of the chapter C2, no matter which direction the user U's line of sight is facing at the end of the reproduction of the chapter C1, the chapter C2 At the start of playback, the user U can be shown the front direction of the chapter C2. As a result, it is possible to provide the user U with a new and highly satisfying virtual experience, and it is possible to improve the virtual experience of the user U.

Further, for example, a plurality of initial directions θ1 to θn (for example, default initial directions α1 to αn and user-specified directions β1 to βn) are set in each chapter C1 to Cn of the moving image content C, which is the content of the 360-degree moving image. May be. The control circuit unit 120 specifies a direction (an example of a first reference direction) to be adopted at the start of reproduction of the chapter C2 from a plurality of initial directions θ2 set for the chapter C2, and the reproduction scene is a chapter. Depending on the switching from C1 to chapter C2, the reference line of sight L may be configured to correspond to the first reference direction and the reproduction of chapter C2 may be started. When a plurality of initial directions are set for each chapter C1 to Cn, the moving image content can be set by setting a specific direction as the first reference direction from the plurality of predetermined initial directions in this way. Reproduction of each chapter C1 to Cn can be started in a state where the reference line of sight L faces in an appropriate direction according to the content of C and the like. Therefore, the virtual experience of the user U can be further improved.

Further, the initial directions θ1 to θn set in each chapter C1 to Cn may be the default initial directions α1 to αn (first viewing direction) set by the provider side of the moving image content C. As a result, the reproduction of each chapter C1 to Cn can be started with the reference line of sight L facing in the direction intended by the provider of the moving image content C, so that the virtual experience of the user U can be further improved.

The initial directions θ1 to θn set in each chapter C1 to Cn may be user-designated directions β1 to βn (second viewing direction) arbitrarily set by the user U. In this case, for example, a menu for setting an arbitrary direction as the initial directions θ1 to θn may be presented to the user U on the platform used when selecting the moving image content C. By selecting the user-designated directions β1 to βn of preference as the initial directions θ1 to θn, the user U can start the reproduction of the chapter C2 with the reference line of sight L facing in the direction suitable for the preference of the user U. can. As a result, each chapter C1 to Cn is viewed from a maniac viewpoint other than the default initial directions α1 to αn (that is, the front direction intended by the video content C provider) set in each chapter C1 to Cn by the video content C provider. It is possible to prevent the user U who wants to start viewing the desired scene from being overlooked, and it is possible to improve the satisfaction level of the user U.

In the present embodiment, as described above, the user U can set any user-designated direction as the initial direction θ1 to θn for each of the plurality of chapters C1 to Cn. As a result, it is possible to provide a field image that reflects the preference of the user U for each scene of the moving image content, so that the satisfaction level of the user U can be further improved.

Further, the initial directions θ1 to θn set in each chapter C1 to Cn are arranged in the virtual space 5 and face toward a predetermined object (selected object) selected by the user U (third viewing direction). It may be. For example, as shown in FIG. 17, when a plurality of persons Ob1 to Ob3 appear in the video content C (concert, etc.), the control circuit unit 120 is on the platform used when selecting the video content C. , A menu for selecting a specific person may be presented to the user U. When the user U selects the object Ob1 as the selected object in the chapter C2, the control circuit unit 120 may set the direction toward the selected object Ob1 as the initial direction θ2 of the chapter C2. According to this method, the reproduction of each chapter C1 to Cn can be started in a state where the direction of the object selected by the user U is aligned with the reference line of sight L, so that the satisfaction level of the user U can be improved.

By selecting different objects for each of the plurality of chapters C1 to Cn of the moving image content, different initial directions θ1 to θn may be set for each chapter C1 to Cn. As a result, even if what the user U wants to see changes depending on the scene, the viewing of each chapter C1 to Cn can be started in the direction according to the preference of the user U, so that the satisfaction of the user U is further improved. Can be improved.

Further, the control circuit unit 120 has a direction in the horizontal plane in the depth direction of the reference line of sight L and a direction in the horizontal plane in the initial direction θ2 of the chapter C2 in response to the switching of the reproduction scene from the chapter C1 to the chapter C2. It is preferable to start the reproduction of the chapter C2 together with the above. As a result, the direction of the reference line of sight L and the direction of the initial direction θ2 can be easily and appropriately matched.

Further, in the description of the above embodiment, it is assumed that the virtual space data indicating the virtual space 5 is updated on the user terminal 1 side, but the virtual space data may be updated on the server 2 side. Further, although it is assumed that the view image data corresponding to the view image is updated on the user terminal 1 side, the view image data may be updated on the server 2 side. In this case, the user terminal 1 displays the field image on the HMD 110 based on the field image data transmitted from the server 2.

Further, in the above embodiment, the data for reproducing the viewing content is stored in advance in the content storage unit 242 of the control circuit unit 120, but the present invention is not limited to this example. For example, data for viewing content such as 360 moving images may be stored in the storage unit 22 of the server 2. In this case, the control circuit unit 120 generates a distribution request signal indicating a distribution request of the data for viewing content based on the user input, and transmits the distribution request signal to the server 2. Based on the received distribution request signal, the server 2 distributes the video content data corresponding to the distribution request signal to the control circuit unit 120. In this way, the server 2 may be configured to distribute the video content data to the user terminal 1 in a streaming format. Further, the information about the user-designated directions β1 to βn and the selected object included in the information table shown in FIG. 10 is not limited to the configuration predetermined in advance in the content storage unit 242 of the control circuit unit 120, and is in a streaming format from the server 2. It may be configured to be delivered by.

Further, in the present embodiment, the virtual space (VR space) in which the user is immersed by the HMD 110 has been described as an example, but a transmissive HMD may be adopted as the HMD. In this case, by outputting a view image obtained by synthesizing a part of the image constituting the virtual space in the real space visually recognized by the user via the transmissive HMD, the augmented reality (AR) space or mixed reality (AR) space or mixed reality (AR) MR: Mixed Reality) A virtual experience in space may be provided to the user.

Further, the virtual camera 300 may include a virtual camera for the left eye and a virtual camera for the right eye. In this case, the control circuit unit 120 generates left-eye view image data showing a left-eye view image based on the virtual space data and the field of view of the left-eye virtual camera. Further, the control circuit unit 120 generates right-eye view image data showing a right-eye view image based on the virtual space data and the field of view of the right-eye virtual camera. After that, the control circuit unit 120 displays the left-eye view image on the left-eye display unit based on the left-eye view image data, and displays the right-eye view on the right-eye display unit based on the right-eye view image data. Display the image. In this way, the user U can visually recognize the visual field image 54 three-dimensionally by the parallax between the visual field image for the left eye and the visual field image for the right eye.

Further, in order to realize various processes executed by the control circuit unit 120 of the user terminal 1 by software, a control program for causing a computer (processor) to execute various processes is preliminarily incorporated in the storage medium 123 or the memory. May be good. Alternatively, the control program includes a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), an optical magnetic disk (MO, etc.), a flash memory (SD card, USB memory, SSD). Etc.) may be stored in a computer-readable storage medium such as. In this case, by connecting the storage medium to the control circuit unit 120, the control program stored in the storage medium is incorporated into the storage medium 123. Then, the control program incorporated in the storage medium 123 is loaded onto the RAM, and the processor 122 executes the loaded program, so that the control circuit unit 120 executes various processes.

Further, the control program may be downloaded from the computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded control program is incorporated into the storage medium 123.

Although the embodiments of the present disclosure have been described above, the technical scope of the present invention should not be construed as being limited by the description of the present embodiments. This embodiment is an example, and it is understood by those skilled in the art that various embodiments can be modified within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the scope thereof.

1: User terminal 2: Server 3: Communication network 21: Communication interface 22: Storage unit 23: Control unit 24: Bus 5: Virtual space 51: Center 52: Virtual space image 53: View area 54: View image 100: Virtual space Delivery system (delivery system)
110: HMD (Head Mounted Device)
112: Display unit 114: HMD sensor 116: Headphones 118: Microphone 120: Control circuit unit 123: Storage medium 124: I / O interface 125: Communication interface 126: Bus 130: Position sensor 140: Gaze sensor 210: Detection unit 211: HMD detection unit 212: Line-of-sight detection unit 213: Operation reception unit 220: Display control unit 221: Virtual camera control unit 222: View area determination unit 223: View image generation unit 230: Virtual space control unit 231: Virtual space regulation unit 232: Line-of-sight management unit 233: Content identification unit 234: Content management unit 240: Storage unit 241: Template storage unit 242: Content storage unit 250: Communication unit 300: Virtual camera 320: External controller 320L: External controller for the left hand (controller)
320R: External controller for right hand (controller)
U: Users Ob1 to Ob3: Object

Claims (10)

A program for providing a virtual experience to the user via an image display device associated with the user's head.
A step of defining a virtual space for providing the virtual experience and displaying a view image including thumbnails of a plurality of video contents on the virtual space, and here, each of the plurality of video contents is temporal. It is composed of at least the first scene and the second scene, which are continuous with each other.
The step of detecting the line of sight of the user and
Among the plurality of video contents, the step of playing the first scene of the video content corresponding to the thumbnail in which the line of sight is detected for a predetermined time or longer in accordance with the virtual space is performed.
A step of controlling the field of view from the reference line of sight in the virtual space according to the line of sight,
A step of specifying the reference direction of the second scene of the moving image content corresponding to the thumbnail, and
A step of automatically switching the playback scene of the moving image content from the first scene to the second scene when the playback of the first scene ends.
A step of starting the reproduction of the second scene based on the direction of the reference line of sight at the end of the reproduction of the first scene in response to the switching to the second scene.
A program that lets your computer run. Each of the plurality of video contents is a 360-degree video content, and is
A plurality of the reference directions are set in the second scene.
Further, the computer is made to execute a step of selecting the first reference direction adopted at the start of playback of the second scene from the plurality of reference directions.
In the specifying step, the first reference direction is specified as the reference direction of the second scene.
In the step of starting the reproduction, the reproduction of the second scene is started by associating the first reference direction with the direction of the reference line of sight in response to the switching to the second scene. The program described in. The program according to claim 1 or 2 , wherein the reference direction is a first viewing direction set by the provider of the moving image content. The program according to any one of claims 1 to 3 , wherein the reference direction is a second viewing direction set by the user. The program according to claim 4 , wherein the second viewing direction can be set as the reference direction for each scene. The program according to any one of claims 1 to 5 , wherein the reference direction is a third viewing direction toward an object designated by the user. The reference direction is a third viewing direction toward the object specified by the user.
The program according to claim 2 , wherein in the selection step, the third viewing direction is selected as the first reference direction for each scene by selecting the object. In the step of starting the reproduction, the orientation of the second scene on the horizontal plane in the depth direction of the reference line of sight is aligned with the orientation of the second scene on the horizontal plane in the reference direction in response to the switching to the second scene. The program according to any one of claims 1 to 7 , wherein the reproduction of the second scene is started. An information processing device for providing a virtual experience to the user via an image display device associated with the user's head.
By controlling the processor included in the information processing device
A step of defining a virtual space for providing the virtual experience and displaying a view image including thumbnails of a plurality of video contents on the virtual space, and here, each of the plurality of video contents is temporal. It is composed of at least the first scene and the second scene, which are continuous with each other.
The step of detecting the line of sight of the user and
Among the plurality of video contents, the step of playing the first scene of the video content corresponding to the thumbnail in which the line of sight is detected for a predetermined time or longer in accordance with the virtual space is performed.
A step of controlling the field of view from the reference line of sight in the virtual space according to the line of sight,
A step of specifying the reference direction of the second scene of the moving image content corresponding to the thumbnail, and
A step of automatically switching the playback scene of the moving image content from the first scene to the second scene when the playback of the first scene ends.
A step of starting the reproduction of the second scene based on the direction of the reference line of sight at the end of the reproduction of the first scene in response to the switching to the second scene.
Is executed, an information processing device. A method of information processing performed by a computer to provide a virtual experience to the user via an image display device associated with the user's head.
A step of defining a virtual space for providing the virtual experience and displaying a view image including thumbnails of a plurality of video contents on the virtual space, and here, each of the plurality of video contents is temporal. It is composed of at least the first scene and the second scene, which are continuous with each other.
The step of detecting the line of sight of the user and
Among the plurality of video contents, the step of playing the first scene of the video content corresponding to the thumbnail in which the line of sight is detected for a predetermined time or longer in accordance with the virtual space is performed.
A step of controlling the field of view from the reference line of sight in the virtual space according to the line of sight,
A step of specifying the reference direction of the second scene of the moving image content corresponding to the thumbnail, and
A step of automatically switching the playback scene of the moving image content from the first scene to the second scene when the playback of the first scene ends.
An information processing method including a step of starting reproduction of the second scene based on the direction of the reference line of sight at the end of reproduction of the first scene in response to switching to the second scene .

Images