JP2018166279A - Video distribution apparatus, system, program and method for distributing video according to client state - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video distribution apparatus capable of distributing, to a client concerned, video data which enables the reproduction of a video according to a client state.SOLUTION: The video distribution apparatus for distributing a video of a virtual space to a client comprises: state-prediction means for acquiring information concerning a state of the client at a point of time or in a time zone to predict the client state at another point of time or in another time zone; virtual state deciding means for deciding a virtual state in a virtual space corresponding to the predicted state at the other point of time or in the other time zone; video production means for producing a video to be distributed based on the virtual state thus decided; and video distribution control means for distributing the produced video to the client in line with reproduction at the other point of time or in the other time zone. In the video distribution apparatus, the video to be distributed is a viewpoint video image in the virtual space; and the information concerning the state is preferably a piece of information concerning at least one of a position, a velocity, an acceleration and a direction of the client.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a reception control technique in a client that acquires a large amount of data from a server.

  As one of media capable of providing a high level of realism in the next generation, VR (Virtual reality) images are attracting attention. VR video is, for example, video that can provide a user with a HMD (Head Mounted Display) a moving image in a 360-degree field of view in a virtual space according to the orientation of the head. It is.

  In order to display a VR video, it is necessary to prepare a panoramic video that is a two-dimensional video developed in a 360-degree field of view in a client such as an HMD. This panoramic video is an ERP (Equi-Rectangular Projection) format (VR) generated using a camera group that supports 360-degree field of view shooting and VR content generated using 3D computer graphics (CG) technology ( It is generated by developing it into a two-dimensional image by a mapping technique such as equirectangular projection. Incidentally, the client converts the received panoramic image for each frame by a predetermined conversion formula, and generates and displays a VR image in a form in which the virtual space can be visually recognized with both eyes, for example.

  Here, the panoramic video is generally encoded by the existing encoding method and delivered to the client in a compressed form. However, according to the ERP format, the panoramic video is converted into a two-dimensional video corresponding to high resolution display such as 4K or 8K. It will be developed. Therefore, the bit rate becomes high, and in particular when distributing via a communication network, the distribution delay increases.

  For example, Patent Document 1 proposes a solution for saving the bit rate of panoramic video in consideration of the visible region. Specifically, the panoramic image is divided into slices, different resolutions and bit rates are set for each slice, and the quality of the user's visible area is set high, while the quality of the non-visible area is reduced, The bit rate is reduced.

  Conventionally, a protocol called HLS (HTTP Live Streaming) has been used as a delivery method capable of dealing with delivery delay. In this method, the video stream is divided into segment files having a playback time length of about several to 10 seconds in advance, and the client side performs reception and playback start in units of segments, thereby playing back while receiving the video stream. Pseudo streaming is realized.

  Furthermore, video streams having different qualities (bit rates) are prepared, and segment files having different bit rates are prepared for each segment, thereby selecting a bit rate for each segment. As a result, the quality of the content requested by the client is changed according to the communication status of the communication network, and the delivery delay is dealt with. Similar known distribution protocols include MPEG-DASH (Dynamic Adaptive Streaming over Hyper transfer protocol).

JP 2006-105593 A

  Certainly, in the prior art including the technique described in Patent Document 1, it is possible to cope with the delivery delay by adjusting the bit rate of the divided image file such as the segment file. However, considering the distribution of panoramic video of VR content, the problem of distribution delay still remains.

  Here, specifically, the following distribution form is considered. That is, a panoramic image in which the viewpoint position of the VR content is moved in conjunction with the movement of the car in the real world is distributed to the HMD worn by the user who gets on the vehicle. Thus, this user is provided with an experience of moving in the virtual space of VR content.

  In this case, the vehicle moves or changes its direction of travel even after the content request with the vehicle position as the transmission source is transmitted to the distribution server and before the playback of the VR video is started with the HMD of the user who gets on the vehicle. And changing the state in the real world every moment. Therefore, even if the HMD receives the panoramic video generated by the distribution server according to the content request and reproduces it as the VR video using the conventional technology as described above, the state in the virtual space provided by the VR video, There will be a deviation from the state of the car at that time. As a result, there is a problem that the realism of the virtual experience deteriorates and the user cannot obtain a sufficient immersive feeling.

  As described above, the problem that the distribution video is shifted with respect to the change in the state of the client as a result of the distribution delay cannot be solved by the bit rate adjustment technique described in Patent Document 1, for example.

  Therefore, an object of the present invention is to provide a video distribution apparatus, system, program, and method that can distribute video data that enables playback of video according to the state of the client to the client.

According to the present invention, there is provided a video distribution device that distributes video in a virtual space to a client,
State prediction means for acquiring information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Virtual state determination means for determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
There is provided a video distribution apparatus including a video distribution control unit that distributes a generated video to the client in accordance with reproduction at the other time point or time interval.

As one embodiment of the video distribution apparatus according to the present invention, the video is a viewpoint video in the virtual space,
The state prediction means obtains the state information relating to at least one of the position, speed, acceleration, and orientation at one time point or time interval as the state of the client, and the position at another time point or time interval Predicting state information relating to at least one of speed, acceleration and orientation,
The virtual state determination means determines virtual state information related to at least one of position, velocity, acceleration, and direction in the virtual space corresponding to the predicted state information at the other time point or time interval. ,
The video generation means determines the viewpoint information related to at least one of the position, speed, acceleration, and orientation of the viewpoint in the virtual space based on the determined virtual state information, and based on the determined viewpoint information It is also preferable to generate a viewpoint video.

  Moreover, in said embodiment, it is also preferable that a state prediction means estimates the state information which concerns on the direction in the said other time point or time interval based on the route map information containing the said client's location. Further, the state predicting means may obtain information on angular acceleration, which is information related to the direction at the one time point or time interval, and predict state information related to the direction at the other time point or time interval. preferable.

  Furthermore, it is preferable that the video distribution apparatus according to the present invention further includes a predicted time determination unit that determines the other time point or time interval based on the acquired information related to the distribution delay time.

According to the present invention, there is also provided a video distribution system having a client and a video distribution device that distributes video in a virtual space to the client,
State information generating means for generating information relating to its own state detected, measured or determined in one time point or time interval;
Client communication control means for transmitting the information relating to the generated state to the video distribution device in association with the one time point or time interval, and the video distribution device described above,
State prediction means for acquiring information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Virtual state determination means for determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
There is provided a video distribution system including a video distribution control unit that distributes a generated video to the client in accordance with reproduction at the other time point or time interval.

As one embodiment of the video distribution system according to the present invention, the video distribution device further includes device communication control means for transmitting the predicted state at the other time point or time interval to the client,
The client is based on the difference in the numerical value representing the state between the received state at the other time point or time interval and the state at the other time point or time interval detected, measured or determined. It is also preferable to further include a reproduction control means for adjusting the reproduction speed of the video related to the other time point or time interval so that the virtual state appearing in the reproduced video matches the state of the client.

According to the present invention, there is further provided a video distribution program for causing a computer mounted on a video distribution device for distributing video in a virtual space to a client,
State prediction means for acquiring information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Virtual state determination means for determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
There is provided a video distribution program for causing a computer to function as a video distribution control means for distributing the generated video to the client in accordance with reproduction at the other time point or time interval.

According to the present invention, there is further provided a video distribution method in a computer mounted on a video distribution apparatus that distributes video in a virtual space to a client,
Obtaining information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
A video distribution method including a step of distributing the generated video to the client in accordance with reproduction at the other time point or time interval.

  According to the video delivery apparatus, system, program, and method of the present invention, video data that enables playback of video according to the state of the client can be delivered to the client.

It is a functional block diagram which shows the function structure in one Embodiment of the video delivery apparatus by this invention. It is a functional block diagram which shows the function structure in one Embodiment of the client which concerns on this invention. It is a sequence diagram which shows roughly one Embodiment of the image | video delivery method by this invention. It is a schematic diagram for demonstrating one comparative example which distributes without considering progress of a client state in a delivery server. It is a schematic diagram for demonstrating one Example which performs a delivery process by estimating a client state in the video delivery apparatus by this invention. It is a schematic diagram for demonstrating the other Example which estimates a client state and performs a delivery process in the video delivery apparatus by this invention. It is a schematic diagram for explaining one embodiment of the reproduction control process in the client according to the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Video distribution system]
FIG. 1 is a functional block diagram showing a functional configuration in an embodiment of a video distribution apparatus according to the present invention.

  A distribution server 1 as an embodiment of a video distribution apparatus according to the present invention shown in FIG. 1 distributes a panoramic video based on VR (Virtual reality) content to a client HMD (Head Mounted Display) 2 in real time. This is a server that allows a user wearing the HMD 2 to enjoy a VR video that is a video in a virtual space (VR space, virtual world).

  Here, in the present embodiment, the user wearing the HMD2 is on a running bus, and the HMD2 (user), along with the bus, has a “state” in the real world such as position, speed, acceleration, and traveling direction. It is changing every moment. In the present embodiment, the bus or the HMD 2 has a positioning function by GPS (Global Positioning System), and transmits positioning information (state information) in which its own measuring position is linked to the positioning time to the distribution server 1. can do.

Similarly, as shown in FIG. 1, the distribution server 1 has the following distinctive features:
(A) The state prediction unit 111 that acquires information related to the state at one time point or time interval of the HMD 2 and predicts the “state at another time point or time interval” of the HMD 2;
(B) a virtual state determination unit 113 that determines a “virtual state” in the virtual space corresponding to the predicted “state at another time point or time interval”;
(C) a video generation unit 114 that generates a video to be distributed based on the determined “virtual state”;
(D) A video distribution control unit 115 that distributes the generated video to the HMD 2 in accordance with playback at another time point or time interval.

  As described above, the distribution server 1 predicts the “virtual state” in the virtual space in accordance with the reproduction time of the future client from the acquired “state” of the client, and generates a distribution video based on this. Since such a characteristic state prediction function is provided, video data that enables playback of video according to the client state can be distributed to the client.

  Incidentally, as a typical embodiment, the “state” in the real world can be at least one of position, velocity, acceleration, and traveling direction in the real space. If the bus or HMD 2 transmits GPS positioning information to the distribution server 1 as in the present embodiment, at least the position is used as the “state”. In this case, the state prediction unit 111 of the distribution server 1 acquires at least position information at one time point or time interval as the state of the HMD 2 and predicts at least position information at another time point or time interval.

  Furthermore, the virtual state determination unit 113 of the distribution server 1 determines virtual position information related to at least a position in the virtual space corresponding to the predicted position information at another time point or time interval. Next, the video generation unit 114 determines viewpoint information related to at least the position of the viewpoint in the virtual space based on the determined virtual position information, and generates a viewpoint video in the virtual space based on the determined viewpoint information. To do.

Here, after the HMD 2 (or bus) transmits the positioning information (status information) in response to the panoramic video distribution request, the HMD 2 receives the panoramic video as a response and starts reproducing the VR video. Of course,
(A) Communication path delay, that is, RTT (Round-Trip Time) + (time required for distribution in the server 1, and
(B) A delay occurs due to the buffering time after reception on the HMD2 side. As a result, with the conventional video distribution technology, VR video corresponding to a position that is different from the position in the real world system at the time of reproduction is provided.

  On the other hand, the distribution server 1 according to the present embodiment distributes the distribution video (panoramic video) in accordance with the playback time in the HMD 2 as described above. The realism of the virtual experience is improved, and the user can enjoy a sufficient immersive feeling.

  Of course, the client that is the distribution destination of the distribution server 1 is not limited to the HMD of the user who gets on the bus. Various clients such as glasses (glasses) wearable terminals can be used as the client as long as the video in the virtual space can be reproduced. Further, what the client gets on can be a transportation means such as an automobile, a railway vehicle, a ship, a bicycle, or the like. Further, the client may be a terminal worn and carried by a user in a walking state or a running state. In this case, it is preferable that the client is a client whose movement path is fixed to some extent, abrupt changes in position, speed, and acceleration hardly occur and movement prediction is relatively easy.

[Functional configuration of device, video distribution method]
Similarly, according to the functional block diagram shown in FIG. 1, the distribution server 1 which is the video distribution apparatus of the present invention includes a communication interface unit 101, a VR (Virtual Reality) content storage unit 102, a panoramic image storage unit 103, , A panoramic video storage unit 104 and a processor memory. Here, the processor memory realizes the state prediction / video generation / video distribution control function according to the present invention by executing a program that causes the computer of the distribution server 1 to function.

  Furthermore, this processor memory includes a state prediction unit 111 including a position prediction unit 111a, a speed / acceleration prediction unit 111b, and a direction prediction unit 111c, a prediction time determination unit 112, and a virtual state determination unit 113 as functional components. A video generation unit 114 including a panoramic image generation unit 114a and a panoramic video generation unit 114b, a video distribution control unit 115, and a communication control unit 116. Here, the processing flow shown by connecting the functional components of the distribution server 1 in FIG. 1 with arrows is understood as an embodiment of the video distribution method according to the present invention.

  First, the communication control unit 116 receives a request for requesting delivery of a panoramic image of VR content from the HMD 2 via the communication interface unit 101. On the other hand, information (state information) related to the position, speed, and acceleration measured by the HMD2 or the bus is received from the bus on which the user of the HMD2 or HMD2 is riding. Here, it is also preferable that this state information is transmitted together with the request for distribution request and received by the communication control unit 16, but for example, periodically or when a change in the state exceeds a predetermined value, the HMD 2 or the bus It is also possible to transmit from.

  Further, the communication control unit 116 determines a communication path delay time (RTT + (time required for distribution in the server 1)) based on the transmission / reception status with the HMD 2 (or bus) such as the above request and status information. Further, the distribution delay time τ, that is, the HMD 2 transmits a panoramic video distribution request after combining the buffering time after reception on the HMD 2 side, which has been acquired or set in advance, as appropriate, and then the panorama as a response from the HMD 2 The time required for receiving the video and starting playback of the VR video is calculated.

  The prediction time determination unit 112 determines a prediction time point or a time interval in which a state is to be predicted in the state prediction unit 111 described below, based on information related to the delivery delay time τ input from the communication control unit 116. For example, when state information at time t = 0 (information on position, velocity, and acceleration) is received, t = τ is determined as a predicted time to be predicted. Alternatively, t = τ to τ + s may be determined as a prediction time interval to be predicted, where s is a time (reproduction time) required to reproduce one file (which is a video distribution unit). Incidentally, the delivery delay time τ can be determined by statistically processing the past actual value of the communication path delay time (RTT + (time required for delivery in the server 1)).

  The state prediction unit 111 is a prediction time determination unit 112 based on state information (information related to position, speed, and acceleration) at one time point or time interval of the HMD 2 (or bus) input from the communication control unit 116. The state of the HMD 2 (or bus) at the determined prediction time point or prediction time interval is predicted. In particular, state information related to at least one of position, speed, acceleration and direction at one time point or time interval is acquired, and among the position, speed, acceleration and direction at this prediction time point or prediction time interval, It is also preferable to predict state information related to at least one of the following.

  In the present embodiment, in the state prediction unit 111, the position prediction unit 111a, the speed / acceleration prediction unit 111b, and the direction prediction unit 111c of the state prediction unit 111 predict the position, speed, and acceleration, and the traveling direction, respectively. A specific mode of such state prediction will be described in detail later with reference to FIGS. 4, 5, and 6.

  Similarly, in FIG. 1, the virtual state determination unit 113 determines a virtual state in the virtual space corresponding to the state at the prediction time point or the prediction time interval. Here, the virtual space is a VR space set in the VR content to be distributed, and is usually a space representing a three-dimensional VR world corresponding to the real world (space) that can be recognized by humans. Specifically, the virtual state determination unit 113 determines virtual state information related to at least one of the position, speed, acceleration, and orientation in the virtual space corresponding to the state information at the prediction time point or the prediction time interval. It is also preferable.

  The video generation unit 114 generates a video to be distributed based on the virtual state determined by the virtual state determination unit 113. In this embodiment, based on the determined virtual state information, viewpoint information related to at least one of the position, speed, acceleration, and orientation of the viewpoint in the virtual space is determined, and based on the determined viewpoint information Generate viewpoint video.

  Specifically, the panorama image generation unit 114a of the video generation unit 114 is a VR content generated using a three-dimensional CG (Computer Graphics) technology, and the VR content stored in the VR content storage unit 102 is displayed. The image is expanded into a panoramic image as a two-dimensional expanded image by a mapping technique such as ERP (Equi-Rectangular Projection) format, and the generated panoramic image is stored in the panoramic image storage unit 103.

  Next, the panoramic video generation unit 114b of the video generation unit 114 reads and adds the generated and stored panoramic images for a frame corresponding to one file (for example, a file such as .ts or .m4s) for distribution. Encode and generate panoramic video as 2D video. Here, the generated panoramic video may be appropriately stored in the panoramic video storage unit 104 as buffering until distribution, for example.

  The communication control unit 116 transmits the generated panoramic video to the video requesting HMD 2 via the communication interface unit 101. At this time, HLS, MPEG-DASH, SmoothStreaming, RTP, RTSP, RTMP, webRTC, HTTP progressive download, or the like is used as a distribution protocol.

  The video distribution control unit 115 controls the communication control unit 116 and the communication interface unit 101 so as to distribute the generated video (panoramic video as a viewpoint video) to the HMD 2 in accordance with the playback at the prediction time point or the prediction time interval. To do. For example, when t = τ to τ + s (τ: delivery delay time, s: playback time for one file) is determined as a predicted time interval in the situation where state information at time t = 0 is received, It is also preferable to control so that a panoramic image for one file is distributed to the HMD 2 at a timing (timing in time for reproduction) in synchronization with reproduction from t = τ.

  FIG. 2 is a functional block diagram showing a functional configuration in an embodiment of the client according to the present invention.

  According to FIG. 2, the HMD 2 as an embodiment of the client according to the present invention includes a communication interface unit 201, a display 206, a communication control unit 211, and a playback control unit 212.

  Moreover, HMD2 further has the positioning part 202, the acceleration sensor 203, the gyro sensor 204, the geomagnetic sensor 205, and the state information generation part 213 in this embodiment. Here, an embodiment in which these components are not included in the HMD 2 but are provided in a bus on which the user wearing the HMD 2 is riding is also possible. In this case, the bus preferably further includes a communication control / interface unit capable of transmitting the generated state information to the distribution server 2. Incidentally, many moving means such as passenger cars and buses have a GPS positioning function and can transmit positioning information to an external server. Furthermore, the bus may be equipped with an OBD (On-Board Diagnositics) 2 that is a self-diagnosis function, and the diagnosis information by this function may be transmitted to the distribution server 2 as bus state information.

  The positioning unit 202 captures a positioning radio wave from the GPS satellite 3 and performs positioning processing, and outputs the latitude, longitude, and altitude (when geoid height information is present) at the time of positioning of the HMD 2 (or bus).

  The acceleration sensor 203 is an acceleration meter that measures and outputs the acceleration at the time of measurement of the HMD 2 (or bus) as a vector quantity. Speed information may be output by integrating the measured acceleration. In addition, as an example of a change, an example is a client worn by a pedestrian user, but the acceleration sensor 203 measures gravitational acceleration, counts the number of steps of the user wearing the HMD2, and walks from a preset stride. The distance may be calculated and output.

  The gyro sensor 204 is a gyroscope that detects and outputs an angular velocity at the time of detection of the HMD 2 (or bus). Direction change (change) may be detected and direction change information may be output.

  The geomagnetic sensor 205 is a magnetometer that measures and outputs the direction and intensity of geomagnetism at the time of measurement of the HMD2 (or bus) position. The geomagnetic sensor 205 and the acceleration sensor 203 may be combined to form an orientation measuring unit that measures and outputs the orientation in which the HMD 2 (or bus) faces (or travels).

  Of course, the types and combinations of the sensors to be equipped are not limited to the above-described forms, and any sensor that can detect, measure, or determine the state of the HMD 2 (or bus) in the real world (real space). Various sensors can be used alone or in combination.

  The state information generation unit 213 detects, measures, or determines the HMD 2 (or the time interval or time interval) based on the sensor output information output from the positioning unit 202, the acceleration sensor 203, the gyro sensor 204, and the geomagnetic sensor 205 (or Bus) state information, for example, information related to position, velocity, and acceleration.

  The communication control unit 211 associates the state information generated by the state information generation unit 213 with the one time point or time interval, and transmits the state information to the distribution server 1 via the communication interface unit 201. In addition, a request for distributing the corresponding panoramic video is transmitted to the distribution server 1 in order to provide the user with the VR video of the VR content. Furthermore, the communication control unit 211 receives the panoramic video distributed in response to the request from the distribution server 1 and outputs it to the reproduction control unit 212.

  The playback control unit 212 performs decoding processing on the input panoramic video, converts each frame using a conversion formula corresponding to a format such as ERP used for mapping on the server 1 side, and displays the VR video on the display 206. Let it play. At this time, for example, in the situation where the status information at time t = 0 is transmitted to the server 1, the reproduction is performed in the time interval t = τ to τ + s (τ: delivery delay time, s: playback time for one file). When the corresponding panoramic image is received, the reproduction control unit 212 generates and buffers the VR image so as to synchronize with the reproduction from the time t = τ (in time), and generates the VR generated from the time t = τ. An image can be displayed on the display 206.

  As a result, the state such as the position of the bus (and HMD2) in the real world and the virtual state in the displayed virtual space coincide with each other without time lag, and as a result, the realism of the virtual experience of the user is increased. It is improved and the user can enjoy a sufficient immersive feeling.

  FIG. 3 is a sequence diagram schematically illustrating an embodiment of a video distribution method according to the present invention. In the present embodiment, the position, speed, and acceleration values are employed as the status information of the client (HMD2).

(S101) The HMD 2 measures its own position, speed, and acceleration at this point.
(S102) The HMD 2 generates state information in which the measured position, velocity, and acceleration values are associated with the measurement time.
(S103) The HMD 2 transmits the generated state information to the distribution server 1.

(S104) The distribution server 1 predicts values of position, speed, and acceleration (in real space) at a predicted time corresponding to a future reproduction time based on the received state information.
(S105) The distribution server 1 determines the position, speed, and acceleration values at the predicted time in the virtual space as a virtual state based on the predicted position, speed, and acceleration values (in real space).

(S106) The distribution server 1 cuts out a panoramic image from the VR content based on the determined virtual state.
(S107) The distribution server 1 collectively encodes the cut out panoramic images and generates a panoramic video.
(S108) The distribution server 1 transmits the generated panoramic image and the predicted position, velocity, and acceleration values at the predicted time to the HMD 2.

(S109) The HMD 2 converts the received panoramic video into a VR video that can be viewed by the user.
(S110) The HMD 2 measures its position, speed, and acceleration at the playback start time (of the playback process to be performed in the future).
(S111) The HMD 2 reproduces the VR video while controlling the reproduction speed based on the comparison of the received position, speed, and acceleration at the predicted time with the values of the position, speed, and acceleration measured in Step S110.

  Hereinafter, the panoramic video generation processing in accordance with the client state in steps S104 to S107 will be described in detail with reference to FIGS. Further, the panoramic video reproduction control process in steps S110 and S111 will be described in detail with reference to FIG. Further, the client orientation prediction process will be described in detail with reference to FIG.

  FIG. 4 is a schematic diagram for explaining a comparative example in which distribution is performed without considering the progress of the client state in the distribution server. FIG. 5 is a schematic diagram for explaining an embodiment in which a client state is predicted and distribution processing is performed in the video distribution apparatus according to the present invention. In these comparative examples and examples, it is assumed that the user wearing the HMD 2 does not move in the bus car, and the position, speed, and acceleration values transmitted from the HMD 2 match those of the bus.

  First, according to the comparative example of FIG. 4 in which the progress of the client state is not taken into account, the bus travels on the road in the real world (real space) and passes through the position 1 to the position 2 in this road, and the time t = 0 Now we ’re at position 3. At time t = 0, the HMD 2 transmits position information in which the coordinates of these positions 1 to 3 are associated with time information (t = 0) to the distribution server.

  The distribution server that has received the position information grasps the position of the HMD 2 (bus) in the real space from the received position information, and converts the position into a virtual position in the virtual space as it is. Specifically, the distribution server calculates the coordinates of the virtual positions 1 to 3 in the virtual space corresponding to the positions 1 to 3 in the real space. Next, using these coordinates as information related to the movement of the viewpoint in the virtual space, based on this viewpoint information, a panoramic image (panoramic images 1 to 3 in FIG. 4) is cut out and a panoramic image is generated by encoding processing. To do. Here, the reason for processing a plurality of (virtual) positions is that it is necessary to cut out a predetermined number of frames for encoding processing.

  Thereafter, the distribution server distributes the generated panoramic video to the HMD 2. The HMD 2 (bus) has already received the position 4 at the time when the panoramic video is received, that is, at time t = τ (τ: distribution delay time). Has moved to. As a result, the HMD 2 displays VR video obtained by converting the panoramic video corresponding to the positions 1 to 3 from the position 4 to the position 6 through the position 5 to the user. As a result, there arises a problem that the position state of the HMD 2 (bus) in the real world is different (shifted) from the position state in the VR video (virtual space) displayed at that time.

  On the other hand, in the embodiment shown in FIG. 5, this problem is solved by predicting the future position of the HMD 2 (bus) and distributing a panoramic image corresponding to the prediction result.

  Also in this embodiment shown in FIG. 5, the bus travels on a road in the real world (real space), and reaches position 3 at time t = 0 from position 1 to position 2 in the road. At time t = 0, the HMD 2 transmits state information in which the coordinates of these positions 1 to 3 and the speed and acceleration of the HMD 2 (bus) are associated with time information (t = 0) to the distribution server 1. .

  The state prediction unit 111 (FIG. 1) of the distribution server 1 that has received this state information grasps the position, speed, and acceleration of the HMD 2 (bus) in the real space from the received state information, and based on this, the prediction time determination unit The position, speed, and acceleration in the predicted time interval t = τ to τ + s determined in 112 (FIG. 1) (τ: delivery delay time, s: playback time for one file) are predicted.

Specifically, when the position in the real space at time t = 0 is x (0), the speed is v (0), and the acceleration is a (0), the position x (τ) at time t = τ, Velocity v (τ) and acceleration a (τ) are calculated using the following equation (1) x (τ) = x (0) + ∫v (t) dt
v (τ) = v (0) + ∫a (t) dt
a (τ) = a (0)
Is calculated by Here, ∫ is a definite integral of t = 0 to τ. The position and speed in the predicted time interval t = τ to τ + s can also be calculated by the above equation (1). In this embodiment, position 4, position 5, and position 6 are calculated as positions at t = τ to τ + s.

  Next, the virtual state determination unit 113 (FIG. 1) of the distribution server 1 converts the calculated position and speed in the predicted time interval t = τ to τ + s into a virtual position and virtual speed in the virtual space. Specifically, the distribution server 1 calculates the coordinates of the virtual positions 4 to 6 in the virtual space corresponding to the predicted positions 4 to 6 in the real space. Further, a virtual velocity value in the virtual space at t = τ to τ + s is calculated.

  Next, using the coordinates and virtual velocity values in these virtual spaces as information relating to the movement of the viewpoint in the virtual space, panoramic images (panoramic images 4 to 6 in FIG. 5) are cut out from the VR content based on this viewpoint information. The panorama image is generated by the encoding process. This is because the panoramic image is cut out from the VR content first so that the position in the real world is x (τ) to x (τ + s) and the speed is v (τ) to v (τ + s). A panoramic video is to be generated. Specifically, first, the positions x (τ) to x (τ + s) are divided at equal intervals according to fps (frame per second) in the subsequent encoding process, and the dividing points are set as a plurality of discrete coordinates. Express. Next, these division points are converted into the coordinates of the virtual space in the corresponding VR content, and the 360-degree visual field at the viewpoint position is converted into a panoramic image. Finally, the panoramic image sequence thus obtained is encoded into a panoramic video in real time.

  Thereafter, the distribution server 1 distributes the generated panoramic video to the HMD 2 in accordance with the playback of the corresponding VR video. Here, the HMD 2 (bus) has already moved to the position 4 at the time when the panoramic video is received, that is, at time t = τ. As a result, the HMD 2 displays VR video obtained by converting the panoramic video corresponding to the positions 4 to 6 from the position 4 to the position 6 through the position 5 to the user. That is, the position, speed, and acceleration state of the HMD2 (bus) in the real world coincide with the position, speed, and acceleration state in the VR image (virtual space) displayed at that time, and the VR image was viewed. The user can enjoy a sufficient sense of realism and immersion.

  Incidentally, although it depends on the contents of the VR content, the state information transmitted from the HMD 2 (bus) may be, for example, position information (for example, latitude and longitude) at each measured time. That is, even if there is no speed or acceleration information, it is possible to determine the speed and acceleration based on position information at a plurality of times. Furthermore, if the position at the reference time is known, the position can be predicted using the above equation (1) from the speed information and the acceleration information even if the position information is not included in the state information.

  FIG. 6 is a schematic diagram for explaining another embodiment for performing a distribution process by predicting a client state in the video distribution apparatus according to the present invention.

  In the embodiment shown in FIG. 6, the state prediction unit 111 (FIG. 1) of the distribution server 1 performs state information (for example, an azimuth direction) related to the direction at the prediction time t = τ or the prediction time interval t = τ to τ + s. Or the amount of change in direction with reference to the direction at t = τ). At this time, the direction can be predicted based on route map (road map) information including the location of the HMD 2 (bus). Further, as the state information of the HMD 2 (bus), for example, angular acceleration information at time t = 0 can be acquired, and the traveling direction can be predicted based on this information.

  Specifically, in the embodiment shown in FIG. 6, the distribution server 1 is in the state that the position x (0) at time t = 0 is the position 5 from the HMD 2 (bus) through the positions 3 and 4. Receive information. Here, based on the received state information, the distribution server 1 indicates that “HMD2 (bus) will reach position 6 at time t = τ, and the traveling direction will be left at this position 6” The information is changed to 90 ° (counterclockwise) and reaches position 8 at time t = τ + s.

  Next, the distribution server 1 calculates virtual positions 6 to 8 in the virtual space corresponding to the predicted positions 6 to 8 in the real space, and further (at the time of corresponding VR video playback at these positions 6 to 8). ) The front direction is determined not as the direction of travel when moving from position 4 to position 6 but as the direction of travel when moving toward positions 6 to 8. Further, the viewpoint information related to the movement of the viewpoint in the virtual space is determined from the virtual positions 6 to 8 and the information of the front direction at that time, and the determined traveling direction is reproduced based on the viewpoint information (reproduction in HMD2). (Occasionally) After specifying to display in front, panoramic images (panoramic images 6-8) are cut out from the VR content. Next, the cut panorama image is encoded to generate a panoramic video. Thereafter, the distribution server 1 distributes the generated panoramic video to the HMD 2 in accordance with the reproduction of the corresponding VR video.

  In addition, for example, when the HMD 2 (bus) is traveling on a road on which an intersection (divided into three traveling directions) is present, state information including angular velocity information is transmitted to the distribution server 1 immediately before the intersection. Then. In this case, the distribution server 1 refers to the road map information to be held, and the direction of the angular velocity and / or the magnitude in the received state information indicates which of the three traveling directions the HMD 2 (bus) is directed to (for example, turn right) Whether to turn left or go straight). At this time, for more reliable prediction of this direction, the relationship between the angular velocity ω value and the bending angle is examined in advance, and it is possible to predict the bending angle from the angular velocity ω value acquired as the state information. preferable.

  Further, the distribution server 1 extracts the degree of bending of the road on which the HMD 2 (bus) travels from the road map information that is also held, and predicts a change in the traveling direction of the HMD 2 (bus) at a future time point. it can. As described above, by appropriately predicting the traveling direction of the HMD 2 (bus), it is possible to reliably generate a distribution video that provides a playback image having a front direction that matches the actual traveling direction. It becomes possible.

  FIG. 7 is a schematic diagram for explaining an embodiment of the reproduction control process in the client according to the present invention.

In the embodiment shown in FIG. 7, first, the state information generation unit 213 (FIG. 2) of the HMD 2 is based on the sensor output information at the playback disclosure time (t = τ) at the playback disclosure time (t = τ). Status (eg, position and / or velocity) information is acquired. Next, the playback control unit 212 (FIG. 2) of the HMD2
(A) the received predicted time (t = τ) or predicted time interval (t = τ to τ + s);
(B) the degree of difference (eg, the distance between positions) between its state at the predicted time (t = τ) or the predicted time interval (t = τ to τ + s) detected, measured or determined Based on the speed value difference or ratio), the predicted time (t = τ) or the predicted time interval (t = τ to τ + s) so that the virtual state appearing in the reproduced video matches the state of the HMD2 (bus) The playback speed of the video related to is adjusted.

  Specifically, in FIG. 7, when the distribution server 1 receives the state information related to time t = 0, the HMD 2 (bus) is at position 4 at time t = τ, and then passes through position 5 to time t = Τ + s is predicted to reach position 6, and a panoramic video based on this prediction is generated and distributed. However, although HMD2 (bus) was actually at position 4 at time t = τ, it was still at position 5 at time t = τ + s, which was different from the prediction at distribution server 1. Yes.

The HMD 2 that has received the panoramic images with different predictions is also received from the distribution server 1.
(A1) Information of positions 4 to 6 in the prediction time interval (t = τ to τ + s), or (a2) Information of position 4 and prediction time interval (t = τ to information on velocity v ′ (τ˜τ + s) at τ + s),
Generated based on sensor output information by own sensor,
(B) The information of the position 4 at the playback start time (t = τ) and the information of the speed value v (τ) are compared, and the predicted speeds v ′ (τ˜τ + s) and v ( (τ) is different (under a predetermined judgment condition), the VR state obtained by reproducing the received panoramic image undergoes a transition of the virtual state different from the state transition in the real world. Recognize Therefore, the HMD 2 limits the moving image position range of the VR video reproduced in the time range t = τ to τ + s (which is actually from the position 4 to the position 5) to the positions 4 and 5.

  Specifically, if the average value of velocity values v ′ (τ) received from the distribution server 1 is 30 m / s, while the velocity value v (τ) acquired and generated by itself is 20 m / s, The playback speed of the VR video from time t = τ is set to 0.67 (= 20/30) times the reference, that is, 0.67 times speed. As a result, it is possible to prevent a deviation caused by a prediction error in the reproduced VR video, and to ensure a realistic feeling of the reproduced video.

  Next, the case where HMD2 (bus | bath) advances ahead of the prediction in the delivery server 1 contrary to said Example is considered. In this case, when the distribution server 1 receives the state information related to time t = 0, the HMD 2 (bus) is at position 4 at time t = τ and reaches position 5 at time t = τ + s. Predict and distribute panoramic video based on this prediction.

  However, in reality, it is assumed that the HMD 2 (bus) eventually advances to positions 4 to 6 during t = τ to τ + s. In preparation for such a case, the distribution server 1 supports panoramic images corresponding to the positions 4 and 5 required from the prediction result, and also supports positions that may further progress, for example, positions 6, 7, and 8. Create and distribute panoramic images.

  The HMD 2 that has received such a panoramic video has, for example, an average value of velocity values v ′ (τ) received from the distribution server 1 of 30 m / s, while the velocity value v (τ) acquired and generated by itself. Is 40 m / s, the playback speed of the VR video from time t = τ is 1.3 (= 40/30) times the reference, that is, 1.3 times the speed. As a result, the HMD 2 (bus) can reproduce the VR video generated from the panoramic video corresponding to the positions 4 and 5 and the position 6 while proceeding to the positions 4 to 6.

  Incidentally, in the distribution server 1, as a timing for generating and distributing a panoramic image corresponding to an additional position (possible to reach) (positions 6 to 8), the reproduction of the VR image is all (for one file). When the video position reaches x% of the playback time s, the actual progress degree (y%) in the predicted progress section (in this case, the section from position 4 to position 5) of HMD2 (bus) exceeds x% ( y> x).

  As described above with reference to FIG. 7, the video distribution system including the HMD 2 of the present embodiment controls the playback speed of the VR video to prevent the occurrence of deviation due to the prediction error in the played VR video. can do. As a result, it is possible to secure a realistic feeling of the reproduced video and provide a more preferable VR world experience to the user.

  As described above in detail, according to the video distribution apparatus, system, program, and method of the present invention, the virtual state in the virtual space that matches the playback time in the future client is predicted from the acquired client state. Based on this, a distribution video is generated. Since such a characteristic state prediction function is provided, video data that enables playback of video in accordance with the client state can be distributed to the client. This also improves the realism of the virtual experience for the user who sees the reproduced video, and allows the user to enjoy a sufficient immersive feeling.

  In particular, in a situation where the user views a video while moving, it is possible to perform distribution by virtually removing delays that cannot be zero in principle, such as communication path delays and encoding delays. Thus, for example, the present invention is to realize a user content-linked VR content distribution service even in the coming 5G (fifth generation mobile communication system) in which a large amount of VR-related data communication is predicted. It is intended to provide important technology.

  Various changes, modifications, and omissions of the various embodiments of the present invention described above within the scope of the technical idea and the viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be any limitation. The present invention is limited only by the claims and the equivalents thereof.

1. Distribution server (video distribution device)
DESCRIPTION OF SYMBOLS 101 Communication interface part 102 VR content storage part 103 Panoramic image storage part 104 Panoramic video storage part 111 State prediction part 111a Position prediction part 111b Speed / acceleration prediction part 111c Direction prediction part 112 Prediction time determination part 113 Virtual state determination part 114 Video generation Unit 114a panorama image generation unit 114b panorama image generation unit 115 video distribution control unit 116 communication control unit 2 HMD (client)
DESCRIPTION OF SYMBOLS 201 Communication interface part 202 Positioning part 203 Acceleration sensor 204 Gyro sensor 205 Geomagnetic sensor 206 Display 211 Communication control part 212 Reproduction | regeneration control part 213 Status information generation part 3 GPS satellite

Claims (9)

A video distribution device that distributes video in a virtual space to a client,
State prediction means for acquiring information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Virtual state determination means for determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
A video distribution apparatus comprising: a video distribution control unit that distributes the generated video to the client in accordance with reproduction at the other time point or time interval. The video is a viewpoint video in the virtual space,
The state prediction means acquires state information relating to at least one of position, velocity, acceleration, and direction at one time point or time interval as the state of the client, and at another time point or time interval. Predicting state information relating to at least one of position, velocity, acceleration and orientation;
The virtual state determination means determines virtual state information related to at least one of position, velocity, acceleration, and direction in the virtual space corresponding to the predicted state information at the other time point or time interval. And
The video generation means determines viewpoint information related to at least one of the position, velocity, acceleration, and orientation of the viewpoint in the virtual space based on the determined virtual state information, and based on the determined viewpoint information The video distribution apparatus according to claim 1, wherein a viewpoint video is generated.   The video distribution according to claim 2, wherein the state prediction unit predicts state information related to a direction at the other time point or time interval based on route map information including a location of the client. apparatus.   The state prediction means obtains angular acceleration information that is information related to the direction at the one time point or time interval, and predicts state information related to the direction at the other time point or time interval. The video distribution apparatus according to claim 2 or 3.   The video delivery according to any one of claims 1 to 4, further comprising a predicted time determination unit that determines the other time point or time interval based on the acquired information related to the delivery delay time. apparatus. A video distribution system having a client and a video distribution device that distributes video in a virtual space to the client,
State information generating means for generating information relating to its own state detected, measured or determined in one time point or time interval;
Client communication control means for transmitting the information relating to the generated state to the video distribution apparatus in association with the one time point or time interval, and the video distribution apparatus,
State prediction means for acquiring information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Virtual state determination means for determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
A video distribution system comprising: a video distribution control unit that distributes the generated video to the client in accordance with reproduction at the other time point or time interval. The video distribution device further includes device communication control means for transmitting the predicted state at the other time point or time interval to the client,
The client is played based on the degree of difference between the received state at that other time or time interval and its state at that other time or time interval detected, measured or determined 7. The video distribution according to claim 6, further comprising reproduction control means for adjusting a reproduction speed of the video relating to the other time point or time interval so that a virtual state appearing in the video matches the state of the client. system. A video distribution program for causing a computer mounted on a video distribution device to distribute video in a virtual space to a client,
State prediction means for acquiring information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Virtual state determination means for determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
A video distribution program for causing a computer to function as video distribution control means for distributing a generated video to the client in accordance with reproduction at the other time point or time interval. A video distribution method in a computer installed in a video distribution device that distributes video in a virtual space to a client,
Obtaining information related to the state of the client at one time point or time interval, and predicting the state of the client at another time point or time interval;
Determining a virtual state in the virtual space corresponding to the predicted state at the other time point or time interval;
Video generation means for generating video to be distributed based on the determined virtual state;
And a step of distributing the generated video to the client in accordance with reproduction at the other time point or time interval.

Images