JP2019068130A - Video distribution server, video distribution method and video distribution program and video reproducer - Google Patents

Abstract

To provide a video distribution server, a video distribution method, a video distribution program and a video reproducer.SOLUTION: A multi-viewpoint video storage section 101 stores multi-viewpoint videos acquired from multiple cameras of different viewpoints. An individual viewpoint stream generating section 102 generates an individual viewpoint stream on the basis of the multi-viewpoint videos. An all viewpoint stream generating section 103 generates an all viewpoint stream by combining the multi-viewpoint videos. A scheduling section 107 determines a schedule for multicasting respective video streams on the basis of a viewing request acquired from each client by a viewing request receiving section 105. A schedule notification section 110 notifies the client of the schedule thus determined. A multicast distribution section 108 multicasts each video stream on the basis of the schedule thus determined.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a video distribution server, a video distribution method, a video distribution program, and a video reproduction apparatus, and in particular, a video distribution server that generates video streams of individual viewpoints and all viewpoints based on multi-view videos and multicasts to clients. , A video distribution method, a video distribution program, and a video reproduction apparatus.

  In recent years, with the increase in network capacity and the spread of smartphones, many users have come to use video services on smartphones. Also, in video media, viewing modes such as multi-view video and free-view video have been proposed as modes capable of transmitting a higher sense of reality. Here, the multi-viewpoint image is an image that the user can view while viewing images taken by a plurality of cameras arbitrarily, and the free-viewpoint image is a virtual image of an image from a viewpoint where no camera existed. Is a video that can be combined and viewed.

  On the other hand, as the delivery method, as described in Non-Patent Document 2, there are a server rendering method and a client rendering method. In the server rendering method, viewpoint operation information of the user is transmitted to the server, the corresponding viewpoint video is synthesized, rendered and encoded, and the video is distributed to the client. Therefore, although the amount of received data on the client side can be reduced, a critical problem of response delay to operation occurs.

  In the client rendering method, the video of all viewpoints encoded by the server is transmitted to the client, and the client side composes and renders the viewpoint video. The client rendering method has the drawback that the amount of received data on the client side increases, but the response delay to the user's viewpoint operation is reduced.

  With respect to the technical problem that the amount of received data on the client side increases, Patent Document 1 distributes video streams for all viewpoints while lowering the bit rate for videos other than the viewpoints currently specified by the user. , Has proposed a method to reduce the overall bit rate. Specifically, the probability distribution is updated so that the bit rate of the viewpoint currently selected by the user becomes maximum, and the bit rate that can be assigned to each viewpoint is determined.

  On the other hand, in the case of watching sports at a stadium, etc., a free viewpoint video is created in real time, and in the case where a large number of users specify and watch at any time and any viewpoint, for example, It is possible to provide a service such as watching the goal scene of the scene from an easy-to-see viewpoint position by chasing afterward.

  In such a scene, transmission data to a large number of users is generated. For this reason, for example, when all users individually request video streams in unicast, there arises a technical problem that the amount of transmission data increases in proportion to the number of users.

  However, in the client rendering method, video streams of all viewpoints are distributed, and this is common to all users. For example, Non-Patent Document 1 proposes a method of distributing video of all viewpoints by multicast.

  Non-Patent Document 2 proposes a method of reducing the total amount of data by combining multicast and unicast appropriately for a single video. Specifically, on the condition that all users start viewing the video from the same playback position, at the time of requesting the video stream, if a multicast issued within a certain time period precedes and is received, A technique has been proposed in which a part already distributed is separately requested by unicast, and if it does not exist, a new multicast is issued.

JP, 2013-183209, A

"High Scalability Free View Telepresence by Multi-view Omnidirectional Video Multicasting", Tomoya Ishikawa, Kazumasa Yamazawa, Naokazu Yokoya: 3D Image Conference 2007, pp. 63-66, July 2007. "Optimal Merging Method of Multicast Streaming", Hiroshi Toyoizumi, Hiroyuki Tanaka: Shingaku Technical Report 2004.

  The above-mentioned prior art does not consider multicast scheduling in the case where a multi-view video stream is viewed from different playback positions for each user. For example, if all data are to be delivered by multicast, it is necessary to always deliver by multicast at all times and all times, and efficiency can not be achieved. Therefore, it is necessary to multicast reproduction positions and viewpoints required from as many clients as possible.

  An object of the present invention is to solve the above technical problems, and to provide a video distribution server, method, program, and video reproduction apparatus capable of efficiently distributing each video stream of individual viewpoints and all viewpoints to clients based on multi-view videos. It is to provide.

  In order to achieve the above object, the present invention is characterized in that the video distribution server, method and program for distributing a video stream to a client have the following configuration.

  (1) Generate individual viewpoint streams based on multi-view videos, combine multi-view videos to generate all-view streams, and determine a schedule for multicasting each video stream based on a viewing request acquired from each client And notifying the client of the determined schedule, and multicasting each video stream based on the determined schedule.

  (2) In order to determine a schedule for multicasting all viewpoint streams, the parameter related to designation of playback position is extracted from the viewing request of all viewpoint streams, and the number of clients for each playback position is based on the playback position extracted from the viewing request. It is assumed that a schedule for multicasting all viewpoint streams corresponding to playback positions where the distribution exceeds a predetermined threshold is determined.

  (3) In order to determine the schedule for multicasting the individual viewpoint stream, the reproduction position and the viewpoint distribution are extracted from the viewing request of the individual viewpoint stream, and based on the reproduction position and the viewpoint distribution extracted from the viewing request A two-dimensional distribution of the number of clients is estimated, and a schedule for multicasting individual viewpoint streams corresponding to reproduction positions and viewpoints in which the two-dimensional distribution exceeds a predetermined threshold is determined.

  According to the present invention, since the reproduction position and the viewpoint of the video stream requested by each client are statistically processed, and the multicast of the individual viewpoint stream and the all viewpoint stream is scheduled so as to be able to meet more viewing requests. It is possible to reduce the total amount of transmission data while satisfying the client's viewing request.

It is the figure which showed an example of the application environment of this invention. It is a functional block diagram showing composition of a picture distribution system which realizes distribution of a multi-view picture. It is a figure showing the delivery method of the picture stream to a client. It is a figure showing an example of one-dimensional probability density distribution for deciding a schedule which multicasts all viewpoints streams. It is a figure showing an example of two-dimensional probability density distribution for deciding a schedule which multicasts an individual viewpoint stream. It is the figure which showed an example of the multicast management table. It is the flowchart which showed the delivery procedure of all the viewpoint streams. It is the flowchart which showed the delivery procedure of the individual viewpoint stream. It is a figure showing the reproduction method of a free viewpoint picture.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present invention, as shown in FIG. 1, after capturing and accumulating multi-view images of field competition (for example, soccer) using a plurality of cameras Ca, a large number of users U can be taken at any timing such as half time. The video stream is delivered by multicast or unicast in response to a viewing request including designation of a playback position from the.

  FIG. 2 is a functional block diagram showing the configuration of a video delivery system that realizes delivery of multi-view video. The video delivery system according to the present invention comprises a video delivery server 1 for delivering a video stream, and a plurality of clients 2 for requesting the delivery of the video stream to the video delivery server 1 and receiving and playing back a video stream to be multicast or unicast. And the network 3 connected.

  In the video delivery server 1, the multi-view video storage unit 101 stores multi-view camera videos acquired from a plurality of cameras Ca having different viewpoints. The individual viewpoint stream generation unit 102 generates an individual viewpoint stream Vi (i is a camera identifier) for each stored camera image. The all viewpoint stream generation unit 103 generates the all viewpoint stream Vs by compressing and combining the stored camera images to a resolution lower than the resolution of the individual viewpoint stream Vi.

  The free viewpoint data generation unit 104 adds to each of the video streams Vs and Vi in order to combine the free viewpoint video based on the all viewpoint streams Vs, and A free viewpoint data set including original position, size information) and (2) camera calibration information of each viewpoint is generated.

  The viewing request receiving unit 105 receives the distribution request of the video stream received from the client 2 and registers the distribution request in the client management table 105. In this embodiment, when the user performs a seek operation or the like on the display of the client, for example, to specify the playback position and the viewpoint of the video stream, viewing requests for all viewpoint streams Vs and viewing requests for individual viewpoint streams Vi are triggered by this. Is sent to the video delivery server 1.

  In the view request receiving unit 105, the reproduction position specification extraction unit 1051 extracts the specification of the reproduction position x1 from the view request. The viewpoint specification extraction unit 1052 extracts the viewpoint distribution d of the client from the viewing request. Here, the viewpoint distribution is a probability distribution of transition from the current viewpoint to another viewpoint in each client as disclosed in Patent Document 1.

  The client management table 106 manages the number of clients and the playback position x for all viewpoint streams. With regard to the individual viewpoint stream, the number of clients, reproduction position x and viewpoint distribution d are managed.

  The scheduling unit 107 determines a delivery schedule of each video stream based on viewing requests from a plurality of clients. In this embodiment, as shown in FIG. 3, for each client 2, one individual viewpoint stream selected from among the all viewpoint stream Vs and the plurality of individual viewpoint streams Vi is based on the result of the scheduling. Delivered by multicast or unicast.

  In the scheduling unit 107, as shown in FIG. 4, the first probability density distribution calculating unit 1071 calculates each reproduction position at the current time t1 based on the reproduction positions x extracted from a plurality of viewing requests for all viewpoint streams. Estimate the one-dimensional probability density distribution of the number of clients (frequency) during playback. Then, based on the designated reproduction time x1, a ratio R1 of clients in reproduction of [x1-δ, x1] within a predetermined time range is calculated.

  As shown in FIG. 5, the second probability density distribution calculating unit 1072 sets each playback position at each current playback time t2 based on the playback position x and the viewpoint distribution d extracted from the plurality of viewing requests for the individual viewpoint stream. Calculate the distribution of the two-dimensional probability density of the number of clients (frequency) during playback. Then, with respect to the designated reproduction position x2, a ratio R2 occupied by the client during reproduction within a predetermined time range [x1−δ, x1] is calculated for each viewpoint p.

  The scheduling unit 107 further calculates the corresponding number of clients by multiplying each of the ratios R1 and R2 by the total number of clients. Then, the issue / stop of the multicast is determined based on whether the calculation result of the number of clients is equal to or more than the reference value. That is, if the calculation result of the number of clients is equal to or more than the reference value, the multicast is issued, and if the calculation result of the number of clients is less than the reference value, the issued multicast is stopped.

  When competing with already issued multicasts (existing multicasts cover the same client or view), it is based on whether the average number of clients covered by multiple multicasts (unique number) is equal to or greater than the reference value. And decide to issue / stop multicast. When multicast is issued, a multicast management table to be described later may be updated in order to guide each client 2 to switch to a multicast closer in time.

  The multicast distribution unit 108 multicasts the individual viewpoint stream and the all viewpoint stream according to the result of the scheduling. When using LTE as a mobile network, multicasting can be performed by eMBMS. The unicast distribution unit 109 unicasts the individual viewpoint stream and the all viewpoint stream according to the request of each client.

  The schedule notification unit 110 gives the opportunity of participation in multicast and unicast by notifying the client 2 that has made a viewing request of the result of the distribution schedule. In the multicast management table 111, the information on the multicast issued above is registered or updated.

  As shown in FIG. 6, the multicast management table 111 manages at least a multicast ID, a client ID, a type, a viewpoint ID, a start time, a reproduction time, and a bit rate as information on each multicast. The schedule notification unit 110 can notify a multicast distribution schedule by distributing the contents of the multicast management table 111 to each client 2.

  In the client 2, the viewing request unit 201 transmits a viewing request for a video stream including the playback position x 1 and the viewpoint distribution d to the video distribution server 1 when the user designates the playback position and the viewpoint by a seek operation or the like.

  The viewing determination unit 202 collates the result of the distribution schedule notified from the video distribution server 1 with the contents of the viewing request, and determines whether it is necessary to receive the all-view stream and the individual-view stream to be multicast by the video distribution server 1 Do. When judging that the all-view stream and / or the individual-view stream to be multicast are not received, the unicast request unit 203 again requests the video distribution server 1 to unicast the requested video stream.

  FIGS. 7 and 8 are flowcharts showing a procedure in which the video delivery server 1 delivers a video stream in response to a request for viewing a video stream transmitted by the client 2. In this embodiment, when the user performs a seek operation or the like on the display of the client 2 and designates the playback position x1 and the viewpoint p, for example, viewing requests for all viewpoint streams and viewing requests for individual viewpoint streams It is sent to the distribution server 1.

  FIG. 7 is a flow chart showing the distribution procedure of all viewpoint streams. When the user designates a desired reproduction position x1 and viewpoint p by seek operation etc. at time tc1, the reproduction position x1 and current time are specified at time tc2. A viewing request of all viewpoint streams including t1 is transmitted from the viewing request unit 201 to the video distribution server 1.

  The video delivery server 1 monitors the viewing condition of the client at time ts1, and when the viewing request for all the viewpoint streams is received at time ts2, the viewing request receiving unit 105 receives the viewing request at time ts3. The reproduction position x1 is extracted from the view request and registered in the client management table 106.

  At time ts4, the one-dimensional probability density distribution described with reference to FIG. 4 is estimated based on the reproduction positions x extracted from the plurality of viewing requests received within a predetermined period from the current time t1.

  At time ts5, based on the probability density distribution, whether or not the number of clients playing back within a predetermined time range [x1-δ, x1] at the current time t1 based on the playback position x1 exceeds a predetermined threshold It is judged. If the threshold is exceeded, multicast of all viewpoint streams including the reproduction position x1 is newly issued at time ts6. At time ts7, the multicast management table is updated. At time ts8, the multicast management table is transmitted to the client 2.

  When each client 2 acquires the multicast management table at time tc3, it is determined at time tc4 whether or not the multicast to be joined is registered. In the case of an all-view stream, if the difference Δx1 between the reproduction position of the multicast and the reproduction position x1 of the client is within a predetermined time, it is determined that it should participate. At time tc5, the multicast is received.

  On the other hand, if the multicast to be joined is not registered, at time tc6, a unicast transmission request including the reproduction position x1 is transmitted to the video distribution server 1, and all viewpoint streams are received by unicast. At time tc7, all viewpoint streams received by multicast or unicast are buffered. At time tc8, the free viewpoint video is synthesized and reproduced based on the designated viewpoint P, as will be described in detail later with reference to FIG.

  FIG. 8 is a flow chart showing the delivery procedure of the individual viewpoint stream. When the user designates a desired playback position x1 and viewpoint p by seek operation etc. at time tc1, the playback position x2 at the time tc2, the current time A viewing request of an individual viewpoint stream including t2 and viewpoint distribution d is transmitted from the viewing request unit 201 to the video distribution server 1.

  The video delivery server 1 monitors the viewing condition of the client at time ts1. When the viewing request for the individual viewpoint stream is received at time ts2, the viewing request receiving unit 105 receives the viewing request at time ts3. The reproduction position x2 and the viewpoint distribution d are extracted from the view request and registered in the client management table 106.

  At time ts4, the distribution of the two-dimensional probability density described with reference to FIG. 5 is estimated based on the reproduction position x and the viewpoint distribution d extracted from the plurality of viewing requests received within a predetermined period from time t2. Ru.

  At time ts5, based on the probability density distribution, the number of clients reproducing within the predetermined time range [x1-δ, x1] at time t2 with reference to the reproduction position x2 and the viewpoint distribution d exceeds the predetermined threshold value It is determined whether the If the threshold is exceeded, at time ts6, a multicast of an individual viewpoint stream corresponding to the viewpoint distribution d including the playback position x2 is newly issued. At time ts7, the multicast management table is updated. At time ts8, the multicast management table is transmitted to the client 2.

  When each client 2 acquires the multicast management table at time tc3, it is determined at time tc4 whether or not the multicast to be joined is registered. In the case of the individual viewpoint stream, if the difference Δx1 between the multicast reproduction position and the reproduction position x1 of the client is within a predetermined time, it is determined that the participation should be made.

  In the case of the individual viewpoint stream, the viewpoint distribution after Δx2 is further calculated, and when the selection probability of the viewpoint in the multicast is high, it is determined that it should participate. In this embodiment, assuming that the viewpoint position is a continuous value and the series values I1, I2, I3... Are a stochastic process (Wiener process), the viewpoint position I (Δx2) after Δx2 can be expressed by the following equation.

  I (Δx2) = I0 + μt ± σ√t (1)

  Here, since I0 is the current viewpoint position, μ is a trend term related to the viewpoint position change, and σ is the magnitude of fluctuation, the viewpoint distribution is a distribution of standard deviation σ centered on I (Δx2). At time tc5, the multicast is received.

  On the other hand, if the multicast to be joined is not registered, at time tc6, the video distribution server 1 is requested to distribute in unicast. At time tc7, the individual viewpoint stream distributed by multicast or unicast is buffered. At time tc8, the free viewpoint video is synthesized and reproduced based on the designated viewpoint p.

  On the other hand, if the multicast to be joined is not registered, a unicast transmission request including the reproduction position x1 and the view distribution d is transmitted to the video distribution server 1 at time tc6.

  When multicast is distributed at a plurality of bit rates, the throughput to be assigned to each view is calculated as described in Document 1, and even if the bit rate of multicast is too high (too low), unicast reception is performed. May be The client requests the server to update the multicast management table.

  In addition, when multicast is distributed at multiple bit rates, calculate the throughput to be assigned to each view as described in Document 1, and receive in unicast even if the bit rate of multicast is too high (too low). May be The client requests the server to update the multicast management table.

  When playing back a free viewpoint video, as shown in FIG. 9, the client 2 further adds a bitstream of object information (mask image, three-dimensional position, size) to be viewed by the free viewpoint along with the multi-view stream. To receive. The client 2 decodes these streams to obtain multi-view decoded images and object information (decoded mask image, three-dimensional position, size), and extracts an object image from the decoded image and mask image.

  The client 2 further arranges this on the three-dimensional model based on the three-dimensional position information and the size, moves the virtual viewpoint according to the viewpoint p designated by the user, and performs projective transformation to a two-dimensional image. Display an image.

  DESCRIPTION OF SYMBOLS 1 ... Video delivery server, 2 ... Client, 3 ... Network, 101 ... Multi-view video storage unit, 102 ... Individual viewpoint stream generation unit, 103 ... All viewpoint stream generation unit, 104 ... Free viewpoint data generation unit, 105 ... Viewing request Reception unit 106: Client management table 107: Scheduling unit 108: Multicast distribution unit 109: Unicast distribution unit 110: Schedule notification unit 111: Multicast management table

Claims (11)

In the video distribution server that distributes video streams to clients,
A unit for generating an individual viewpoint stream based on multi-view video;
A means for combining multi-view videos to generate an all-view stream;
A means for determining a schedule for multicasting each video stream based on a request from each client;
A means for notifying the client of the determined schedule;
And a means for multicasting each video stream based on the determined schedule. The means for determining the schedule is
A means for extracting a parameter relating to designation of a playback position from a viewing request of all viewpoint streams;
Means for estimating the distribution of the number of clients for each reproduction position based on the reproduction position extracted from the viewing request,
The video delivery server according to claim 1, wherein a schedule for multicasting all viewpoint streams corresponding to playback positions whose distribution exceeds a predetermined threshold is determined. The means for determining the schedule is
Means for extracting playback position and viewpoint distribution from viewing request of individual viewpoint stream;
Means for estimating a two-dimensional distribution of the playback position and the number of clients for each viewpoint based on the playback position and viewpoint distribution extracted from the viewing request,
The video distribution server according to claim 1 or 2, wherein a schedule for multicasting individual viewpoint streams corresponding to playback positions and viewpoints in which the two-dimensional distribution exceeds a predetermined threshold is determined. In a video delivery method in which a video delivery server delivers a video stream to a client,
The client
A procedure for transmitting a request for viewing a video stream;
The video distribution server
A procedure for generating an individual viewpoint stream and an all viewpoint stream based on multi-view video;
A procedure for determining a schedule for multicasting each video stream;
A procedure for notifying the client of the determined schedule;
A procedure for multicasting each video stream based on the determined schedule;
Furthermore, the client
And d) receiving each video stream based on the notified schedule. The procedure for determining the schedule is
A procedure of extracting a parameter relating to designation of a playback position from a viewing request of all viewpoint streams;
Estimating the distribution of the number of clients for each playback position based on the playback position extracted from the viewing request,
5. The video delivery method according to claim 4, wherein a schedule for multicasting all viewpoint streams corresponding to playback positions where the distribution exceeds a predetermined threshold is determined. The procedure for determining the communication schedule is
A procedure of extracting a reproduction position and a viewpoint distribution from a viewing request of the individual viewpoint stream;
Estimating a two-dimensional distribution of the playback position and the number of clients for each viewpoint based on the playback position and the viewpoint distribution extracted from the viewing request;
The video distribution method according to claim 4 or 5, wherein a schedule for multicasting individual viewpoint streams corresponding to playback positions and viewpoints in which the two-dimensional distribution exceeds a predetermined threshold is determined. In a video delivery program for delivering a video stream from a video delivery server to a client,
A procedure for generating an individual viewpoint stream based on multi-view video;
A procedure for combining multi-view videos to generate an all-view stream;
A procedure for determining a schedule for multicasting each video stream based on a viewing request acquired from each client;
A procedure for notifying the client of the determined schedule;
And a procedure of multicasting each video stream based on the determined schedule. The procedure for determining the delivery schedule is
A procedure of extracting a parameter relating to designation of a playback position from a viewing request of all viewpoint streams;
Estimating the distribution of the number of clients for each playback position based on the playback position extracted from the viewing request,
The video delivery program according to claim 7, wherein a schedule for multicasting all viewpoint streams corresponding to playback positions where the distribution exceeds a predetermined threshold is determined. The procedure for determining the schedule is
A procedure of extracting a reproduction position and a viewpoint distribution from a viewing request of the individual viewpoint stream;
Estimating a two-dimensional distribution of the playback position and the number of clients for each viewpoint based on the playback position and the viewpoint distribution extracted from the viewing request;
9. The video delivery program according to claim 7, wherein a schedule for multicasting individual viewpoint streams corresponding to playback positions and viewpoints in which the two-dimensional distribution exceeds a predetermined threshold is determined. In a video reproduction apparatus for reproducing a video stream distributed by a video distribution server,
Means for transmitting a viewing request of the video stream to the video distribution server;
A means for receiving a schedule for multicasting each video stream determined and notified by the video distribution server based on the viewing request;
And a means for the video distribution server to receive a video stream to be multicast according to the schedule based on the received schedule. The viewing request includes parameters relating to designation of a playback position,
11. The video reproduction apparatus according to claim 10, wherein the schedule is determined based on a parameter related to designation of the reproduction position.