JP6770939B2 - Video distribution server, video distribution method and video distribution program - Google Patents

Description

The present invention relates to a video distribution server, a video distribution method, and a video distribution program . In particular, a video distribution server and a video distribution method that generate video streams of individual viewpoints and all viewpoints based on multi-viewpoint video and multicast them to a client. And about video distribution programs.

In recent years, due to the increase in network capacity and the spread of smartphones, many users have come to use video services on smartphones. Also, in video media, viewing forms such as multi-viewpoint video and free-viewpoint video have been proposed as a form capable of transmitting a higher sense of presence. Here, the multi-viewpoint image is an image that can be viewed by the user while arbitrarily switching the images taken by a plurality of cameras, and the free-viewpoint image is a virtual image from a viewpoint in which the cameras did not exist. It is a video that can be combined and viewed.

On the other hand, as described in Non-Patent Document 2, there are a server rendering method and a client rendering method as the distribution method. In the server rendering method, the user's viewpoint operation information is transmitted to the server, the corresponding viewpoint video is synthesized, rendered and encoded, and the video is delivered to the client. Therefore, although the amount of received data on the client side can be suppressed, a critical problem of delay in response to the operation occurs.

In the client rendering method, the video of all viewpoints encoded by the server is transmitted to the client, and the viewpoint video is synthesized and rendered on the client side. The client rendering method has a drawback that the amount of received data on the client side increases, but the response delay to the user's viewpoint operation becomes small.

In response to the technical problem that the amount of received data on the client side increases, Patent Document 1 describes by lowering the bit rate of video other than the viewpoint currently specified by the user while delivering the video stream for all viewpoints. , Proposes a method to reduce the overall bit rate. Specifically, the probability distribution is updated so that the bit rate of the viewpoint selected by the user is maximized, and the bit rate that can be assigned to each viewpoint is determined.

On the other hand, in the case of watching sports at a stadium, where a free-viewpoint video is created in real time and a large number of users specify an arbitrary time and an arbitrary viewpoint to watch, for example, soccer that cannot be seen from their own seat. It is possible to provide a service such as watching the goal scene of the game from a viewpoint position that is easy to see afterwards.

In such a scene, transmission data to a large number of users is generated. Therefore, for example, if all users individually request a video stream by unicast, there arises a technical problem that the amount of transmitted data increases in proportion to the number of users.

However, in the client rendering method, a video stream of all viewpoints is distributed, which is common to all users. Therefore, for example, Non-Patent Document 1 proposes a method of distributing a video of all viewpoints by multicast.

Non-Patent Document 2 proposes a method for reducing the total amount of data by appropriately combining multicast and unicast, although it is for a single video. Specifically, under the condition that all users start viewing the video from the same playback position, if a multicast issued within a certain time exists in advance at the time of requesting the video stream, it is received. A technology has been proposed in which the already delivered portion is individually requested by unicast, and if it does not exist, a new multicast is issued.

Japanese Unexamined Patent Publication No. 2013-183209

"High-scalability free-viewpoint telepresence by multicasting multi-viewpoint omnidirectional video", Tomoya Ishikawa, Kazumasa Yamazawa, Naokazu Yokoya: 3D Image Conference 2007, pp. 63-66, July 2007. "Optimal Marging Method for Multicast Streaming", Hiroshi Toyoizumi, Hiroyuki Tanaka: Shingaku Giho 2004.

In the above-mentioned prior art, the multicast scheduling when viewing a multi-view video stream from a different playback position for each user is not considered. For example, if all data is to be distributed by multicast, it is necessary to always perform multicast distribution from all viewpoints and all times, which cannot improve efficiency. Therefore, it is necessary to multicast the playback position / viewpoint required by as many clients as possible.

An object of the present invention is to solve the above technical problems and to provide a video distribution server, a video distribution method, and a video distribution program capable of efficiently distributing each video stream of individual viewpoints and all viewpoints to a client based on a multi-viewpoint video. To provide.

In order to achieve the above object, the present invention is characterized in that the video distribution server, method and program for distributing the video stream to the client have the following configurations.

(1) Generate an individual viewpoint stream based on the multi-viewpoint video, combine the multi-viewpoint video to generate an all-viewpoint stream, and determine a schedule for multicasting each video stream based on the viewing request obtained from each client. , The determined schedule is notified to the client, and each video stream is multicast based on the determined schedule.

(2) In order to determine the schedule for multicasting the all-viewpoint stream, the parameters related to the specification of the playback position are extracted from the viewing request of the all-viewpoint stream, and the number of clients for each playback position is based on the playback position extracted from the viewing request. The distribution of the above is estimated, and the schedule for multicasting all viewpoint streams corresponding to the reproduction positions where the distribution exceeds a predetermined threshold value is determined.

(3) In order to determine the schedule for multicasting the individual viewpoint stream, the playback position and viewpoint distribution are extracted from the viewing request of the individual viewpoint stream, and the playback position and viewpoint are based on the playback position and viewpoint distribution extracted from the viewing request. The two-dimensional distribution of the number of clients for each is estimated, and the schedule for multicasting the individual viewpoint streams corresponding to the playback positions and viewpoints whose two-dimensional distribution exceeds a predetermined threshold is determined.

According to the present invention, the playback position and viewpoint of the video stream requested by each client are statistically processed, and the multicast of the individual viewpoint stream and the all viewpoint stream is scheduled so as to respond to more viewing requests. It becomes possible to reduce the total amount of transmitted data while satisfying the viewing request of the client.

It is a figure which showed an example of the application environment of this invention. It is a functional block diagram which showed the structure of the video distribution system which realizes the distribution of a multi-viewpoint video. It is a figure which showed the distribution method of the video stream to a client. It is a figure which showed an example of the one-dimensional probability density distribution for determining the schedule for multicasting an all-viewpoint stream. It is a figure which showed an example of the two-dimensional probability density distribution for determining the schedule for multicasting the individual viewpoint stream. It is a figure which showed an example of the multicast management table. It is a flowchart which showed the distribution procedure of the whole viewpoint stream. It is a flowchart which showed the distribution procedure of the individual viewpoint stream. It is a figure which showed the reproduction method of a free viewpoint image.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present invention, as shown in FIG. 1, after shooting and accumulating a multi-viewpoint image of a field competition (for example, soccer) using a plurality of cameras Ca, a large number of users U can be used at any timing such as half-time. The video stream is distributed by multicast or unicast according to the viewing request including the specification of the playback position from.

FIG. 2 is a functional block diagram showing a configuration of a video distribution system that realizes distribution of multi-viewpoint video. The video distribution system of the present invention includes a video distribution server 1 that distributes a video stream, and a plurality of clients 2 that request the video distribution server 1 to distribute the video stream and receive and reproduce the multicast or unicast video stream. And are connected by the network 3.

In the video distribution server 1, the multi-viewpoint video storage unit 101 stores multi-viewpoint camera images acquired from a plurality of cameras Ca having different viewpoints. The individual viewpoint stream generation unit 102 generates an individual viewpoint stream Vi (i is a camera identifier) for each stored camera image. The all-viewpoint stream generation unit 103 generates all-viewpoint streams Vs by compressing and combining the stored camera images to a resolution lower than the resolution of the individual viewpoint stream Vi.

In order to synthesize the free viewpoint video based on the all viewpoint streams Vs, the free viewpoint data generation unit 104, in addition to the above video streams Vs and Vi, (1) object information (mask image, tertiary) existing in each viewpoint. (Original position, size information) and (2) Generate a free viewpoint data set containing camera calibration information for each viewpoint.

The viewing request receiving unit 105 receives the distribution request of the video stream received from the client 2 and registers it in the client management table 105. In the present embodiment, when the user specifies the playback position and viewpoint of the video stream on the display of the client, for example, by performing a seek operation or the like, the viewing request of the all-viewpoint stream Vs and the viewing request of the individual viewpoint stream Vi are triggered by this. Is transmitted to the video distribution server 1.

In the viewing request receiving unit 105, the reproduction position designation extraction unit 1051 extracts the designation of the reproduction position x1 from the viewing request. The viewpoint designation extraction unit 1052 extracts the viewpoint distribution d of the client from the viewing request. Here, the viewpoint distribution is a probability distribution in which each client transitions from the current viewpoint to another viewpoint, as disclosed in Patent Document 1.

The client management table 106 manages the number of clients and the playback position x for all viewpoint streams. For the individual viewpoint stream, the number of clients, playback position x, and viewpoint distribution d are managed.

The scheduling unit 107 determines the distribution schedule of each video stream based on viewing requests from a plurality of clients. In the present embodiment, as shown in FIG. 3, for each client 2, one individual viewpoint stream selected from the all viewpoint streams Vs and the plurality of individual viewpoint streams Vi is based on the result of the scheduling. Delivered by multicast or unicast.

In the scheduling unit 107, the first probability density distribution calculation unit 1071 sets each playback position at the current time t1 based on the playback positions x extracted from a plurality of viewing requests for all viewpoint streams, as shown in FIG. Estimate the one-dimensional probability density distribution of the number of clients (frequency) during playback. Then, the ratio R1 of the clients playing [x1-δ, x1] within the predetermined time range is calculated with reference to the designated playback time x1.

As shown in FIG. 5, the second probability density distribution calculation unit 1072 sets each playback position at the current time t2 at each viewpoint based on the playback position x and the viewpoint distribution d extracted from a plurality of viewing requests for the individual viewpoint stream. Calculate the distribution of the two-dimensional probability density of the number of clients (frequency) being played in. Then, the ratio R2 occupied by the clients playing [x1-δ, x1] within the predetermined time range is calculated for each viewpoint p based on the specified playback position x2.

The scheduling unit 107 further calculates the corresponding number of clients by multiplying the respective ratios R1 and R2 by the total number of clients. Then, the issue / stop of multicast is determined based on whether or not the calculation result of the number of clients is equal to or greater than the reference value. That is, if the calculation result of the number of clients is equal to or more than the reference value, multicast is issued, and if the calculation result of the number of clients is less than the reference value, the issued multicast is stopped.

If there is a conflict with an already issued multicast (an existing multicast covers the same client or viewpoint), it is based on whether the average number of clients (unique number) covered by multiple multicasts is equal to or greater than the reference value. To decide whether to issue / stop multicast. When a multicast is issued, the multicast management table described later may be updated in order to guide each client 2 to switch to the multicast closer in time.

The multicast distribution unit 108 multicasts the individual viewpoint stream and the whole viewpoint stream according to the result of the scheduling. When using LTE as a mobile network, multicast can be performed by eMBMS. The unicast distribution unit 109 unicasts the individual viewpoint stream and the whole viewpoint stream according to the request of each client.

The schedule notification unit 110 gives the client 2 who requested viewing the opportunity to participate in multicast and unicast by notifying the result of the distribution schedule. The multicast information issued above is registered or updated in the multicast management table 111.

As shown in FIG. 6, the multicast management table 111 manages at least the multicast ID, client ID, type, viewpoint ID, start time, reproduction time, and bit rate as information related to each multicast. The schedule notification unit 110 can notify the multicast distribution schedule by distributing the contents of the multicast management table 111 to each client 2.

In the client 2, when the user specifies a playback position and a viewpoint by a seek operation or the like, the viewing request unit 201 transmits a viewing request for a video stream including the playback position x1 and the viewpoint distribution d to the video distribution server 1.

The viewing determination unit 202 collates the result of the distribution schedule notified from the video distribution server 1 with the content of the viewing request, and determines whether or not the video distribution server 1 needs to receive the multicast all-viewpoint stream and the individual viewpoint stream. To do. When the unicast request unit 203 determines that the multicast all-viewpoint stream and / or the individual viewpoint stream is not received, the unicast request unit 203 requests the video distribution server 1 to unicast the requested video stream again.

7 and 8 are flowcharts showing a procedure in which the video distribution server 1 distributes the video stream in response to the viewing request of the video stream transmitted by the client 2. In the present embodiment, when the user specifies the playback position x1 and the viewpoint p on the display of the client 2, for example, by performing a seek operation or the like, the viewing request of the all-viewpoint stream and the viewing request of the individual viewpoint stream are video. It is transmitted to the distribution server 1.

FIG. 7 is a flowchart showing a distribution procedure of the entire viewpoint stream. At time tc1, when the user specifies a desired playback position x1 and viewpoint p by a seek operation or the like, at time tc2, the playback position x1 and the current time A viewing request for all viewpoint streams including t1 is transmitted from the viewing request unit 201 to the video distribution server 1.

The video distribution server 1 monitors the viewing status of the client at time ts1, and when the viewing request for the all-viewpoint stream is received at time ts2, the viewing request is received by the viewing request receiving unit 105 at time ts3. , The playback position x1 is extracted from the viewing request and registered in the client management table 106.

At time ts4, the one-dimensional probability density distribution described with reference to FIG. 4 is estimated based on the reproduction positions x extracted from a plurality of viewing requests received within a predetermined period from the current time t1.

At time ts5, whether or not the number of clients playing [x1-δ, x1] within a predetermined time range based on the playback position x1 at the current time t1 exceeds a predetermined threshold value based on the probability density distribution. Is judged. If the threshold is exceeded, at time ts6, a new multicast for all viewpoint streams including the playback position x1 is issued. At time ts7, the multicast management table is updated. At time ts8, the multicast management table is transmitted to the client 2.

When each client 2 acquires the multicast management table at time tc3, it is determined at time tc4 whether or not the multicast to be joined is registered. In the case of an all-viewpoint stream, it is judged that participation should be performed if the difference Δx1 between the multicast playback position and the client playback position x1 is within a certain period of time. At time tc5, the multicast is received.

On the other hand, if the multicast to be joined is not registered, the unicast transmission request including the playback position x1 is transmitted to the video distribution server 1 at time tc6, and the entire viewpoint stream is received by unicast. At time tc7, all viewpoint streams received by multicast or unicast are buffered. At time tc8, the free viewpoint video is synthesized and reproduced based on the designated viewpoint P, as will be described in detail later with reference to FIG.

FIG. 8 is a flowchart showing a distribution procedure of the individual viewpoint stream. At time tc1, when the user specifies a desired playback position x1 and viewpoint p by a seek operation or the like, at time tc2, the playback position x2 and the current time A viewing request for an individual viewpoint stream including t2 and the viewpoint distribution d is transmitted from the viewing request unit 201 to the video distribution server 1.

The video distribution server 1 monitors the viewing status of the client at time ts1, and when the viewing request for the individual viewpoint stream is received at time ts2, the viewing request is received by the viewing request receiving unit 105 at time ts3. , The playback position x2 and the viewpoint distribution d are extracted from the viewing request and registered in the client management table 106.

At time ts4, the distribution of the two-dimensional probability density described with reference to FIG. 5 is estimated based on the playback position x and the viewpoint distribution d extracted from a plurality of viewing requests received within a predetermined period from time t2. To.

At time ts5, the number of clients playing back within a predetermined time range [x1-δ, x1] based on the playback position x2 and the viewpoint distribution d at time t2 exceeds a predetermined threshold value based on the probability density distribution. It is judged whether or not it is. If the threshold is exceeded, at time ts6, a new multicast of the individual viewpoint stream including the reproduction position x2 and corresponding to the viewpoint distribution d is issued. At time ts7, the multicast management table is updated. At time ts8, the multicast management table is transmitted to the client 2.

When each client 2 acquires the multicast management table at time tc3, it is determined at time tc4 whether or not the multicast to be joined is registered. In the case of an individual viewpoint stream, it is judged that participation should be performed if the difference Δx1 between the multicast playback position and the client playback position x1 is within a certain period of time.

In the case of an individual viewpoint stream, the viewpoint distribution after Δx2 is further calculated, and if the selection probability of the viewpoint during multicast is high, it is judged that participation should be performed. In this embodiment, if the viewpoint position is a continuous value and the series values I1, I2, I3 ... Are stochastic processes (Wiener process), the viewpoint position I (Δx2) after Δx2 can be expressed by the following equation.

I (Δx2) ＝ I0 + μt ± σ√t… (1)

Here, I0 is the current viewpoint position, μ is the trend term related to the viewpoint position change, and σ is the magnitude of fluctuation, so the viewpoint distribution is the distribution of the standard deviation σ centered on I (Δx2). At time tc5, the multicast is received.

On the other hand, if the multicast to be joined is not registered, the video distribution server 1 is requested to deliver by unicast at time tc6. At time tc7, the individual viewpoint stream delivered by multicast or unicast is buffered. At time tc8, the free viewpoint video is synthesized and played back based on the specified viewpoint p.

On the other hand, if the multicast to be joined is not registered, the unicast transmission request including the reproduction position x1 and the viewpoint distribution d is transmitted to the video distribution server 1 at time tc6.

When multicast is distributed at multiple bit rates, the throughput allocated to each viewpoint is calculated as in Reference 1, and even if the multicast bit rate is too high (too low), it is received by unicast. You may. The client requests the server to update the multicast management table.

In addition, when multicast is delivered at multiple bit rates, the throughput allocated to each viewpoint is calculated as in Reference 1, and if the multicast bit rate is too high (too low), it is received by unicast. You may. The client requests the server to update the multicast management table.

When playing back the free-viewpoint video, as shown in FIG. 9, the client 2 further adds a bitstream of object information (mask image, three-dimensional position, size) to be viewed from the free-viewpoint together with the multi-viewpoint stream. Receive. By decoding these streams, the client 2 acquires the decoded image and the object information (decoding mask image, three-dimensional position, size) of the multi-viewpoint, and extracts the object image from the decoded image and the mask image.

The client 2 further arranges this on the 3D model based on the 3D position information and the size, moves the virtual viewpoint according to the viewpoint p specified by the user, and performs projective conversion to the 2D image. Display the image.

1 ... Video distribution server, 2 ... Client, 3 ... Network, 101 ... Multi-viewpoint video storage unit, 102 ... Individual viewpoint stream generation unit, 103 ... All viewpoint stream generation unit, 104 ... Free viewpoint data generation unit, 105 ... Viewing request Reception unit, 106 ... Client management table, 107 ... Scheduling unit, 108 ... Multicast distribution unit, 109 ... Unicast distribution unit, 110 ... Schedule notification unit, 111 ... Multicast management table

Claims (9)

In the video distribution server that distributes the video stream to the client
A means to generate an individual viewpoint stream based on a multi-view video,
A means to combine multi-view videos to generate an all-view stream,
A means of determining a schedule for multicasting each video stream based on requests from each client,
A means of notifying the client of the determined schedule and
A video distribution server including means for multicasting each video stream based on the determined schedule. The means for determining the schedule is
A means of extracting parameters related to the specification of the playback position from the viewing request of the entire viewpoint stream,
It is provided with a means for estimating the distribution of the number of clients for each playback position based on the playback position extracted from the viewing request.
The video distribution server according to claim 1, wherein a schedule for multicasting all viewpoint streams corresponding to playback positions whose distribution exceeds a predetermined threshold value is determined. The means for determining the schedule is
A means for extracting the playback position and viewpoint distribution from the viewing request of the individual viewpoint stream,
A means for estimating a two-dimensional distribution of the number of clients for each playback position and viewpoint based on the playback position and viewpoint distribution extracted from the viewing request.
The video distribution server according to claim 1 or 2, wherein the schedule for multicasting the individual viewpoint streams corresponding to the playback positions and viewpoints whose two-dimensional distribution exceeds a predetermined threshold value is determined. In the video distribution method in which the video distribution server distributes the video stream to the client
The client
The procedure for sending a video stream viewing request and
The video distribution server
Procedures for generating individual viewpoint streams and all viewpoint streams based on multi-view video,
Procedures for determining the schedule for multicasting each video stream,
The procedure for notifying the client of the determined schedule and
The procedure for multicasting each video stream based on the determined schedule and
The client also
A video distribution method comprising a procedure for receiving each video stream based on the notified schedule. The procedure for determining the schedule is
The procedure for extracting the parameters related to the specification of the playback position from the viewing request of the all-view stream, and
Including a procedure for estimating the distribution of the number of clients for each playback position based on the playback position extracted from the viewing request.
The video distribution method according to claim 4, wherein the schedule for multicasting the all-viewpoint stream corresponding to the playback position whose distribution exceeds a predetermined threshold value is determined. The procedure for determining the schedule is
The procedure for extracting the playback position and viewpoint distribution from the viewing request of the individual viewpoint stream, and
Including a procedure for estimating a two-dimensional distribution of the number of clients for each playback position and viewpoint based on the playback position and viewpoint distribution extracted from the viewing request.
The video distribution method according to claim 4 or 5, wherein the schedule for multicasting the individual viewpoint streams corresponding to the reproduction positions and viewpoints whose two-dimensional distribution exceeds a predetermined threshold value is determined. In a video distribution program that distributes a video stream from a video distribution server to a client
Procedures for generating individual viewpoint streams based on multi-view video, and
The procedure for combining multi-view videos to generate an all-view stream,
The procedure for determining the schedule for multicasting each video stream based on the viewing request obtained from each client, and
The procedure for notifying the client of the determined schedule and
A video distribution program comprising a procedure for multicasting each video stream based on the determined schedule. The procedure for determining the schedule is
The procedure for extracting the parameters related to the specification of the playback position from the viewing request of the all-view stream, and
Including a procedure for estimating the distribution of the number of clients for each playback position based on the playback position extracted from the viewing request.
The video distribution program according to claim 7, wherein the schedule for multicasting all viewpoint streams corresponding to playback positions whose distribution exceeds a predetermined threshold value is determined. The procedure for determining the schedule is
The procedure for extracting the playback position and viewpoint distribution from the viewing request of the individual viewpoint stream, and
Including a procedure for estimating a two-dimensional distribution of the number of clients for each playback position and viewpoint based on the playback position and viewpoint distribution extracted from the viewing request.
The video distribution program according to claim 7 or 8, wherein the schedule for multicasting the individual viewpoint streams corresponding to the playback positions and viewpoints whose two-dimensional distribution exceeds a predetermined threshold value is determined.