JP2020506614A - Predicted Bit Rate Selection for 360 Video Streaming - Google Patents

Abstract

Kind Code: A1 Predicted stream prefetch for 360 degree video is described. User view orientation metadata is obtained for a 360 degree video stream that includes data for multiple viewports. Data corresponding to one or more high resolution frames for a particular one of the viewports is prefetched based on the user view orientation metadata and the frames are displayed. High resolution frames are characterized by a higher resolution than for the remaining viewports. [Selection diagram] Fig. 1

Description

[Priority claim]
This application claims the benefit of priority of US Patent Application No. 15 / 481,324, filed April 6, 2017, the entire contents of which are incorporated herein by reference.

  Aspects of the present disclosure relate to video streaming. In particular, the present disclosure relates to streaming 360 degree video.

  A 360 degree video is created by taking video streams from multiple cameras arranged around a single point and splicing the video streams to create a single continuous video image. Modern encoders interpose a continuous video image in a video stream of multiple frames. To watch 360 degree video over the network, the server sends a stream of these multiple frames to the client. The client decodes and reconstructs the stream into a continuous image being presented on the display.

  The system may send a single request for frames, download the requested frames, and then construct them for display. This combination of actions is sometimes called a fetch action. In general, to ensure uninterrupted video streaming, the client must also prefetch the video. That means that the system must download the frames and process them before the previously downloaded frames are displayed. In this way, the system builds a buffer of processed frames between the displayed processed frame and subsequent frames that need to be downloaded and processed.

  Buffering can be very costly on system resources, especially when processing and storing high definition video. To conserve bandwidth and reduce the amount of buffering required, the client may request only high resolution video stream frames, also called viewports, within the client's field of view. In this case, the client receives the low resolution video stream for everything but the client's current view. The problem with this system is that clients are often able to move their field of view sooner than a high quality stream can be requested, delivered and buffered. Accordingly, there is a need in the art for a system that allows a client to predict where a view can be directed within a 360 degree video stream and fetch the corresponding high resolution video stream before the view is moved. There is a need.

  Disadvantages associated with the prior art include obtaining user view orientation metadata for the 360 degree video stream, prefetching frames determined by the user view orientation metadata, and 360 degree video with user view orientation metadata. It is overcome by aspects of the present disclosure relating to a method of prefetching 360 degree video, including displaying higher resolution frames of a stream.

  The teachings of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings.

FIG. 4 is an illustration of a virtual camera view in a 360 degree video, in accordance with aspects of the present disclosure. FIG. 4 is a diagram of predictive video fetch in equirectangular projection according to aspects of the present disclosure. FIG. 4 is an illustration of predictive video fetch in a cubic mapping projection, in accordance with aspects of the present disclosure. FIG. 2 is a block diagram of a system for displaying 360-degree video according to aspects of the present disclosure. FIG. 4 is a simplified circular view of a Markov chain for determining when to change a frame, in accordance with aspects of the present invention. FIG. 4 is a flow diagram illustrating a method for displaying 360-degree video according to aspects of the present disclosure.

  The following detailed description includes numerous specific details for the purpose of illustration, but those skilled in the art will recognize that numerous variations and modifications to the following details are within the scope of the invention. . Accordingly, the exemplary embodiments of the invention described below are set forth without loss of generality and without limitation to the claimed invention.

[Introduction]
Typically, streaming 360 degree video over a network involves receiving a set of all one quality video streams. Newer streaming technologies allow bandwidth utilization to be reduced by only loading high quality streams in the area of interest to the viewer. This technique has the added advantage of allowing viewers to load higher resolution video streams without requiring much time or buffer resources.

  The disclosed technology allows viewers to view higher quality video streams, but experiences annoying resolution loss when the user suddenly moves away from the high quality stream in the viewport. May be. Aspects of the present disclosure have been developed to eliminate such an unpleasant experience.

  In other cases, the 360-degree video creator may have some artistic vision of what the viewer should see in a scene of the 360-degree video. According to prior art methods, such details in the scene displayed to the viewer may be lost due to the display of the low resolution video or while the view is looking in another direction. There is. Accordingly, aspects of the present disclosure have been developed to improve the quality of perceived 360 degree video streams, and to allow authors to define viewports and video quality for viewers.

[Author-driven prefetch]
Within a 360 degree video, there can be many orientations that the camera can take to view the image, as can be seen in FIG. The camera 101 is fixed at a point 108, and the scene 100 wraps around the camera 101 to create a 360-degree viewable area. The camera may rotate about a fixed point 108 to view the image 103 in the scene. The scene may be broken down into different regions called viewports 102. Each viewport 102 may be a different video stream loaded by the client. As seen in FIG. 2, each section of the video may be divided into these viewports. The camera 101 may change orientation in any direction, for example, up 104, down 106, left 105, or right 107 to shift from one viewport 102 to another.

  FIG. 2 illustrates an equirectangular projection in accordance with aspects of the present disclosure. The 360 degree scene 200 is composed of a series of equally sized rectangles 201-208. Each rectangle may be a separate video stream that the client device loads and splices for display. Each rectangle may be large enough to encompass the camera's view, in which case they are viewports 201. Alternatively, multiple rectangles may together represent a single viewport (not shown).

  Video and image authors typically have some idea of what they want a viewer of the content to see. The creator of the 360 video is no different. As mentioned above, the 360 degree video displayed in the prior art had only one resolution. At lower resolutions, important aspects of the video image could be lost to the viewer.

  According to aspects of the present disclosure, an author may define a position 202 of a high resolution frame for a client to load. The author also defines metadata 209 in the video stream that the client can use to predictively load high-resolution video content for the stream corresponding to the portion of the 360-degree video that the user is likely to see. You may. By way of example, and not limitation, metadata 209 may be in the form of a vector having both magnitude and direction indicating importance. In some embodiments, time may be encoded alongside the metadata 209, or vectors may be placed in the stream at fixed time step intervals. Predefined is also called user view orientation metadata.

  In some implementations, the metadata may be generated on the back end or server side without explicit transmission of view orientation information by the client. By way of example, and not limitation, the server may construct a probability field based on which stream the client requests and when, and then map which stream belongs to which viewport. This assumes that the client selects the highest quality stream for the current client viewport.

  Metadata 209 may relate to the likelihood of the user moving viewport 201 in the video stream. Alternatively, the metadata 209 in the form of an author-defined prefetch vector may be a movement vector that is idealized for a viewer according to the author's artistic concept. The client device may track both the actual viewport movement 210 and the metadata 209. The client device may prefetch frames in the actual viewport 201 and frames 202, 203 along with the author-defined prefetch vector. Alternatively, the client may only fetch high resolution frames 202, 203 along an author-defined vector to prompt the viewer to move the viewport to a different position in the 360 degree stream. Good.

  Author-defined prefetch metadata need not be a vector. Authors may simply define the frames 202 they desire for a high resolution for a certain period of display. Thus, a client may fetch an author-defined high resolution frame at a time defined by the author.

  The author may also define certain areas of the frame as high resolution for the zoom function. The author may provide a level of detail for the subsections of the frame so that certain subsections of the frame are encoded as high resolution. Metadata that informs the client of the location of this high resolution subsection may be sent to the client so that the client can prefetch the stream.

  Metadata 209 may also be used to control viewport 201 during video display. The author may choose to move viewport 201 along metadata 209 without viewer input. In this way, a virtual photographer function can be realized. The author can better display the artistic concept on a 360 degree display.

[Direction definition prefetch]
During the addition of effects of frames of a video stream, also referred to as renditions, it may be desirable for the client to prefetch high resolution frames to match the rendition effects. By way of example, and not limitation, it may be desirable for the client to prefetch high resolution frames in the apparent direction of the loud sound. Alternatively, during the staging, it may be desirable for the client to prefetch frames where there are many special effects or special camera movements.

  According to aspects of the present disclosure, the client may receive metadata that causes the client to prefetch certain frames defined during the presentation. These predefined frames may correspond to special effects and sound cues, rather than artistic concepts such as those described above. The presentation definition prefetch metadata may also be referred to as user orientation metadata.

[Predictive Prefetch]
According to an alternative aspect of the present disclosure, the client uses predictive metadata to prefetch a stream determined to be in a potential future viewport, as seen in FIGS. You may. Predictive metadata may be generated during the rendition to ensure that end users receive higher resolution frames within their field of view than in systems employing a single resolution stream.

  The studio may use the screening data collected from the 360-degree video viewer to generate a probabilistic model of where the viewer may be watching the video at any time. This probability model is based on variables such as the current view orientation in the 360 degree video, the time code in the video, the past view of the video, etc. The possibility of staying in the current frame 201 may be defined. The probability of changing a frame may be represented by the probability associated with each frame in the currently displayed 360 degree video. This is represented by the percentage in each frame 201-208 seen in FIG. The predicted prefetch data may be referred to as user orientation metadata.

  Alternatively, the predictive prefetch metadata may be negative or reverse / reverse data. In other words, instead of probabilities of places where the user is likely to see, the prefetch metadata may instead represent one or more probabilities of places where the viewer is unlikely to see.

  Prefetching video frames is not limited to high and low resolution streams. As seen in FIG. 2, the system may choose to prefetch an intermediate resolution for frames 205, 206 having a certain threshold level of probability. Similarly, frames 204, 207, 208 that have a low probability of being viewports are only prefetched with the displayed low resolution image. As an additional aspect, low viewport probability frames 204, 207, 208 may receive fewer updates than higher probability frames, by way of example and not limitation. The low probability frame may be updated, for example, only once every two updates of the high probability stream. More generally, any factor of the optimal / high probability stream may be used to update the low probability stream depending on whether they are in view or not. In some implementations, low-probability streams that are still in view may be updated at a full high-probability update rate to avoid significant out-of-sync between spliced frames.

  To determine whether to prefetch the stream using this probability metadata, the client may have a defined threshold probability level. The client prefetches the frame when the probability metadata determines that the probability of the viewer moving the viewport within a frame exceeds a threshold probability level. In an alternative embodiment, the client may prefetch frames based on the fidelity of the probability metadata.

[Predictive fidelity check]
FIG. 3 illustrates a cubic mapping projection in accordance with aspects of the present disclosure. The predicted data represented as a percentage in each frame does not match the actual motion 305 of the viewport 301. In this example, the author prediction prefetch vector 303 also does not match the actual movement vector 305 of the viewport 301. In the case presented in FIG. 3, the actual viewport is eventually 302, while the system prefetches either a frame 301 based on predicted data or a frame 304 based on an author-defined prefetch vector. Thus, in the case of FIG. 3, the system may determine whether to continue following the viewer orientation metadata or default to a single quality level.

  The client may perform a continuous or intermittent check of the probabilistic and author-defined metadata fidelity. The client may first prefetch high resolution frames based on the probability metadata and the actual viewport 301 orientation. The client may then display the high resolution frame according to the viewport orientation and metadata. The client may check to determine if the viewer has moved the viewport to a higher resolution frame according to probability metadata or author-defined metadata.

  If the viewport determines that it is not in a frame that was prefetched according to the probability metadata or author-defined prefetch vector, the client stops using the metadata for prefetching and the high resolution within the viewport's current field of view. It may just fetch the frame. In an alternative embodiment, the client may continuously check the viewport movement using the metadata about the correlation. The client may have a tolerance level for viewport movement that does not follow the probability metadata. This tolerance level may be, by way of example and not limitation, the ratio of overlooked predicted frames to observed predicted frames. In that case, as the ratio increases to one, the client may shift to fetching only frames in the actual viewport. More generally, the tolerance level may be determined by statistically quantifying the amount of change in the set of values. Any suitable measure of variability, eg, standard deviation, may be applied to the fixed or sliding window user set and metadata.

[Generate prefetch after production]
Another aspect of the present disclosure is generation of prediction metadata based on final viewer data. The client device may collect data from the viewer regarding the viewport orientation. The client device may use the viewer data to generate prediction data and prefetch the video stream according to the prediction data. Client devices may also share the prediction data with other clients or servers to generate or refine stochastic metadata for the video stream.

  The client device may generate the prediction data using the viewport orientation data. By way of example, and not limitation, the client device may use the viewport's motion vector 305 to predictively fetch a high-resolution video stream that is on the vector and takes into account the speed of movement. The user-generated predicted prefetch data may be referred to as user orientation metadata.

  The client may send the viewport orientation data to the data collection center to generate better probability metadata for future viewing. The data collection center may use the viewport orientation data in addition to the author-defined prefetch data to generate metadata for the video stream. Unless the client strictly adheres to past user-defined viewing vectors, the generated metadata may weight the author-defined prefetch data so that author-defined stream prefetching takes precedence.

[Server side prefetch]
In an alternative embodiment of the present disclosure, the server uses the metadata to select a high-resolution stream to send to the client device. According to aspects of this embodiment, the metadata used by the server may be author definitions, probability data generated from viewers, or other predictive data. The server may receive a request for a 360 degree video stream from a client device. The server may check for viewer orientation metadata for the requested stream. Upon finding such metadata, the server may send a high-resolution stream to the client according to the metadata. The server may also receive the actual viewer orientation data and transmit the high resolution stream in the view of the actual viewport. Further, the server may perform a predictive fidelity check as described above to determine whether the server should continue to transmit the high resolution video stream based on the metadata.

[Embodiment]
FIG. 4 illustrates a system according to aspects of the present disclosure. The system may include a computing device 400 coupled to a display 401 and a user input device 402. Display device 401 may be in the form of a cathode ray tube (CRT), flat panel screen, touch screen, or other device that displays text, numbers, graphical symbols, or other visual objects. User input device 402 may be a controller, touch screen, or other device that allows the user to interact with the user view orientation and select a 360 degree video stream. In some embodiments, the display 401 may be a 360 degree display configured to simultaneously display multiple viewports of the 360 degree video for an immersive 360 degree video experience. In another embodiment, display 401 may be a conventional two-dimensional display. In such an embodiment, the user may be able to determine the viewport by interaction with the computing device.

  Computing device 400 may include one or more processor units 403, which may be configured according to well-known architectures such as, for example, single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The computing device may also include one or more memory units 404 (eg, random access memory (RAM), dynamic random access memory (DRAM), read only memory (ROM), etc.).

  Processor unit 403 may execute one or more programs, some of which may be stored in memory 404, and processor 403 may access the memory via data bus 405, for example, It may be operatively connected to a memory. The program may be configured to request frames for the video stream based on the metadata 410 received for the video stream. When executed by the processor, the program may cause the system to decode 408 the high resolution frame and store in buffer 409 a possible frame in the viewer's viewport.

  Computing device 400 may also include well-known support circuits such as input / output (I / O) circuit 407, power supply (P / S) 411, clock (CLK) 412, and cache 413, which include: For example, it may communicate with other components of the system via bus 405. The computing device may include a network interface 414. The processor unit 403 and the network interface 414 are configured to implement a local area network (LAN) or a personal area network (PAN) via a suitable network protocol for the PAN, for example, Bluetooth®. You may. The computing device may optionally include a mass storage device 415, such as a disk drive, CD-ROM drive, tape drive, flash memory, and the like. The mass storage device may store programs and / or data. The computing device may also include a user interface 416 to facilitate interaction between the system and a user. The user interface may include a keyboard, mouse, light pen, game control pad, touch interface, or other device. In some implementations, the user may use the interface 416 to change the viewport, for example, by scrolling with a mouse or operating a joystick. In some embodiments, display 401 may be a hand-held display, such as in a smartphone, tablet computer, or portable gaming device. In such an embodiment, user interface 416 may include an accelerometer that is part of display 401. In such an embodiment, the processor 403 may be configured to detect a change in the orientation of the display and to use this information to determine the viewport, for example, by appropriate programming. Thus, the user can change the viewport by simply moving the display.

  In some embodiments, the prediction metadata may be configured to implement a Markov chain. FIG. 5 illustrates a graphical representation of a Markov chain, according to aspects of the present disclosure. Each circle represents a state 501-506, for example, a viewport orientation in a 360 degree video. There are six states in this Markov chain, representing a cube-mapped video stream. Each arrow represents the probability that the viewport will change to another of the video stream viewports. These transfer probabilities are manually set by the supervisor or producer and can be determined from focus group data or from actual viewer data. Each frame may have its own Markov probability at each point in the video stream. According to one embodiment of the present disclosure, the system may only receive metadata representing a Markov probability for a transition from the current viewport orientation to a different orientation. For example, the client displaying the current viewport at 501 receives the probability of moving from the current viewport 501 to a different one of the plurality of viewports 502-506, or the probability of staying at the current viewport 501. Just do it. In this way, the system may reduce initial load time and bandwidth utilization by receiving only the probability of transition from the current state while receiving the stream. Alternatively, the system may receive the entire Markov probability model for the video stream at each point in time at the start of the video stream. This embodiment increases the initial load time for the stream, but reduces the overall information that needs to be processed during streaming. In another embodiment, the system may receive a Markov stochastic model at each point during the streaming of the 360 degree video. This embodiment reduces the load time of the upfront investment at the expense of stream processing, but further includes additional Markov probability data in case of out-of-sync between client and server.

  To implement author-defined prefetch metadata as described above using a Markov chain model, the system may weight the transition probability to a state along the author-defined prefetch vector. Thus, if the weighted probability of movement is above the threshold, the client initiates prefetch for that state.

  According to aspects of the present disclosure, a client may receive viewer orientation metadata for a requested video stream independently of the video stream. If the metadata is received independently of the video stream, it must be time synchronized to the stream. In an alternative aspect of the present disclosure, the metadata may be received as part of the video stream itself, by way of example and not limitation, in the header of the 360 degree video stream.

  FIG. 6 shows a flow diagram for predictive bit rate selection according to aspects of the present disclosure. The system may request 601 a video stream from the network. Upon request for a video stream, the system may operate from a network or from a storage medium, such as a compact disk, stored locally in mass storage memory 415, or from any suitable type of data storage known in the art. , User view orientation metadata may be obtained 602. It should be noted that, as described above, user view orientation metadata may also be obtained during streaming. Viewer orientation metadata may inform the system of frames to fetch before display, including potential viewports that the user has not yet entered, as described above. Thus, the system is a potential or desired viewport as well as the initial starting viewport frame at the higher resolution and the remaining frames in the 360 degree video stream at the lower resolution, as described above. The frame is prefetched 603. The system then displays 604 the prefetched frame, for example, through display device 401 as described above. While displaying 604 the prefetched frames, the system may continue 605 prefetching the frames 603 to create an uninterrupted 360 degree video streaming experience using the high resolution frames in the user's field of view. Optionally, the system uses 606 the predictive fidelity check 607 to continue prefetching frames according to the view orientation metadata, or the current load where only frames in the current viewport are prefetched at a higher resolution. A decision 608 may be made to enter the state. In the current load state, the system may continue 609 checking the prediction fidelity until the prediction is correct as described above. At that point, the system can resume 608 prefetch 603 based on the user view orientation metadata.

  While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. The scope of the invention should, therefore, be determined without reference to the above description, but instead should be determined with reference to the appended claims along with their full scope of equivalents. is there. Any feature, whether or not preferred, may be combined with any other feature, whether or not preferred. In the following claims, the indefinite article "A" or "An" refers to the quantity of one or more of the items following the article, unless explicitly stated otherwise. Unless such limitations are expressly recited in a given claim using the phrase “means for”, the appended claims are to be interpreted as including means-plus-function limitations. Should not be done. Any element in the claims that does not explicitly state a "means for" performing the specified function may be a "means" or a "means" specified in 35 USC 112,6. It should not be interpreted as a "step."

Claims (20)

a) obtaining user view orientation metadata for a 360 degree video stream including data for a plurality of viewports;
b) prefetching data corresponding to one or more high resolution frames for a particular viewport of the plurality of viewports as determined by the user view orientation metadata;
c) displaying the one or more high resolution frames, wherein the one or more high resolution frames are characterized by a higher resolution than for a remaining one of the plurality of viewports; Said displaying;
A method that includes   The method of claim 1, wherein the user view orientation metadata is an author-defined prefetch vector.   The method of claim 1, wherein the user view orientation metadata is a prefetch vector created to match staging effects in scenes of the 360 ° video stream.   The method of claim 1, wherein the view orientation metadata is a predicted probability of a viewport frame change.   5. The method of claim 4, wherein the predicted probabilities are generated by a focus group.   5. The method of claim 4, wherein the predicted probabilities are generated using user generated frame position data.   5. The method of claim 4, wherein the predicted probabilities also include an author-defined prefetch vector.   The method of claim 7, wherein the author-defined prefetch vector is a weight applied to the probability of a viewport frame change.   The method of claim 4, wherein prefetching a frame comprises applying a first threshold to the user view orientation metadata to determine the particular viewport of the plurality of viewports. Method.   5. The method of claim 4, wherein the predicted probability of a viewport frame change is a probability for a Markov model.   The method of claim 1, wherein the user view orientation metadata is a motion vector.   10. The method of claim 9, wherein b) comprises prefetching an intermediate resolution frame when the prediction probability exceeds a second threshold.   The method of claim 1, wherein the user view orientation metadata represents a probability of where a user is likely to see.   The method of claim 1, wherein the user view orientation metadata represents a probability of a location unlikely to be seen by a user. Checking the user view orientation metadata against the actual user view orientation;
Disabling prefetching when the user view orientation metadata does not match the actual user view orientation being displayed;
The method of claim 1, further comprising:   Disabling prefetching when the user view orientation metadata does not match the actual user view orientation being displayed comprises applying an oversight frame threshold to determine whether to disable prefetching. The method of claim 15, comprising:   2. The method of claim 1, wherein prefetching data corresponding to one or more high resolution frames determined by the user view orientation metadata comprises prefetching frames having higher resolution sections than the remaining frames. The method described in. a) obtaining user view orientation metadata for a 360 degree video stream including data for a plurality of viewports;
b) prefetching data corresponding to one or more high resolution frames for a particular viewport of the plurality of viewports as determined by the user view orientation metadata;
c) displaying the one or more high resolution frames, wherein the one or more high resolution frames are characterized by a higher resolution than for a remaining one of the plurality of viewports; Said displaying;
Non-transitory computer readable medium containing program instructions for causing a computer to perform the method of claim 1. A processor,
A display coupled to the processor;
A memory coupled to the processor having processor-executable instructions embodied therein, wherein the instructions are configured to perform a method when executed by the processor;
A system comprising:
The method comprises:
a) obtaining user view orientation metadata for a 360 degree video stream including data for a plurality of viewports;
b) prefetching data corresponding to one or more high resolution frames for a particular viewport of the plurality of viewports as determined by the user view orientation metadata;
c) displaying the one or more high-resolution frames using the display, wherein the one or more high-resolution frames have a higher resolution than the remaining viewports of the plurality of viewports. Displaying, characterized by:
Including the system.   20. The system of claim 19, wherein the display is a 360 degree display configured to simultaneously display the plurality of viewports.