RU2629001C2 - Improved blocks transmission steaming system on request for streaming processing with small delay - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: streaming system is proposed on request of blocks for streaming with a low multimedia presentation delay. According to the encoding protocol, a plurality of multimedia segments is formed, where each media segment includes a random access point. In accordance with the same protocol, a plurality of multimedia fragments is coded, and the multimedia segments are aggregated from a plurality of media fragments.

EFFECT: streamlining the presentation of streaming media and providing efficient simultaneous or time-shared streaming of multimedia data.

18 cl, 33 dwg

Description

FIELD OF THE INVENTION

The present invention relates to improved media streaming systems and methods, and more particularly, to systems and methods that adapt to network and buffer conditions to optimize streaming media performance and enable efficient simultaneous or time-distributed streaming of multimedia data.

BACKGROUND

Streaming multimedia can become more and more important as it becomes common for high-quality audio and video that need to be transmitted over packet networks such as the Internet, cellular and wireless networks, power line networks, and other types of networks. The quality with which streaming multimedia can be presented may depend on a number of factors, including the resolution (or other attributes) of the original content, the quality of encoding of the original content, the capabilities of the receiving devices to decode and present multimedia, the timeliness and quality of the signal received at the receivers, etc. In order to create a good impression of the perception of streaming multimedia, transport and the timeliness of the signal received at the receivers can be especially important. Good transport can ensure the accuracy of the stream received at the receiver as to what the sender sends, while timeliness can represent how quickly the receiver can start playing content after the initial request for that content.

A multimedia transmission system can be characterized as a system having multimedia sources, multimedia destinations and channels (in time and / or space) separating the sources and destinations. Typically, the source includes a transmitter with access to multimedia with electronic control, and a receiver with the ability to electronically control the reception of multimedia (or its approximation) and issue it to a multimedia consumer (for example, a user having a display device connected in some way to a memory receiver device or element, another channel, etc.).

Although there are many options, in a general example, a multimedia transmission system contains one or more servers that have access to multimedia content in electronic form, and one or more client systems or devices generate multimedia requests to the servers, and the servers transmit multimedia using the transmitter as parts of the server transmitting to the receiver at the client so that the client can consume received multimedia in some way. In a simple example, there is one server and one client for a given request and response, but this is not necessary.

Typically, multimedia transmission systems can be described by either a “download” model or a “streaming” model. The “download” model may differ in the independence of the time distribution between the transmission of multimedia data and the playback of multimedia to a user or a recipient device.

As an example, multimedia is loaded in sufficient quantities in advance before when it is needed or will be used, and when it is used, it is already available in the right amount at the recipient. Transmission in the context of downloading is often accomplished using a file transfer protocol, such as HTTP, FTP, or Unidirectional File Delivery (FLUTE), and the transmission speed can be determined by the underlying stream and / or congestion tracking protocol, such as TCP / IP. The operation of the flow or congestion tracking protocol may not depend on multimedia playback to the user or destination device, which may occur simultaneously with the download or at some other time.

The “streaming” mode can be characterized by a strong connection between the distribution in time of the transmission of multimedia data and the playback of multimedia to a user or a recipient device. Transmission in this context is often performed using a streaming protocol, for example, a real-time streaming protocol (RTSP) for control and a real-time transport protocol (RTP) for multimedia data. The transmission speed can be determined using the streaming server, often it coincides with the speed of data playback.

Some disadvantages of the “download” model may be that due to the independence of the distribution over the transmission and playback time, multimedia data may or may not be available when it is needed for playback (for example, due to the fact that the available bandwidth is less than speed multimedia data transfer), causing playback to stop momentarily (“stopping”), which results in poor user experience, or you may need to download multimedia data much earlier than playback ( For example, due to the fact that the available bandwidth is greater than the transmission speed of multimedia data), consuming storage resources on the receiving device, which may be scarce, and consuming valuable network resources for transmission, which may be wasted if the content is ultimately account is not reproduced or used in any other way.

An advantage of the “download” model may be that the technology needed to perform such downloads, such as HTTP, is very mature, widely used and applicable to a wide range of applications. Download servers and solutions for the scalability of such file downloads (for example, HTTP Web servers and content transfer networks) can easily be available, making deployment of services based on this technology simple and inexpensive.

Some of the drawbacks of the streaming model may be that the multimedia data rate usually does not adapt to the available bandwidth from the server to the client, and that specialized streaming servers or a more complex network architecture providing bandwidth and guarantees are needed delays. Although there are streaming systems that support changing the data rate according to the available bandwidth (for example, Adobe Flash Adaptive Streaming), they are usually not as effective as transport control streaming protocols, for example, TCP, using all available in bandwidth availability.

Recently, new multimedia transmission systems have been developed and deployed based on a combination of streaming and download models. An example of such a model is referred to in this document as a “block-demand streaming” model in which the multimedia client requests multimedia blocks from the serving infrastructure using a download protocol, such as HTTP. The question in such systems may be the ability to start playing a stream, for example decoding and visualizing received audio and video streams using a personal computer and showing the video on a computer screen, and playing audio through the built-in speakers, or, as another example, decoding and visualizing the received audio and video streams using a set-top box, and showing video on a television display device and playing audio through a stereo system.

Other issues, such as the ability to decode source blocks fast enough to keep up with the source streaming speed, minimize decoding delay, and reduce the use of available CPU resources, are problems. Another issue is the provision of a sustainable and scalable streaming solution that allows system components to fail without adversely affecting the quality of streams transmitted to receivers. Other problems may arise based on rapidly changing presentation information when it is distributed. Thus, it is desirable to have improved processes and apparatus.

SUMMARY OF THE INVENTION

A block-based streaming system provides improvements in user perception and bandwidth efficiency in such systems, typically using a capture system that generates data in a form suitable for serving with a traditional file server (HTTP, FTP or the like), while the capture system allows content to be received and prepares it in the form of files or data elements for servicing by a file server, which may or may not include a cache.

In accordance with an embodiment, a multimedia server of a streaming system upon request of a block enables low-delay streaming of multimedia presentation content. Relatively large multimedia segment files for streaming a live profile can be aggregated from relatively small multimedia fragments for low latency streaming. The multimedia segments and multimedia fragments are encoded in accordance with the same encoding protocol.

The following detailed description, together with the accompanying drawings, will provide a more complete understanding of the subject and advantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts elements of a block-upon-demand streaming system in accordance with embodiments of the present invention.

FIG. 2 illustrates a block request streaming system of FIG. 1, showing more details in elements of a client system that connects to a serving block infrastructure (“BSI”) for receiving data that is processed by a content capturing system.

FIG. 3 illustrates a hardware / software implementation of a capture system.

FIG. 4 illustrates a hardware / software implementation of a client system.

FIG. 5 illustrates possible structures of the content store shown in FIG. 1, including segments and multimedia presentation descriptor (“MPD”) files, and segment decoding, time distribution, and other structure in the MPD file.

FIG. 6 illustrates details of a typical source segment that may be stored in the content repository illustrated in FIG. 1 and 5.

FIG. 7a and 7b illustrate simple and hierarchical indexing in files.

FIG. 8 (a) illustrates the definition of variable block sizes with aligned search points on multiple versions of a multimedia stream.

FIG. 8 (b) illustrates the definition of variable block sizes with non-aligned search points on multiple versions of a multimedia stream.

FIG. 9 (a) illustrates a metadata table.

FIG. 9 (b) illustrates the transfer of blocks and metadata tables from server to client.

FIG. 10 illustrates blocks that are independent of RAP boundaries.

FIG. 11 illustrates a continuous and discontinuous time distribution over segments.

FIG. 12 is a drawing showing an aspect of scalable blocks.

FIG. 13 depicts a graphical representation of the evolution of some variables over time in a streaming system upon request of blocks.

FIG. 14 depicts another graphical representation of the evolution of some variables over time in a streaming system upon request of blocks.

FIG. 15 depicts a grid of states versus threshold values.

FIG. 16 is a flowchart of a process that may be performed at a receiver that may request single blocks and multiple blocks in a request.

FIG. 17 is a flowchart of a flexible conveyor process algorithm.

FIG. Figure 18 illustrates an example at some point of a potential set of requests, their priorities, and over which connections they can be issued.

FIG. 19 illustrates an example of a potential set of requests, their priorities, and by what connections they can be issued, which passed from one moment to another.

FIG. 20 is a block diagram of a compatible cache proxy selection based on a file identifier.

FIG. 21 illustrates the definition of syntax for a suitable expression language.

FIG. 22 illustrates an example of a suitable hash function.

FIG. 23 illustrates examples of file ID compilation rules.

FIG. 24 (a) to (e) illustrate bandwidth variations in TCP connections.

FIG. 25 illustrates several HTTP source and recovery data requests.

FIG. 26 illustrates an example channel switching time with and without FEC.

FIG. 27 illustrates details of a recovery segment generator, which, as part of FIG. 1 capture system, forms recovery segments from the source segments and control parameters.

FIG. 28 illustrates the relationship between source blocks and recovery blocks.

FIG. 29 illustrates a procedure for live streaming services at various times on a client.

FIG. 30 illustrates the relationship between multimedia fragments for low-latency streaming and multimedia fragments.

In the drawings, the same elements are referenced using the same numbers, and subindexes are indicated in parentheses to indicate multiple instances of similar or identical elements. Unless otherwise indicated, the final subindex (for example, “N” or “M”) is not intended to be limiting to any particular value, and the number of instances of one element may differ from the number of instances of another element, even when the same number is illustrated and reused subindex.

DETAILED DESCRIPTION OF THE INVENTION

As described in this document, the task of a streaming system is to move multimedia from a storage location (or a place where it is formed) to a place where it is consumed, i.e. it appears to the user or is otherwise “used” by a person or an electronic consumer. Ideally, a streaming system can provide continuous playback (or, more generally, continuous “consumption”) at the receiving end and can start playing a stream or a set of streams shortly after a user requests a stream or streams. For reasons of efficiency, it is also desirable that each stream stop as soon as the user indicates that the stream is no longer needed, for example, when the user switches from one stream to another stream or follows a stream view, for example, a subtitle stream. If a multimedia component, such as a video, continues to be presented, but another stream is selected to represent this multimedia component, it is often preferable to take up the limited bandwidth of the new stream and stop the old stream.

A block-request streaming system in accordance with the embodiments described herein provides many advantages. It should be understood that a viable system should not include all the features described in this document, since some applications may not provide a satisfactory perception with all the features described in this document.

HTTP STREAM

HTTP streaming is a special type of streaming. When streaming HTTP, sources can be standard web servers and content transfer networks (CDNs) and can use standard HTTP. This technique may involve thread segmentation and the use of multiple threads, all within the framework of standardized HTTP requests. Multimedia, such as video, can be encoded at multiple data rates to form different versions, or displays. The terms "version" and "display" are used synonymously in this document. Each version or display can be broken down into smaller portions, perhaps of the order of several seconds each, to form segments. Each segment can then be saved on a web server or CDN as a separate file.

On the client side, you can then perform requests using HTTP to individual segments that seamlessly fit together with the client. The client can switch to different data rates based on the available bandwidth. The client can also request several displays, each representing a different multimedia component, and can represent multimedia in these displays simultaneously and synchronously. Triggers for switching may include, for example, buffer occupancy and network measurements. When operating in a stable state, the client can set the rate of requests to the server in order to maintain target buffer occupancy.

The benefits of HTTP streaming can include speed adaptation, fast startup and search, and minimal unnecessary transmission. These benefits stem from transmission control so that it only slightly outstrips playback, utilizing the maximum available bandwidth (through variable speed multimedia), and optimizing stream segmentation and smart client procedures.

A description of the multimedia presentation may be provided to the HTTP streaming client, so that the client can use a plurality of files (for example, in formats specified by 3GPP, referred to in this document as 3gp segments) to provide the user with streaming services. The description of the multimedia presentation, and if possible updating this description of the multimedia presentation, describes a multimedia presentation, which is a structured set of segments, each containing multimedia components, so that the client can present the included multimedia in a synchronized manner and can provide advanced functionality, for example, search, speed switching data transmission and joint presentation of multimedia components in different displays. A client may use multimedia presentation description information in various ways of providing a service. In particular, from the description of the multimedia presentation, the HTTP streaming client can determine which segments together can be accessed so that the data is applicable to the client and user capabilities within the streaming service.

In some embodiments, the description of the multimedia presentation may be static, although segments may be formed dynamically. The description of the multimedia presentation can be as compact as possible in order to minimize the access and download times for the service. The remaining connectivity to a dedicated server can be minimized, for example, by regular or frequent synchronization of the time distribution between the client and the server.

A multimedia presentation can be configured to allow access to terminals with different capabilities, for example, access to different types of access networks, with different current network conditions, display sizes, access speeds and codec support. The client can then retrieve suitable information to provide the user with streaming services.

A media presentation description can also provide deployment flexibility and compactness as required.

In the simplest case, each alternate mapping can be stored in a single 3GP file, i.e. in a 3GPP TS26.244 compliant file, or in any other file that conforms to the base ISO media file format specified in ISO / IEC 14496-12 or derived specifications (for example, the 3GP file format described in 3GPP 26.244 datasheet). In the remainder of this document, when referring to a 3GP file, it should be understood that ISO / IEC 14496-12 and its derivative specifications can correlate all the features described with the more general basic ISO multimedia file format defined in ISO / IEC 14496-12 or any derivative specifications . The client can then request the start of the file to find out the multimedia metadata (which is usually stored in the Movie header block, also called the “moov” block), along with movie fragments and byte offsets. The client can then issue partial HTTP GET requests to receive movie fragments if necessary.

In some embodiments, it may be desirable to divide each display into several segments, where segments. In the case where the segment format is based on the 3GP file format, then the segments contain non-overlapping time sections of movie fragments called “time sharing”. Each of these segments can contain multiple movie fragments, and each can be a valid standalone 3GP file. In another embodiment, the display is divided into an initial segment containing metadata (usually a moov movie title block), and a set of multimedia segments, each containing multimedia data, and a concatenation of the initial segment and any multimedia segment constitute a valid 3GP file as well as a concatenation the initial segment and all multimedia segments of one display forms a valid 3GP file. The whole view can be formed by playing each segment in turn, correlating the local time stamps inside the file with the global time of the presentation in accordance with the start time of each display.

It should be noted that throughout this description, references to a “segment” should be understood as including any data object that is fully or partially generated or read from a storage medium or otherwise obtained as a result of a request using a file upload protocol, including, for example, a request HTTP For example, in the case of HTTP, data objects can be stored in actual files located on a disk or other storage medium connected to or forming part of the HTTP server, or data objects can be generated using a CGI script or other dynamically executable program that is executed in response to HTTP request. The terms “file” and “segment” are used synonymously in this document unless otherwise indicated. In the case of HTTP, the segment can be considered as the body of the contents of the response to the HTTP request.

The terms “presentation” and “content item” are used synonymously in this document. In many examples, the presentation is an audio, video, or other multimedia presentation that has a predetermined “play” time, but other options are possible.

The terms “block” and “fragment” are used synonymously in this document unless otherwise indicated, and usually refer to the smallest aggregation of data that is indexed. Based on the available indexing, a client can request different parts of a fragment in different HTTP requests, or it can request one or more consecutive fragments or parts of fragments in a single HTTP request. In the case where segments based on the basic ISO multimedia file format are used or segments based on the 3GP file format, a fragment usually refers to a movie fragment defined as a combination of a movie fragment title block (“moof”) and multimedia data block (“mdat”) .

In this document, a data network is assumed to be a packet network in order to simplify the descriptions in this document, with the understanding that after reading this disclosure, one skilled in the art can apply the embodiments of the present invention described in this document to other types of networks transmissions, for example, networks with a continuous bit stream.

In this document, FEC codes are intended to provide protection against long and variable data transmission times in order to simplify the descriptions in this document, with the understanding that after reading this disclosure, one skilled in the art can apply embodiments of the present invention to other types of data transmission problems , for example, distortion when inverting data bits. For example, in the absence of FEC, if the last part of the requested fragment arrives much later or has a large variation in the arrival time than the previous parts of the fragment, then the content switching time can be large and variable, while using FEC and parallel requests, only the majority of the data requested for the fragment must arrive before they can be restored, thereby reducing the content switching time and the variability of the content switching time. In this description, it can be assumed that the data that needs to be encoded (ie, the original data) is divided into “characters” of equal length, which can be of any length (up to a single bit), but the characters can have different lengths for different parts data, for example, different character sizes can be used for different data blocks.

In this description, in order to simplify the descriptions in this document, it is assumed that FEC is applied to a “block” or “fragment” of data at a time, i.e. “Block” is the “source block” for the purposes of encoding and decoding FEC. The client device may use the segment indexing described in this document to help determine the structure of the source block in the segment. One skilled in the art can apply embodiments of the present invention to other types of structures of the source block, for example, the source block may be part of a fragment, or include one or more fragments or parts of fragments.

FEC codes considered for use with block-request streaming are usually systematic FEC codes, i.e. the source symbols of the source block may be included as part of the encoding of the source block, and thus the source symbols are transmitted. As one skilled in the art recognizes, the embodiments described herein apply equally to FEC codes that are not systematic. The FEC systematic encoder generates a number of recovery symbols from a source block of source symbols, and a combination of at least some of the source symbols and recovery symbols are coded symbols that are sent on a channel representing the source block. Some FEC codes may be useful for efficiently generating as many recovery symbols as necessary, for example, “information additive codes” or “fountain codes”, and examples of these codes include “chain reaction codes” and “multi-stage chain reaction codes”. Other FEC codes, such as Reed-Solomon codes, can practically only generate a limited number of recovery characters for each source block.

In many of these examples, it is assumed that the client connects to a multimedia server or a plurality of multimedia servers, and the client requests streaming multimedia on a channel or plurality of channels from a multimedia server or a plurality of multimedia servers. However, more complex configurations are also possible.

EXAMPLES OF ADVANTAGES

When streaming at the request of blocks, the media client maintains a relationship between the time distribution of these block requests and the time distribution of media playback for the user. This model may retain the advantages of the “download” model described above, while avoiding some of the disadvantages that arise from the usual disconnection between multimedia playback and data transfer. The block-based streaming model allows the use of speed and congestion tracking mechanisms in transport protocols, such as TCP, to ensure that the maximum available bandwidth is used for multimedia data. Moreover, dividing the multimedia presentation into blocks allows each block of encoded multimedia data to be selected from a set of several available encodings.

This selection can be based on any number of criteria, including matching the media speed with the available bandwidth, even when the available bandwidth changes over time, matching the media resolution or decoding complexity with the capabilities or configuration of the client, or matching with user preferences e.g. languages. The selection may also include loading and presenting auxiliary components, for example, accessibility components, closed captioning, subtitle, deaf-mute video, etc. Examples of existing systems using the block-demand streaming model include Move Networks ™, Microsoft Smooth Streaming, and the Apple iPhone ™ streaming protocol.

Typically, each block of multimedia data can be stored on the server as a separate file, and then a protocol, such as HTTP, in combination with the HTTP server software running on the server, is used to request the file as a unit. Typically, metadata files are sent to the client, which can be, for example, an extensible markup language (XML) format or a text format for a playlist or a binary format that describes the features of a multimedia presentation, such as the encodings available (for example, the necessary bandwidth, resolution, encoding settings , type of multimedia, language), commonly referred to as “mappings” in this document, and the way that encodings are divided into blocks. For example, metadata may include a uniform resource locator (URL) for each block. URLs themselves can provide a scheme, for example preceded by the string "http: //" to indicate that the protocol that you want to use to access the documented resource is HTTP. Another example is “ftp: //” to indicate that the protocol to be used is FTP.

In other systems, for example, multimedia blocks can be formed by the server “on the fly” in response to a request from a client that indicates a part of the multimedia presentation at the moment that is requested. For example, in the case of HTTP with the “http: //” scheme, executing a request with this URL produces a response to the request, which contains some specific data in the body of the contents of this response to the request. The network implementation of how to generate this response to the request can be quite different, depending on the implementation of the server serving such requests.

Typically, each block can be decoded independently. For example, in the case of video multimedia, each block can begin with a “search point”. In some coding schemes, the search point is called “Random Access Points” or “RAP”, although not all RAPs can be assigned as a search point. Similarly, in other coding schemes, the search point begins in the “Independent Data Update”, or “IDR” frame, in the case of H.264 video encoding, although not all IDRs can be assigned as a search point. The search point is a position in the video (or other) multimedia where the decoder can start decoding without the need for any data about previous frames or data or samples, which may be the case when the frame or sample that is being decoded was not encoded independently, but, for example, as the difference between the current frame and the previous frame.

The question in such systems may be the ability to start playing a stream, for example decoding and visualizing received audio and video streams using a personal computer and showing the video on a computer screen and playing audio through the built-in speakers, or, as another example, decoding and visualizing the received audio and video streams using a set-top box and demonstration of video on a television display device and playback of audio through a stereo system. The primary task may be to minimize the delay between when the user decides to watch the new content transmitted as a stream and performs an action that expresses this decision, for example, the user clicks on the link in the browser window or on the play button on the remote control, and when the content begins to be displayed on a user’s screen, hereinafter referred to as the “content switching time”. Each of these tasks can be solved using the elements of the improved system described in this document.

An example of content switching is when a user watches the first content transmitted by the first stream, and then the user decides to watch the second content transmitted by the second stream, and initiates an action to start viewing the second content. The second stream may be sent from the same set or another set of servers as the first stream. Another example of content switching is when a user visits a website and decides to start viewing the first content transmitted through the first stream by clicking on a link in a browser window. Similarly, the user can decide to start playing content not from the beginning, but from some time in the stream. The user instructs his client device to go to a position in time, and the user could assume that the selected time is visualized instantly. Minimizing the switching time of content is important for watching videos in order to provide users with the perception of high-quality quick viewing of content when searching and selecting a wide range of available content.

Recently, it has become an established practice to consider the use of forward error correction (FEC) codes to protect streaming media during transmission. When sent over a packet network, examples of which include the Internet and wireless networks, for example, standardized by groups such as 3GPP, 3GPP2 and DVB, the original stream is placed in packets when it is formed, or provided, and accordingly packets can be used to carry the original stream or a content stream in the order in which it is generated or transmitted to receivers.

In a typical application of FEC codes to these types of scripts, the encoder can use the FEC code to form recovery packets, which are then sent in addition to the original packets containing the source stream. Recovery packages have the property that when a source package is lost, received recovery packages can be used to recover data contained in the lost source packages. Recovery packages can be used to recover the contents of lost source packages that are completely lost, but can also be used to recover when a partial loss of packages occurs, or fully accepted recovery packages, or even partially accepted recovery packages. Thus, fully or partially received recovery packages can be used to recover completely or partially lost source packages.

In yet other examples, other types of distortion may occur in the sent data, for example, bit values may be inverted, and accordingly, recovery packets may be used to correct such distortion and to provide as accurate recovery of the original packets as possible. In other examples, the original stream is not necessarily sent in discrete packets, but instead can be sent, for example, as a continuous stream of binary signals.

There are many examples of FEC codes that can be used to provide protection for the source stream. Reed-Solomon codes are well-known codes for error correction and erasure correction in communication systems. For correction with erasure, for example, in packet data networks, the well-known effective implementation of Reed-Solomon codes uses Cauchy or Vandermonde matrices, which are described in L. Rizzo, “Effective Erasure Codes for Reliable Computer Communication Protocols”, Computer Communication Review, 27 (2 ): 24-36 (April 1997) (hereinafter “Rizzo”), and Bloemer et al., “An XOR-Based Erasure-Resilient Coding Scheme”, Technical Report TR-95-48, International Institute of Computing Engineering, Berkeley, California (1995) (hereinafter XOR-Reed-Solomon) or elsewhere.

Other examples of FEC codes include LDPC codes, chain reaction codes, such as those described in Luby I, and multi-stage chain reaction codes, such as Shokrollahi I.

Examples of the FEC decoding process for Reed-Solomon code varieties are described in Rizzo and XOR-Reed-Solomon. In those examples, decoding can be applied after a sufficient number of source data and recovery data packets have been received. The decoding process can have a large amount of computation, and, depending on the available CPU resources, it may take considerable time to complete relative to the length of time the multimedia is occupied in the block. The receiver can take this length of time necessary for decoding into account when calculating the delay required between the start of receiving the multimedia stream and playing the multimedia. This delay due to decoding is perceived by the user as the delay between the request of a particular multimedia stream and the start of playback. Accordingly, it is desirable to minimize this delay.

In many applications, packets can be further subdivided into characters to which the FEC process applies. A packet may contain one or more characters (or less than one character, but usually the characters are not divided between groups of packets, as long as the error conditions among the groups of packets are known as highly interrelated). A character can be of any size, but often the size of the character is no more than the size of the packet. The source characters are those characters that encode the data to be transmitted. Recovery symbols are symbols formed from the original symbols, directly or indirectly in addition to the original symbols (i.e., the data to be transferred can be completely restored if all the original symbols are present and there are no restoration symbols).

Some FEC codes may be block in which the encoding operations depend on the character (s) that are in the block and may not depend on the characters outside that block. Using block coding, the FEC encoder can generate recovery characters for a block from the source characters in that block, then move on to the next block and not need to access the source characters besides the characters for the current encoded block. Upon transmission, the source block containing the source characters can be represented as a coded block containing encoded characters (which may be some source characters, some recovery characters, or both). If recovery characters are present, not all source characters are needed in each coded block.

For some FEC codes, in particular Reed-Solomon codes, the encoding and decoding times may impractically increase with the number of encoded characters per source block. Thus, in practice, there is often a practical upper bound (255 is an approximate practical limit for some applications) for the total number of coded symbols that can be generated per source block, especially in the typical case when the Reed-Solomon encoding or decoding process is performed by special hardware, for example, MPE-FEC processes that use Reed-Solomon codes, included as part of the DVB-H standard to protect streams from packet loss, are implemented in cialized hardware in a cell phone, which is limited to only 255 encoded Reed-Solomon characters per source block. Since characters often need to be placed in separate packet payloads, this sets the practical upper bound to the maximum length of the encoded source block. For example, if the payload of a packet is limited to 1024 bytes or less, and each packet carries one encoded character, then the encoded source block can be no more than 255 kilobytes, and this, of course, is also the upper bound on the size of the source block itself.

Other issues, such as the ability to decode source blocks fast enough to keep up with the source streaming speed, minimize the decoding delay introduced by FEC decoding, and use only a small portion of the available CPU in the receiver at any time during FEC decoding, are solved using the elements described in this document.

The need to provide a sustainable and scalable streaming solution that allows system components to fail without adversely affecting the quality of streams transmitted to receivers.

A block-based streaming system should support changes to the structure or presentation metadata, for example, changing the number of multimedia encodings available, or changing multimedia encoding parameters, such as data rate, resolution, aspect ratio, audio or video codecs or codec parameters, or changes other metadata, such as URLs associated with content files. Such changes may be necessary for a number of reasons, including editing content at the same time from different sources, for example advertising or different segments of a larger view, modification of URLs or other parameters that become necessary as a result of changes in the service infrastructure, for example, due to configuration changes hardware failures or recovery from hardware failures, or for other reasons.

There are ways in which a presentation can be controlled using a constantly updated playlist file. Since this file is constantly updated, at least some of the changes described above can be made as part of these updates. The disadvantage of the traditional method is that client devices must always search for a playlist file, which is called a “polling”, creating a load on the serving infrastructure, and that this file cannot be cached for longer than the update interval, making the task for the serving infrastructure even more difficult. This is solved using the elements in this document, so that updates of the type described above are provided without the need for constant polling of the metadata file by clients.

Another problem, especially in live broadcast services, usually known from broadcast distribution, is the inability of the user to view content that was broadcast earlier than the time when the user joined the program. Usually, local personal recording consumes unnecessary local storage or is not possible, because the client has not tuned in to the program, or is prohibited by the content protection rules. Network recording and lazy viewing are preferable, but require separate user connections to the server and a separate transfer protocol and infrastructure in addition to the live broadcast services, resulting in a duplicated infrastructure and significant server costs. This is also solved using the elements described in this document.

SYSTEM OVERVIEW

One embodiment of the invention is described with reference to FIG. 1, which shows a simplified diagram of a block-upon-stream streaming system implementing the invention.

In FIG. 1 illustrates a block streaming system 100 containing a serving block infrastructure 101 (“BSI”), which in turn contains a capture system 103 for capturing content 102, preparing this content, and packaging it for a service from the HTTP streaming server 104 by storing it in storage 110 content that is available to the capture system 103 and the HTTP streaming server 104. As shown, system 100 may also include an HTTP cache 106. In operation, a client 108, such as an HTTP streaming client, sends requests 112 to the HTTP streaming server 104 and receives responses 114 from the HTTP streaming server 104 or from the HTTP cache 106. In each case, the elements shown in FIG. 1 can be implemented at least in part in software comprising program code that runs on a processor or other electronics.

The content may include movies, audio, 2D flat video, 3D video, other types of video, images, time-aligned text, time-aligned metadata, or the like. Some content may include data that needs to be presented or consumed concurrently in time, for example, data to represent supporting information (station identifier, advertisement, stock quotes, Flash ™ sequences, etc.) along with other playable multimedia. Other hybrid representations may also be used that combine other multimedia and / or go beyond just audio and video.

As illustrated in FIG. 2, the multimedia blocks may be stored in the serving block infrastructure 101 (1), which may be, for example, an HTTP server, a content network device, an HTTP intermediary, an intermediary or an FTP server, or some other media server or system. The serving block infrastructure 101 (1) is connected to a network 122, which may be, for example, an Internet Protocol (“IP”) network, such as the Internet. The client of the streaming system upon request of the blocks is shown with six functional components, namely, a block selector 123, which is supplied with the metadata described above and performs the function of selecting the blocks or partial blocks to be requested from among the many available blocks indicated by the metadata, the block interrogator 124 , which receives request instructions from the selector 123 blocks and performs the operations necessary to send a request to a given block, parts of a block or several blocks to the serving infrastructure ure 101 (1) blocks on the network 122 and for receiving data containing the block in response, as well as a block buffer 125, a buffer tracking block 126, a media decoder 127, and one or more media converters 128 that facilitate multimedia consumption.

The block data received by the block interrogator 124 is transmitted for temporary storage to a block buffer 125 that stores multimedia data. Alternatively, the received block data may be stored directly in the block buffer 125, as illustrated in FIG. 1. The multimedia decoder 127 is provided with multimedia data using a block buffer 125 and performs such transformations on this data as is necessary to provide suitable input to the multimedia converters 128 that render the multimedia in a form suitable for a user or other consumption. Examples of multimedia converters include visual display devices that can be found on mobile phones, computer systems, or televisions, and may also include sound reproducing devices, such as speakers or headphones.

An example of a multimedia decoder is a function that converts data in the format described in the H.264 video encoding standard to analog or digital displays of video frames, for example, a YUV-format pixel map with associated presentation timestamps for each frame or sample.

The buffer monitor 126 receives information regarding the contents of the block buffer 125, and based on this information and possibly other information provides input to the block selector 123, which is used to determine the selection of blocks for the request, as described herein.

In the terminology used in this document, each unit has a “play time” or “duration”, which is the amount of time that it would take for the receiver to play the multimedia included in the unit at normal speed. In some cases, multimedia playback in a block may depend on whether data from previous blocks has already been received. In rare cases, the playback of some multimedia in a block may depend on the subsequent block, in which case the playback time for the block is set relative to the multimedia that can be played in the block without reference to the subsequent block, and the playback time for the subsequent block is increased by the multimedia playback time in this a block that can only be played after receiving a subsequent block. Since the inclusion of multimedia in a block that depends on subsequent blocks is a rare case, in the remainder of this disclosure, we assume that multimedia in one block does not depend on subsequent blocks, but we note that those skilled in the art recognize that this variation can easily be added to the embodiments described below.

The receiver may have controls, such as “pause”, “fast forward,” “rewind,” etc., which can lead to block consumption when playing at different speeds, but if the receiver can receive and decode each subsequent sequence of blocks at an aggregated time less than or equal to their aggregated playback time with the exception of the last block in the sequence, the receiver can present multimedia to the user without stopping. In some descriptions in this document, a specific position in a multimedia stream is called a specific “time” in multimedia, corresponding to the time that would elapse between the start of multimedia playback and the time when a specific position in the video stream is reached. Time or position in a multimedia stream is a traditional concept. For example, where the video stream contains 24 frames per second, we can say about the first frame that it has a position or time t = 0.0 seconds, and about the 241st frame we can say that it has a position or time t = 10 , 0 seconds. Naturally, in a frame-based video stream, the position or time should not be continuous, since each of the bits in the stream from the first bit of the 241st frame to exactly before the first bit of the 242nd frame can all have the same time value.

Accepting the above terminology, a block-request streaming system (BRSS) contains one or more clients that generate requests to one or more content servers (eg, HTTP servers, FTP servers, etc.). A capture system comprises one or more capture processors, the capture processor receiving content (in real time or not in real time), processes the content for use by BRSS, and stores it in a storage accessible to the content servers, if possible also together with metadata formed by the capture processor.

BRSS may also contain content caches that are coordinated with content servers. Content servers and content caches can be traditional HTTP servers and HTTP caches, which accept file or segment requests as HTTP requests, which include URLs, and can also include a byte range to request not the entire file or segment specified with using the URL. Clients can include a traditional HTTP client that requests HTTP servers and processes responses to those requests where the HTTP client is controlled by a new client system that formulates the requests, passes them to the HTTP client, receives responses from the HTTP client and processes them (or saves, converts them etc.) to issue them to the presentation player for playback by the client device. Usually, the client system does not know in advance what kind of multimedia is needed (since the needs may depend on user input, changes in user input, etc.), therefore they say that this is a “streaming” system in which multimedia is “consumed”, as only it is accepted, or soon after. As a result, response delays and bandwidth limitations can cause delays in the presentation, for example, causing the presentation to pause when the stream catches up to where the user consumes the presentation.

To provide a presentation that is perceived to be of good quality, a number of details can be implemented in BRSS on the client side, on the capture side, or on both sides. In some cases, the details that are being implemented are performed taking into account and to consider the client-server interface on the network. In some embodiments, the client and capture systems are aware of the improvement, while in other embodiments, only one side is aware of the improvement. In such cases, the whole system benefits from improvement, even if one side is not aware of it, although in other cases the benefit arises only if both parties are aware of it, and when one side is not aware, it still works without failure.

As illustrated in FIG. 3, the capture system may be implemented as a combination of hardware and software components in accordance with various embodiments. A capture system may contain a set of instructions that may be executed to cause the system to follow any one or more of the methodologies discussed in this document. The system can be implemented as a special machine in the form of a computer. The system may be a server computer, a personal computer (PC), or any system capable of executing a set of instructions (sequentially or otherwise) that indicate the actions the system should take. In addition, although only a single system is illustrated, the term “system” should also be used as including any set of systems that individually or collectively carry out a set (or several sets) of instructions for performing any one or more of the methodologies discussed in this document.

The capture system may include a capture processor 302 (eg, a central processing unit (CPU)), a memory 304 that can store program code at runtime, and a disk storage device 306, all of which communicate with each other via bus 300. The system may further include a display device 308 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The system may also include an alphanumeric input device 310 (eg, a keyboard) and a network interface device 312 for receiving a content source and transmitting to a content store.

Disk storage device 306 may include a computer-readable storage medium that can store one or more sets of instructions (eg, software) embodying any one or more methodologies or functions described herein. The instructions may also be located, in whole or at least partially, in the memory 304 and / or in the capture processor 302 during their execution by the system, the memory 304 and the capture processor 302 also constituting computer-readable storage media.

As illustrated in FIG. 4, the client system may be implemented as a combination of hardware and software components in accordance with various embodiments. The client system may contain a set of instructions that can be executed to prompt the system to follow any one or more of the methodologies discussed in this document. The system can be implemented as a special machine in the form of a computer. The system may be a server computer, a personal computer (PC), or any system capable of executing a set of instructions (sequentially or otherwise) that indicate the actions to be taken by the system. In addition, although only a single system is illustrated, the term “system” should also be used as including any set of systems that individually or collectively carry out a set (or several sets) of instructions for performing any one or more of the methodologies discussed in this document.

The client system may include a client processor 402 (eg, a central processing unit (CPU)), a memory 404 that can store program code at runtime, and a disk storage device 406, all of which communicate with each other via bus 400. The system may further include a display device 408 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The system may also include an alphanumeric input device 410 (e.g., a keyboard) and a network interface device 412 for sending requests and receiving responses.

Disk storage device 406 may include a computer-readable medium that can store one or more sets of instructions (eg, software) embodying any one or more methodologies or functions described herein. The instructions may also be located, in whole or at least partially, in the memory 404 and / or in the client processor 402 during their execution by the system, the memory 404 and the client processor 402 also constituting computer-readable media.

USING 3GPP FILE FORMAT

The 3GPP file format or any other file based on the basic ISO media file format, for example, the MP4 file format or the 3GPP2 file format, can be used as a container format for HTTP streaming with the following features. A segment index can be included in each segment to signal time offsets and byte ranges, so that the client can download the appropriate chunks of files or multimedia segments if necessary. The time distribution of the global representation of the entire multimedia presentation and the local time distribution in each 3GP file or multimedia segment can be precisely aligned. Tracks in a single 3GP file or multimedia segment can be precisely aligned. Tracks between displays can also be aligned by assigning each of them a global timeline, so that switching between displays can be smooth, and the joint presentation of multimedia components in different displays can be synchronous.

The file format may contain a profile for Adaptive Streaming with the following properties. All movie data may be contained in movie fragments - the moov block may not contain any sample information. Audio and Video sample data can be alternated, with similar requirements for the progressive download profile defined in TS26.244. The moov block can be placed at the beginning of the file, followed by fragment offset data, also called segment index, containing offset information in time and byte ranges for each fragment or at least a subset of fragments in the containing segment.

The media presentation description can also be referenced to files that follow the existing progressive download profile. In this case, the client can use the description of the multimedia view simply to select a suitable alternative version from among several available versions. Clients can also use partial HTTP GET requests with files compatible with the progressive download profile to request subsets of each alternative version and thereby implement a less efficient form of adaptive streaming. In this case, different displays containing multimedia in the progressive download profile can still adhere to a common global timeline to enable smooth switching between displays.

REVIEW OF IMPROVED WAYS

The following sections describe methods for advanced block-demand streaming systems. It should be understood that some of these improvements can be used together or without other of these improvements, depending on the needs of the application. In general, the receiver asks the server or another transmitter for specific blocks or parts of data blocks. Files, also called segments, can contain several blocks and are associated with a single display of a multimedia view.

Preferably, indexing information is generated, also called “segment indexing” or “segment map”, which provides correlation of playback or decoding moments with byte offsets of the corresponding blocks or fragments in the segment. This segment indexing can be included in a segment, usually at the beginning of a segment (at least some part of the segment map is at the beginning), and is often small. A segment index can also be provided in a separate segment or index file. Especially in cases where the segment index is contained in the segment, the receiver can quickly load part or all of this segment map and subsequently use it to display between the time offsets and the corresponding byte positions of the fragments associated with those time offsets in the file.

The receiver can use the byte offset to request data from fragments associated with specific time offsets, without having to download all the data associated with other fragments not associated with the time offsets of interest. Thus, a segment map or indexing a segment can significantly improve the receiver's ability to directly access portions of the segment that are suitable for current time offsets of interest, with benefits including improved content switching times, the ability to quickly switch from one display to another when network conditions change , and reduced loss of network resources when downloading media that does not play on the receiver.

If you are considering switching from one display (in this document called a “display from which to switch”) to another display (in this document called a “display to which switching occurs”), then the segment index can also be used to identify the start time of an arbitrary point access in the display to be switched to identify the amount of data to be requested in the display from which to switch to ensure that n avnoe switch is provided in the sense that the multimedia display at which switching occurs is loaded up to the presentation time, so that the display reproduction, which occurs at switching can smoothly start the random access point.

These blocks represent segments of video multimedia or other multimedia that the requesting receiver needs to generate output for the receiver user. The multimedia receiver may be a client device, for example, when the receiver receives content from a server that transmits the content. Examples include set-top boxes, computers, game consoles, specially equipped televisions, handheld devices, specially equipped mobile phones, or other client receivers.

This document describes many advanced buffer management techniques. For example, a buffer control method allows clients to request the highest quality media blocks that can be continuously received during playback. The variable block size property improves compression efficiency. The ability to have multiple connections for transferring blocks to the requesting device along with limiting the frequency of requests provides improved transmission efficiency. Partially received data blocks may be used to continue the presentation of multimedia. A join can be reused for multiple blocks without having to fix the join at the beginning for a particular set of blocks. Consistency in selecting servers from among several possible servers by several clients is improved, which reduces the frequency of duplicate content on nearby servers and improves the likelihood that the server contains the entire file. Clients can request multimedia blocks based on metadata (for example, multimedia encodings available) that are embedded in URLs for files containing multimedia blocks. The system can provide for calculating and minimizing the amount of buffering time required before content can be started, without subsequent pauses in multimedia playback. The available bandwidth can be shared between several multimedia units, adjusted when the playback time of each unit approaches, so that if necessary, a large portion of the available bandwidth can be allocated to the unit with the closest playback time.

HTTP streaming can use metadata. Presentation level metadata includes, for example, stream lengths, available encodings (data rates, codecs, spatial resolutions, frame rates, language, media types), pointers to stream metadata for each encoding, and content protection (digital management information Rights (DRM)). Stream metadata can be URLs for segment files.

Segment metadata can include a byte range versus time information for queries in a segment and the identification of Random Access Points (RAPs) or other search points where some or all of this information may be part of indexing a segment or segment map.

Streams may contain multiple encodings of the same content. Each encoding can then be segmented, with each segment corresponding to a storage unit or file. In the case of HTTP, a segment is usually a resource that can be accessed by URL, and requesting such a URL will return the segment as the body of the content of the response message to the request. Segments can contain several groups of pictures (GoP). Each GoP may additionally contain several fragments, and segment indexing provides information on the time / byte offset for each fragment, i.e. the unit of indexing is a fragment.

Fragments or parts of fragments can be requested over parallel TCP connections to increase throughput. This can mitigate the problems that arise when sharing connections over a restricted communication line or when connections are lost due to congestion, thereby increasing the overall speed and reliability of the transmission, which can significantly improve the speed and reliability of the content switching time. Bandwidth can be sacrificed for delay by over-querying, but care should be taken to avoid generating queries too far into the future, which may increase the risk of “information hunger”.

Multiple segment requests on the same server can be organized into a pipeline (by executing the following request before the current request completes) to avoid repeated delays in starting TCP. Sequential fragment requests can be configured in a single request.

Some CDNs prefer large files and may initiate background fetching of the entire file from the original server when they first see a range request. However, most CDNs will serve range requests from the cache if data is available. Therefore, it may be useful to attribute some client requests to the entire segment file. These requests can later be canceled if necessary.

Valid switch points can be search points, in particular RAPs, in the target stream. Different implementations are possible, for example, fixed GoP structures or stream-aligned RAPs (based on the beginning of multimedia or based on GoP).

In one embodiment, segments and GoPs can be stream aligned at different speeds. In this embodiment, GoPs may be of variable size and may contain multiple fragments, but fragments are not aligned between streams at different speeds.

In some embodiments, file redundancy may be advantageously applied. In these embodiments, an erase code is applied to each fragment to generate redundant versions of the data. Preferably, the formatting of the source does not change due to the use of FEC, and additional recovery segments, for example, in the form of a dependent display of the initial display, containing FEC recovery data, are generated and provided as an additional step in the capture system. A client that is able to recover a fragment using only the source data for that fragment can request from the servers only the source data for the fragment in the segment. If the servers are unavailable or the connections to the servers are slow, which can be determined either before or after requesting the initial data, then you can request additional recovery data for the fragment from the recovery segment, which reduces the time for reliable transfer of sufficient data to restore the fragment, possibly using FEC decoding to use a combination of the received source data and recovery data to restore the source data of the fragment. In addition, additional recovery data may be requested to make it possible to recover the fragment if the fragment becomes urgent, i.e. its playback time becomes close, which increases the proportion of data for that fragment on the communication line, but more efficient than closing other connections on the communication line to free up bandwidth. It can also reduce the risk of “information hunger” from using parallel connections.

The fragment format can be a stored real-time transport protocol packet stream (RTP) with audio / video synchronization achieved by real-time transmission control protocol (RTCP).

The segment format can also be a stored MPEG-2 packet stream with audio / video synchronization achieved by the internal time distribution of the MPEG-2 TS.

Using signaling and / or blocking to improve streaming performance

A number of properties may or may not be used in a block-demand streaming system to provide improved performance. Performance may relate to the ability to play a presentation without stopping, to receive multimedia data within the limits of bandwidth and / or to do so within the limited processor resources of a client, server and / or capture system. Some of these properties will be described now.

SEGMENT INDEXING

In order to formulate partial GET requests for movie fragments, the client can be informed about the byte offset and the start time when decoding or the presentation time of all multimedia components contained in the fragments within the file or segment, as well as which fragments start or contain Random Access Points ( and therefore suitable for use as a switching point between alternative mappings), this information often referred to as segment indexing or segment map. The decoding start time or presentation time can be expressed directly or can be expressed as deltas relative to the start time.

This temporal and byte offset indexing information may require at least 8 bytes of data per movie fragment. As an example, for a two-hour movie contained in a single file with 500ms movie fragments, this would total about 112 kilobytes of data. Downloading all of this data when starting a view can lead to a significant additional delay in starting. However, the time and byte offset data can be hierarchically encoded so that the client can quickly find a small portion of the time and offset data corresponding to the moment in the view in which it wants to start. Information can also be distributed in the segment, so that some refinement of the segment index can be interleaved with multimedia data.

Note that if the display is time segmented into several segments, the use of this hierarchical coding may not be necessary, since the complete time and offset data for each segment may already be quite small. For example, if the segments are one minute instead of two hours in the above example, then the time-byte offset indexing information is about 1 kilobyte of data, which can usually fit in a single TCP / IP packet.

Different options are possible for adding data about the temporal and byte offset of a fragment to a 3GPP file:

First, a random access block for a movie fragment (“MFRA”) can be used for this purpose. MFRA provides a table that can assist readers in finding random access points in a file using movie fragments. In support of this feature, MFRA, by the way, contains byte offsets of MFRA blocks containing random access points. MFRA can be placed at or near the end of a file, but this is not necessarily the case. By scanning from the end of the file in search of a random access bias block to a movie fragment and using the size information in it, you can determine the location of the beginning of a random access block to a movie fragment. However, putting MFRA at the end for HTTP streaming usually requires at least 3-4 HTTP requests to access the data you need: at least one for the MFRA request from the end of the file, one to get the MFRA and finally one to get the right fragment in file. Therefore, placing at the beginning may be desirable, since then the MFRA can be loaded along with the first multimedia data in a single request. Also, using MFRA for HTTP streaming can be inefficient, since no information is needed in MFRA except time and moof_offset, and specifying offsets instead of lengths may require more bits.

Secondly, an element location block (“ILOC”) may be used. ILOC provides a directory of metadata resources in this or other files by determining the location of the file containing them, their offset in that file, and their length. For example, a system can combine all metadata resources with an external link in a single file, respectively re-adjusting file offsets and file links. However, “ILOC” is intended to set the location of metadata, so it may be difficult for it to coexist with real metadata.

Finally, and perhaps most appropriate, is a description of a new block called the time index block (“TIDX”), specifically designated to provide exact moments or durations of fragments and byte offset in an efficient way. This is described in more detail in the next section. An alternative block with the same functionality could be a segment index block (“SIDX”). In this document, unless otherwise indicated, these two blocks can be interchangeable, since both blocks provide the ability to provide exact moments or durations of fragments and byte offset in an effective way. The difference between TIDX and SIDX is described below. It should be obvious how to change TIDX blocks and SIDX blocks, since both blocks implement a segment index.

SEGMENT INDEXING

A segment has an identified start time and an identified number of bytes. Multiple fragments can be concatenated into a single segment, and clients can issue requests that identify a specific byte range in the segment that corresponds to the desired fragment or subset of the fragment. For example, when HTTP is used as the request protocol, an HTTP range header can be used for this purpose. This approach requires the client to have access to the “segment index” of the segment, which indicates the position of the different fragments in the segment. This “segment index" may be provided as part of the metadata. This approach results in the fact that you need to create and administer a much smaller number of files compared to the approach when each block is stored in a separate file. Managing the generation, transfer and storage of very large numbers of files (which can grow up to many thousands for a 1 hour presentation, say, 1 hour) can be difficult and error prone, and therefore reducing the number of files is an advantage.

If the client knows only the necessary start time of a smaller part of the segment, then he can request the entire file, then read the file from beginning to end to determine a suitable place to start playback. To improve bandwidth utilization, segments can include an index file as metadata, wherein the index file relates the byte ranges of the individual blocks to the time ranges to which the blocks correspond, called segment indexing or segment map. This metadata can be formatted as XML data or can be binary, for example, corresponding to the atom and block structure in the 3GPP file format. Indexing can be simple, in which the time and byte ranges of each block are absolute relative to the beginning of the file, or they can be hierarchical when some blocks are grouped into parent blocks (and those into ancestor blocks, etc.), and a time and byte range for a given block is expressed relative to the time and / or byte range of the parent block of this block.

EXAMPLE STRUCTURE OF THE INDEXING CARD

In one embodiment, the original source data for one display of a multimedia stream may be contained in one or more multimedia files, referred to herein as a “multimedia segment,” where each multimedia segment contains multimedia data used to play a continuous time media segment, for example, 5 minutes of multimedia playback.

FIG. 6 shows an exemplary overall structure of a multimedia segment. In each segment, either at the beginning or across the entire source segment, indexing information may also be present that contains a segment map with a time / byte offset. A segment map with a time / byte offset in one embodiment may be a list of time / byte offset pairs (T (0), B (0)), (T (1), B (1)), ..., (T (i) , B (i)), ..., (T (n), B (n)), where T (i-1) represents the start time in the segment for playing the i-th media fragment relative to the original multimedia start time among all multimedia segments, T (i) represents the end time for the i-th fragment (and, accordingly, the start time for the next fragment), and the byte offset B (i-1) is the corresponding index of the data start byte in this m initial segment where i-th multimedia fragment starts from the beginning of the initial segment, and B (i) is the appropriate index of the final byte of i-th fragment (and accordingly the index of the first byte of the next track). If a segment contains several multimedia components, then T (i) and B (i) can be provided for each component in the segment in an absolute way, or they can be expressed relative to another multimedia component that serves as a reference multimedia component.

In this embodiment, the number of fragments in the source segment is n, where n may vary from segment to segment.

In another embodiment, the time offset in the segment index for each fragment can be determined using the absolute start time of the first fragment and the durations of each fragment. In this case, the segment index can document the start time of the first fragment and the duration of all the fragments that are included in the segment. A segment index can also only document a subset of fragments. In this case, the segment index documents the duration of the sub-segment, which is defined as one or more consecutive fragments ending either at the end of the containing segment or at the beginning of the next sub-segment.

For each fragment, a value can also be present that indicates whether the fragment begins at the search point or whether it contains it, i.e. at a point at which no multimedia after that point depends on any multimedia before that point, and accordingly, multimedia from that fragment can be reproduced independently of previous fragments. Search points are usually points in multimedia, where playback can begin independently of all previous multimedia. FIG. 6 also shows a simple example of a possible segment indexing for a source segment. In that example, the time offset value is expressed in units of milliseconds, and accordingly, the first fragment of this source segment starts 20 seconds after the multimedia starts, and the first fragment has a playback time of 485 milliseconds. The byte offset of the beginning of the first fragment is 0, and the byte offset of the end of the first fragment / beginning of the second fragment is 50.245, and accordingly, the first fragment is 50.245 bytes in size. If the fragment or subsegment does not start from the random access point, but the random access point is contained in the fragment or subsegment, then the decoding or presentation time difference between the start time and the actual RAP time can be set. This provides the client with the opportunity, in the case of switching to this multimedia segment, to know the exact time before the switching from the display will be necessary.

In addition to or instead of simple or hierarchical indexing, dummy indexing and / or hybrid indexing can be used.

Since the duration of the samples for different tracks may not be the same (for example, video samples can be displayed for 33 ms, while the audio sample can last for 80 ms), different tracks in the movie fragment may not start and end exactly the same time i.e. audio can start a little earlier or a little later than the video, and the opposite is true for the previous fragment, to compensate. To avoid ambiguity, the time stamps specified in the time and byte offset data can be set relative to a particular track, and this can be the same track for each display. This will usually be a video track. This allows the client to identify exactly the next video frame when he switches the display.

You can take steps during the presentation to maintain a strict relationship between track timelines and presentation time to ensure smooth playback and audio / video synchronization, despite the aforementioned problem.

FIG. 7 illustrates some examples, for example, a simple index 700 and a hierarchical index 702.

The following are two specific examples of a block that contains a segment map, one called a time index block (‘TIDX’) and the other called (‘SIDX’). The definition corresponds to the block structure in accordance with the basic format of the ISO multimedia file. Other executions for such blocks, in order to define a similar syntax and the same semantics and functionality, should be obvious to the reader.

Time index block

Definition

Block Type: ‘tidx’

Container: File

Mandatory: No

Quantity: Any number from zero or one

The time index block may provide a set of time and byte offset indices that associate some areas of the file with some presentation time intervals. The time index block may include a targettype field that indicates the type of data referenced. For example, a time index block with targettype “moof” provides an index of the multimedia fragments contained in a file in terms of both time and byte offsets. The time index block with the targettype of the time index block can be used to build a hierarchical time index, allowing users of the file to quickly jump to the necessary part of the index.

A segment index may contain, for example, the following syntax:

aligned (8) class TimeIndexBox

extends FullBox (‘frai’) {

unsigned int (32) targettype;

unsigned int (32) time_reference_track_ID;

unsigned int (32) number_of_elements;

unsigned int (64) first_element_offset;

unsigned int (64) first_element_time;

for (i = 1; i <= number_of_elements; i ++)

{

bit (1) random_access_flag;

unsigned int (31) length;

unsigned int (32) deltaT;

}

}

Semantics

targettype: is the data type of the block referenced by this time index block. This can be either the title of a movie fragment (“moof”) or a time index block (“tidx”).

time-reference_track_id: indicates the track with respect to which time offsets in this index are set.

number_of_elements: the number of elements indexed by this time index block.

first_element_offset: The byte offset from the beginning of the file of the first indexed element.

first_element_time: The start time of the first indexed item using the timeline specified in the title block of the media track identified by time_reference_track_id.

random_access_flag: Unit if the start time of the item is a random access point. Zero otherwise.

length: The length of the indexed element in bytes

deltaT: difference in timeline metrics specified in the title block of the media track identified by time_reference_track_id between the start time of this item and the start time of the next item.

SEGMENT INDEX BLOCK

A segment index block (’sidx’) provides a compact index of movie fragments and other segment index blocks in a segment. There are two cyclic constructs in a segment index block. The first cycle documents the first subsegment sample, i.e. a sample in the first fragment of the movie referenced by the second cycle. The second cycle provides a sub-segment index. The container for the ‘sidx’ block is a file or a segment itself.

Syntax

aligned (8) class SegmentIndexBox extends FullBox (‘sidx’, version, 0) {

unsigned int (32) reference_track_ID;

unsigned int (16) track_count;

unsigned int (16) reference_count;

for (i = 1; i <= track_count; i ++)

{

unsigned int (32) track_ID;

if (version == 0)

{

unsigned int (32) decoding_time;

} else

{

unsigned int (64) decoding_time;

}

}

for (i = 1; i <= reference_count; i ++)

{

bit (1) reference_type;

unsigned int (31) reference_offset;

unsigned int (32) subsegment_duration;

bit (1) contains_RAP;

unsigned int (31) RAP_delta_time;

}

}

Semantics:

reference_track_ID provides the track_ID for the reference track.

track_count: the number of tracks indexed in the next cycle (1 or more);

reference_count: the number of elements indexed by the second loop (1 or more);

track_ID: track ID for which a track fragment is included in the first movie fragment identified by this index; exactly one track_ID in this loop is equal to reference_track_ID;

decoding_time: decoding time for the first sample on the track identified by track_ID in the movie fragment referenced by the first element in the second cycle, expressed in the track's timeline (as documented in the timeline field in the title block of the multimedia track);

reference_type: when set to 0, indicates that the link goes to the block of the movie fragment (‘moof’); when set to 1, indicates that the link goes to the segment index block (‘sidx’);

reference_offset: distance in bytes from the first byte following the containing block index block to the first byte of the referenced block;

subsegment_duration: when the link goes to the segment index block, this field carries the sum of the subsegment_duration fields in the second cycle of that block; when the link goes to a movie fragment, this field carries the sum of the durations of the samples in the reference track, in the specified movie fragment and subsequent movie fragments, up to either the first movie fragment, documented by the next input in the loop, or the end of the sub-segment, which will come earlier; duration is expressed in the timeline of the track (as documented in the timeline field in the title block of the multimedia track);

contains_RAP: when the link goes to the movie fragment, this bit can be 1 if the track fragment in that movie fragment for the track with track_ID equal to reference_track_ID contains at least one random access point, otherwise this bit is set to 0; when the link goes to the segment index, then this bit is set to 1 only if any of the links in that segment index has this bit set to 1, and 0 otherwise;

RAP_delta_time: if contains_RAP is 1, provides the presentation (composition) time of a random access point (RAP); reserved with a value of 0 if contains_RAP is 0. The time is expressed as the difference between the decoding time of the first subsegment sample documented by this record and the time of presentation (composition) of the random access point, on a track with track_ID equal to reference_track_ID.

DIFFERENCES BETWEEN TIDX AND SIDX

TIDX and SIDX provide the same functionality with respect to indexing. The first SIDX cycle also provides global time distribution for the first fragment of the film, but the global time distribution with the same success can be contained in the fragment of the film, either absolutely or relative to the reference track.

The second SIDX cycle implements TIDX functionality. In particular, SIDX allows you to have a mixture of targets for a link for each index, called reference_type, while TIDX refers to either only TIDX or only MOOF. Number_of_elements in TIDX corresponds to reference_count in SIDX, time-reference_track_id in TIDX corresponds to reference_track_ID in SIDX, first_element_offset in TIDX corresponds to reference_offset in the first input of the second cycle, first_element_time in TIDX corresponds to decoding_time in reference_track in the first cycle, random_accessX in the first contains in that in SIDX, RAP may not necessarily be placed at the beginning of the fragment, therefore requiring RAP_delta_time, length in TIDX corresponds to reference_offset in SIDX, and ultimately deltaT in TIDX corresponds to subsegment_duration in SIDX. Therefore, the functionality of the two blocks is equivalent.

SETTING UNIT VARIABLE SIZES AND SUB-GOP BLOCKS

For multimedia video, the relationship between the video encoding structure and the block structure for queries may be important. For example, if each block starts with a search point, for example, Random Access Points (“RAPs”), and each block represents an equal time period of the video, then the positioning of at least some search points in the multimedia video is fixed, and the search points will occur at equal intervals as part of video encoding. As is well known to specialists in the field of video coding, the compression efficiency can be improved if the search points are placed in accordance with the relationship between the video frames, and in particular, if they are placed in frames that have little in common with previous frames. This requirement that the blocks represent equal time intervals, accordingly imposes a restriction on video encoding, so that compression can be suboptimal.

It is desirable to allow the video coding system to select the position of the search points in the video representation instead of requiring the search points in fixed positions. Allowing the video coding system to select search points results in improved video compression, and accordingly, higher quality multimedia video can be achieved using a given available bandwidth, resulting in an improved user experience. Modern streaming systems at the request of blocks may require that all blocks be of the same duration (video time), and that each block must start from a search point, and this is accordingly a drawback of existing systems.

Now will be described a new system for streaming on demand blocks, which provides advantages over the above. In one embodiment, the video encoding process of the first version of the video component can be configured to select the positions of the search points in order to optimize the compression efficiency, but with the requirement that there is a maximum duration between the search points. This last requirement really limits the selection of search points using the coding process, and accordingly reduces the compression efficiency. However, the reduction in compression efficiency is small compared to what would have happened if constant fixed positions were required for the search points, provided that the maximum duration between the search points is not very small (for example, about more than a second). In addition, if the maximum duration between the search points is several seconds, then the reduction in compression efficiency compared to the completely free positioning of the search points is usually very small.

In many embodiments, including this embodiment, it may be that some RAPs are not search points, i.e., there may be a frame that is a RAP that is between two consecutive search points that is not selected as a search point, for example, because the RAP is too close in time to the surrounding search points, or because the amount of multimedia data between the search point preceding or following the RAP and this RAP is very small.

The position of the search points in all other versions of the multimedia presentation may be limited to the same as the search points in the first version (for example, with the highest speed of multimedia data). This reduces the compression efficiency for this other version compared to allowing the encoder to freely select the search points.

The use of search points typically requires an independently decoded frame, which usually results in low compression efficiency for that frame. Frames that should not be decoded independently can be encoded with reference to data in other frames, which usually increases the compression efficiency for that frame by an amount that depends on the degree of generality between the frame to be encoded and the reference frames. An effective choice of search point positioning preferably selects a frame that has low similarity with previous frames as the search point, and thereby minimizes the degradation in compression efficiency caused by encoding the frame in a manner that is independently decoded.

However, the level of generality between the frame and the possible reference frames strongly correlates between different displays of the content, since the original content is the same. As a result, limiting the search points in other embodiments to the same positions as the search points in the first embodiment does not provide a significant difference in compression efficiency.

The structure of the search point is preferably used to determine the structure of the block. Preferably, each search point determines the start of a block, and there may be one or more blocks that include data between two consecutive search points. Since the duration between the search points is not fixed for encoding with good compression, not all blocks need to have the same playback duration. In some embodiments, the blocks are aligned between versions of the content — i.e. if there is a block covering a certain group of frames in one version of the content, then there is a block covering the same group of frames in a different version of the content. Blocks for a given version of content do not overlap, and each frame of content is contained in exactly one block of each version.

A useful property that allows the effective use of variable durations between search points and, correspondingly, GoP of variable duration is a segment indexing or segment map, which can be included in the segment or provided to the client in another way, i.e. this is the metadata associated with this segment in this display, which may be provided containing the start time and duration of each presentation block. The client can use this segment indexing data to determine the block in which to start the presentation when the user has requested the start of the presentation at a specific point in the segment. If such metadata is not provided, then the presentation can begin only from the beginning of the content or at a random or approximate point next to the desired point (for example, when choosing a starting block by dividing the requested starting point (in time) by the average length of the block to set the index of the starting block )

In one embodiment, each block may be provided as a separate file. In another embodiment, several consecutive blocks may be aggregated into a single file to form a segment. In this second embodiment, metadata for each version may be provided containing the start time and duration of each block and the byte offset in the file from which the block begins. This metadata may be provided in response to the original request over the protocol, i.e. available separately from a segment or file, or may be contained in the same file or segment as the blocks themselves, for example, at the beginning of the file. As will be clear to those skilled in the art, this metadata can be encoded in compressed form, such as gzip encoding or delta encoding, or in binary form to reduce the network resources needed to transport metadata to the client.

FIG. 6 shows an example of indexing a segment where the blocks are of variable size and where the boundaries of the blocks are partial GoP, i.e. A partial amount of multimedia data between one RAP and the next RAP. In this example, the search points are indicated by the RAP indicator, wherein a RAP indicator value of 1 indicates that the block begins with or contains RAP, or a search point, and where a RAP indicator of 0 indicates that the block does not contain RAP or search points. In this example, the first three blocks, i.e. bytes 0 through 157.033 contain the first GoP, which has a presentation duration of 1.623 seconds, with a presentation time of 20 seconds to 21.623 seconds in content. In this example, the first of the first three blocks contains 0.485 seconds of presentation time and contains the first 50.245 bytes of multimedia data in the segment. In this example, blocks 4, 5, and 6 contain a second GoP, blocks 7 and 8 contain a third GoP, and blocks 9, 10, and 11 contain a fourth GoP. Note that there may be other RAPs in the multimedia data that are not intended as search points, and they accordingly are not signaled as RAPs in the segment map.

Referring again to FIG. 6, if the client or receiver wants to access the content starting at a time offset of approximately 22 seconds in the multimedia view, then the client can first use other information, such as MPD, described in more detail below, to first determine that the corresponding multimedia data is in this segment. The client can download the first part of the segment to obtain indexing of the segment, which in this case is only a few bytes, for example, using an HTTP byte range request. Using segment indexing, the client can determine that the first block that it should load is the first block with a time offset of no more than 22 seconds and starts with RAP, i.e. from the search point. In this example, although block 5 has a temporal offset that is less than 22 seconds, i.e. its time offset is 21.965 seconds, segment indexing indicates that block 5 does not start with RAP, and accordingly, based on segment indexing, the client selects block 4 for loading, since its start time is no more than 22 seconds, i.e. its time offset is 21.623 seconds, and it starts with RAP. Thus, based on segment indexing, the client will make an HTTP range request, starting at byte offset 157.034.

If segment indexing were not provided, then the client would have to download all the previous 157,034 bytes of data before loading this data, resulting in much longer startup time, or channel switching time, and uneconomical loading of data that is not needed. Alternatively, if segment indexing were not provided, then the client could roughly determine where the desired data begins in the segment, but the approximation could be poor, and it may miss the appropriate time, and then it will be necessary to go back, which again increases the start delay.

As a rule, each block includes a part of multimedia data, which together with the previous blocks can be played by a multimedia player. Thus, the block structure and signaling to the client the block structure of the segment indexing contained in the segment or provided to the client in another way can significantly improve the client's ability to provide fast channel switching and smooth playback, despite fluctuations and network failures. Support for variable-duration blocks and blocks that include only parts of GoP, which is allowed by indexing a segment, can greatly improve the perception of streaming. For example, referring again to FIG. 6 and the example described above, when the client wants to start playback from approximately 22 seconds in the view, the client can request data in block 4 using one or more requests and then submit it to the media player as soon as it is available to start playback . Thus, in this example, playback starts as soon as 42.011 bytes of block 4 are received on the client, respectively, providing fast channel switching times. If instead the client needed to request the entire GoP before playback was supposed to begin, then the channel switching time would be longer, since this amounts to 144.211 bytes of data.

In other embodiments, RAPs or search points may also occur in the middle of the block, and segment indexing data may be present that indicates where that RAP or search point is in the block or fragment. In other embodiments, the time offset may be the decoding time of the first frame in the block instead of the time the first frame was presented in the block.

FIG. 8 (a) and (b) illustrate an example of setting variable block sizes in the aligned structure of a search point over multiple versions or mappings; FIG. 8 (a) illustrates setting variable block sizes with aligned search points on a plurality of media stream versions, while FIG. 8 (b) illustrates the definition of variable block sizes with non-aligned search points on multiple versions of a multimedia stream.

The time is shown in seconds, and the blocks and search points of two segments for two displays are shown from left to right in terms of their distribution in time relative to this timeline, and accordingly, the length of each shown block is proportional to the playback time and not proportional to the number of bytes in the block. In this example, indexing a segment for both segments of two mappings would have the same time offsets for the search points, but potentially different numbers of blocks or fragments between the search points, and different byte offsets for the blocks due to different amounts of multimedia data in each block. In this example, if the client wants to switch from display 1 to display 2 during the presentation in approximately 23 seconds, then the client can query up to block 1.2 in the segment for display 1, and start querying the segment for display 2 starting with block 2.2, and accordingly, switching can occur in a view that matches the search point 1.2 in display 1, which is at the same time as the search point 2.2 in display 2.

As should be clear from the above, the described block-based streaming system does not restrict video encoding by placing search points at specific positions in the content, and this mitigates one of the problems of existing systems.

In the embodiments described above, it is organized such that search points are aligned for different displays of the same content representation. However, in many cases, it is preferable to relax this alignment requirement. For example, sometimes it happens that for the formation of mappings, coding tools are used that do not have the ability to form mappings with aligned search points. As another example, a content representation may be encoded into different displays independently, without aligning the search points between different displays. As another example, a display may contain more search points since it has lower speeds and in most cases needs to be switched, or it contains search points to support complex modes, such as fast forward or rewind or fast search. Thus, it is desirable to provide methods that make a block-based streaming system capable of efficiently and smoothly handling non-aligned search points across various displays to represent content.

In this embodiment, the positions of the search points in the mappings may not be aligned. Blocks are formed so that a new block starts at each search point, and accordingly there may be no alignment between blocks of different versions of the view. An example of such a structure with not aligned search points between different mappings is shown in FIG. 8 (b). The time is shown in seconds, and the blocks and search points of two segments for two displays are shown from left to right in terms of their distribution in time relative to this timeline, and accordingly, the length of each shown block is proportional to its playback time and not proportional to the number of bytes in the block. In this example, indexing a segment for both segments of two mappings would have potentially different time offsets for the search points, as well as potentially different numbers of blocks or fragments between the search points, and different byte offsets for the blocks due to different amounts of multimedia data in each block. In this example, if the client wants to switch from display 1 to display 2 during the presentation in approximately 25 seconds, then the client can query up to block 1.3 in the segment for display 1, and start requesting a segment for display 2 starting with block 2.3, and accordingly, switching can occur in a view that matches the search point 2.3 in display 2, which is in the middle of playback of block 1.3 in display 1, and accordingly, part of the media for block 1.2 will not be played (although it may rebovatsya download multimedia data for 1.3 block frames which are not reproduced in the receiver buffer for decoding of other frames block 1.3, reproduced).

In this embodiment, the operation of the block selector 123 can be changed in such a way that whenever it is necessary to select a block from a display that differs from the previously selected version, the last block is selected whose first frame is no later than the frame following the last frame of the last selected block.

This last described embodiment may eliminate the requirement to limit the position of search points in versions other than the first version, and accordingly increase the compression efficiency for these versions, leading to a higher quality presentation for a given available bandwidth, and accordingly to an improved user experience. An additional consideration is that video encoding tools that perform the function of aligning search points across multiple encodings (versions) of content may not be widely available, and therefore the advantage of this last described embodiment is that the currently existing encoding tools can be used video. Another advantage is that coding of different versions of content can occur in parallel without any need for coordination between coding processes for different versions. Another advantage is that additional versions of the content can be encoded and added to the presentation later, without having to provide encoding tools with lists of specific positions of search points.

As a rule, where pictures are encoded as groups of pictures (GoP), the first picture in the sequence may be a search point, but this should not always be so.

OPTIMUM BLOCK SEPARATION

One problematic issue in a block-request streaming system is the interaction between a coded multimedia structure, such as video multimedia, and a block structure used for block requests. As is well known to specialists in the field of video coding, it often happens that the number of bits required for the encoded display of each video frame changes, sometimes significantly, from frame to frame. As a result, the relationship between the amount of received data and the duration of the media encoded by this data may not be direct. In addition, dividing multimedia data into blocks in a streaming system at the request of blocks adds an additional dimension of complexity. In particular, in some systems, multimedia data of a block may not be reproduced until the entire block is received, for example, the configuration of multimedia data in a block or the relationship between multimedia samples in a block using erase codes can lead to this property. As a result of these complex interactions between the block size and the block length and the possible need to accept the entire block before playback starts, it is customary for client systems to choose a conservative approach in which multimedia data is buffered before playback starts. Such buffering leads to a long channel switching time and, accordingly, a poor user experience.

Pakzad describes “block separation methods,” which are new and effective ways to determine how to divide a data stream into adjacent blocks based on the structure of the data stream, and further describes several of the advantages of these methods with respect to the streaming system. An additional embodiment of the invention will now be described for applying Pakzad block separation methods to a block request streaming system. This method may include configuring the multimedia data to be presented in an approximate presentation time, so that the playback time of any given multimedia data element (for example, a video frame or audio sample) differs from that of any neighboring multimedia data element by less than the provided threshold . The multimedia data arranged in this way can be considered a Pakzad data stream, and any of the Pakzad methods applied to this data stream identifies the block boundaries with the data stream. Data between any pair of neighboring block boundaries is considered a “block” in the terminology of the present description, and the methods of the present description are used to provide presentation of multimedia data in a streaming system upon request of the blocks. As will be clear to those skilled in the art upon reading this disclosure, several advantages of the methods disclosed in Pakzad can then be realized in the context of a block-based streaming system.

As described in Pakzad, determining the block structure of a segment including blocks that include partial GoPs or portions larger than GoPs can affect the ability of a client to provide fast channel switching times. Pakzad provides methods that, taking into account the target launch time, provide a block structure and target download speed, which would ensure that if the client started loading the display at any search point and started playback after the target launch time has expired, then playback will continue smoothly when provided that at each moment of time the amount of data that the client has downloaded is at least equal to the target download speed multiplied by the elapsed time from the start of the download. It is advantageous for the client to have access to the target launch time and the target download speed, as this provides the client with a means to determine when to start playing the display at the earliest point in time, and allows the client to continue playing the display, provided that the download satisfies the condition described above. Thus, the method described below provides a means for including the target launch time and the target download speed in the description of the multimedia presentation so that it can be used for the purposes described above.

MULTIMEDIA REPRESENTATION DATA MODEL

FIG. 5 illustrates possible structures of the content store shown in FIG. 1, including segments and multimedia presentation description (“MPD”) files, and segment decoding, time distribution, and other structure in the MPD file. Details of possible implementations of MPD structures or files will now be described. In many examples, MPD is described as a file, but with the same success non-file structures can be used.

As illustrated, the content store 110 stores a plurality of source segments 510, MPD 500, and recovery segments 512. The MPD may comprise period records 501, which in turn may include display records 502 that contain segment information 503, for example, references to initialization segments 504 and multimedia segments 505.

FIG. 9 (a) illustrates an example metadata table 900, while FIG. 9 (b) illustrates an example of how an HTTP streaming client 902 obtains a metadata table 900 and multimedia blocks 904 through a connection to an HTTP streaming server 906.

In the methods described in this document, a “multimedia presentation description” is provided that contains information regarding the displays of the multimedia presentation that are available to the client. Mappings can be alternatives in the sense that the client selects one of different alternatives, or they can be complementary in the sense that the client selects several mappings, each possibly also from a set of alternatives, and present them simultaneously. Mappings can advantageously be assigned to groups, the client being programmed or configured to understand that for mappings in the same group they are alternatives to each other, while mappings from different groups are such that more than one mapping needs to be presented at the same time. In other words, if there is more than one mapping in a group, then the client selects one mapping from that group, one mapping from the next group, etc., to form a presentation.

Information describing the display, mainly may include details of the applied multimedia codecs, including profiles and codec levels that are required to decode the display, video frame rate, video resolution and data rate. A client receiving a description of a multimedia presentation can use this information to determine in advance whether the display is suitable for decoding or presentation. This is advantageous because if differentiating information were only contained in the binary display data, it would be necessary to query the binary data from all the mappings and analyze and retrieve the relevant information in order to find suitability information. These few queries and analysis, in addition to extracting the data, may take some time, which would lead to a long startup time, and therefore poor user experience.

Moreover, the description of the multimedia presentation may contain information restricting client requests based on time of day. For example, for a live broadcast service, a client may be limited to requesting parts of the presentation that are close to the “current broadcast time”. This is an advantage, since it may be desirable for live broadcasting to remove data from the serving infrastructure for content that was broadcast ahead of the current broadcast time by more than the provided threshold. It may be desirable to reuse storage resources in the serving infrastructure. This may also be desirable depending on the type of service offered, for example, in some cases, the presentation can only be provided in the form of a live broadcast due to the specific subscription model of the receiving client devices, while other multimedia presentations can be provided in the form of a live broadcast and on demand , and other representations can be provided only in the form of a live broadcast for the first class of client devices, only on demand for the second class of client devices and in combination either in the form of a live broadcast, or on demand for a third class of client devices. The methods described in the multimedia presentation data model (below) allow you to inform the client about such policies so that the client can avoid generating queries and setting up offers to the user for data that may not be available in the serving infrastructure. Alternatively, for example, the client may provide a notification to the user that this data is not available.

In a further embodiment of the invention, the multimedia segments may be compatible with the basic ISO multimedia file format described in ISO / IEC 14496-12 or derived specifications (for example, the 3GP file format described in 3GPP Technical Description 26.244). The “Using the 3GPP File Format” section (above) describes the new improvements in the basic ISO multimedia file format that allow the efficient use of data structures of this file format in a streaming system upon request of blocks. As described in this reference, information can be provided in the file, allowing a quick and efficient correlation between the time segments of the multimedia presentation and the byte ranges in the file. The multimedia data itself can be structured according to the construction of a movie fragment specified in ISO / IEC 14496-12. This information, providing time and byte offsets, can be structured hierarchically or as a single block of information. This information may be provided at the beginning of the file. Providing this information using efficient encoding, as described in the “Using the 3GPP File Format” section, leads to the client’s ability to quickly retrieve this information, for example using partial HTTP GET requests, if the file download protocol used by the block-based streaming system is HTTP, which leads to a short time to start, search or switch the stream, and therefore to an improved user experience.

The displays in the multimedia presentation are synchronized on the global timeline to ensure smooth switching between displays, which are usually alternatives, and to guarantee the synchronous presentation of two or more displays. Therefore, the time distribution of samples of the contained multimedia in the mappings within the framework of multimedia presentation with adaptive HTTP streaming can refer to a continuous global timeline across several segments.

An encoded multimedia block containing several types of multimedia, such as audio and video, may have different presentation ending times for different types of multimedia. In a streaming system at the request of blocks, such multimedia blocks can be played sequentially so that each type of multimedia is played back continuously, and accordingly, multimedia samples of one type from one block can be played before multimedia samples of another type in the previous block, which is called in this document “ continuous docking of blocks. " Alternatively, such multimedia blocks may be reproduced in such a way that the earliest sample of any type of one block is reproduced after the last sample of any type of the previous block, which is referred to herein as “interrupted block joining”. Continuous docking of blocks may be appropriate when both blocks contain multimedia from the same content element and the same display, encoded in sequence, or in other cases. As a rule, in one display, continuous docking of blocks can be used when docking two blocks. This is beneficial since existing coding can be applied, and segmentation can be performed without the need to align media tracks at block boundaries. This is illustrated in FIG. 10, where the video stream 1000 comprises a block 1202 and other blocks with a RAP, for example, RAP 1204.

DESCRIPTION OF THE PRESENTATION OF MULTIMEDIA

A media view can be thought of as a structured collection of files on an HTTP streaming server. An HTTP streaming client can download enough information to present a streaming service to a user. Alternative mappings may comprise one or more 3GP files or portions of 3GP files corresponding to a 3GPP file format or at least a well-defined set of data structures that can be easily converted from or to a 3GP file.

The presentation of multimedia can be characterized by describing the presentation of multimedia. A media presentation description (MPD) may contain metadata that a client can use to generate suitable file requests, for example, HTTP GET requests, to access data at an appropriate time and provide the user with streaming services. The media presentation description can provide enough information for the HTTP streaming client to select the appropriate 3GPP files and file portions. The units that are signaled to the client as available are called segments.

Among other things, the description of the multimedia presentation may contain elements and attributes as follows.

MediaPresentationDescription Element

An element encapsulating the metadata used by the HTTP Streaming Client to provide the streaming service to the end user. A MediaPresentationDescription element may contain one or more of the following attributes and elements.

Version: The version number for the protocol to ensure extensibility.

PresentationIdentifier: Such information by which a presentation can be uniquely identified among other representations. It may also contain personal fields or names.

UpdateFrequency: The refresh rate of the media presentation description, i.e. how often the client can reload the actual media presentation description. If absent, then the multimedia presentation may be static. Updating the media view may mean that the media view cannot be cached.

MediaPresentationDescriptionURI: URI for dating media presentation descriptions.

Stream: Describes the type of stream or media presentation: video, audio or text. The type of video stream may contain audio and may contain text.

Service: Describes the type of service with additional attributes. Types of services can be in the form of live broadcast and on demand. This can be used to inform the client that search and access after some current time is not allowed.

MaximumClientPreBufferTime: The maximum amount of time that the client can pre-buffer the media stream. This time distribution may distinguish streaming from progressive download if the client is limited to downloading after this maximum pre-buffering time. A value may not be present, indicating that no pre-buffering restrictions may apply.

SafetyGuardIntervalLiveService: Information about the maximum turnaround time of the live broadcast service on the server. Provides an indication to the client which of the information is currently available. This information may be necessary if the client and server are assumed to be UTC and no time synchronization is provided.

TimeShiftBufferDepth: Information on how far back a client can travel in a live broadcast service relative to the current time. By increasing this depth, deferred viewing and capturing services can be enabled without any changes to the provision of the service.

LocalCachingPermitted: This flag indicates whether the HTTP client can cache downloaded data locally after it is played.

LivePresentationInterval: Contains the time intervals during which the presentation may be available by specifying StartTime and EndTime. StartTime indicates the start time of the services, and EndTime indicates the end time of the service. If EndTime is not set, then the end time is not known at this time, and UpdateFrequency can ensure that clients get access to the end time before the actual end time of the service.

OnDemandAvailabilityInterval: The presentation interval indicates the availability of the service on the network. Several presentation intervals may be provided. An HTTP client may not be able to access the service outside of any given time window. By providing OnDemand Interval, additional deferred viewing can be specified. This attribute may also be present for the live broadcast service. If it is present for the live broadcast service, the server can guarantee that the client can access the service as the OnDemand service (on demand) during all the provided availability intervals. Therefore, the LivePresentationInterval cannot overlap with any OnDemandAvailabilityInterval.

MPDFileInfoDynamic: Describes the dynamic generation of files by default in a multimedia view. Other details are given below. Specifying a default MPD level can save unnecessary repetition if the same rules are used for several or all alternative mappings.

MPDCodecDescription: Describes the main default codecs in the media view. Other details are given below. Specifying a default MPD level can save unnecessary repetition if you use the same codecs for several or all displays.

MPDMoveBoxHeaderSizeDoesNotChange: A flag to indicate whether the MoveBox title is resized among individual files in the entire media view. This flag can be used to optimize loading and may only be present in the case of certain segment formats, especially those for which segments contain the moov header.

FileURIPattern: A template used by the Client to generate Request Messages for files in a multimedia view. Different attributes allow the formation of unique URIs for each of the files in the media view. The base URI may be an HTTP URI.

Alternative Representation: Describes a list of mappings.

AlternativeRepresentation element:

An XML element that encapsulates all metadata for a single display. An AlternativeRepresentation element may contain the following attributes and elements.

RepresentationID: The unique ID for this particular alternate display in the media view.

FilesInfoStatic: Provides an explicit list of start times and URIs of all files in one alternative view. Static provision of the file list may provide the advantage of accurately describing the distribution over time of the presentation of multimedia, but it may not be so compact, especially if the alternative display contains many files. Also, file names can have arbitrary names.

FilesInfoDynamic: Provides an implicit way to build a list of start times and URIs of one alternate view. Dynamically providing a list of files can provide the advantage of a more compact view. If only a sequence of initial times is provided, then the advantages of time distribution are also preserved here, but file names should be generated dynamically in FilePatternURI. If only the duration of each segment is provided, then the display is compact and may be suitable for use in live broadcasting services, but file generation may be determined by global time distribution.

APMoveBoxHeaderSizeDoesNotChange: A flag that indicates whether the MoveBox title is resized among individual files in an alternate view. This flag can be used to optimize loading and may only be present in the case of certain segment formats, especially those for which segments contain the moov header.

APCodecDescription: Describes the main file codecs in an alternative view.

Media Description Element

MediaDescription: An element that can encapsulate all of the media metadata contained in this display. In particular, it may contain track information in this alternative representation, as well as the recommended grouping of the tracks, if applicable. The MediaDescription attribute contains the following attributes:

TrackDescription: An XML attribute that encapsulates all of the media metadata contained in this display. The TrackDescription attribute contains the following attributes:

TrackID: Unique ID for the track in an alternative display. Can be used if the track is part of a grouping description.

Bitrate: Track data rate.

TrackCodecDescription: An XML attribute that contains all the attributes of the codec used on this track. The TrackCodecDescription attribute contains the following attributes:

MediaName: An attribute that identifies the type of media. Types of multimedia include “audio,” “video,” “text,” “application,” and “message.”

Codec: CodecType, which includes the profile and level.

LanguageTag: LanguageTag, if applicable.

MaxWidth, MaxHeight: For video The height and width of the contained video in pixels.

SamplingRate: For audio, sample rate

GroupDescription: An attribute that provides a recommendation to the client for a suitable grouping based on various parameters.

GroupType: The type based on which the client can decide how to group the tracks.

The information in the description of the multimedia presentation is mainly used by the HTTP streaming client to execute requests for files / segments or parts thereof at appropriate times, choosing segments from proportional displays that match their capabilities, for example, in terms of access bandwidth, display capabilities, codec capabilities and so on, as well as user preferences, such as language and so on. In addition, since the description of the multimedia presentation describes displays that are time-aligned and correlated with the global timeline, the client can also use the information in the MPD during the current multimedia presentation to initiate appropriate actions to switch between displays to share displays or search in multimedia view.

SEGMENT START TIME ALARM

The display can be divided in time into several segments. A synchronization problem between tracks exists between the last fragment of one segment and the next fragment of the next segment. In addition, another time distribution problem exists when segments of constant duration are used.

Using the same duration for each segment may have the advantage that MPD is both compact and static. However, each segment can still start at a Random Access Point. Thus, either video encoding can be limited to providing random access points at these specific points, or the actual segment durations may not be as accurate as specified in the MPD. It may be desirable that the streaming system does not impose unnecessary restrictions on the video encoding process, and therefore the second option may be preferred.

In particular, if the file duration is set in the MPD in the form of d seconds, then the nth file can start from a random access point at the moment (n-1) d or immediately following.

In this approach, each file may include information regarding the exact start time of the segment in terms of global presentation time. Three possible ways to signal this include:

(1) First, limit the start time of each segment to the exact synchronization specified in the MPD. But then the multimedia encoder may not have the flexibility to place IDR frames and may require special encoding for streaming files.

(2) Secondly, add the exact start time in the MPD for each segment. For the on-demand case, the compactness of the MPD can be reduced. For cases such as live broadcasts, this may require regular updates to the MPD, which may reduce scalability.

(3) Third, add the global time or the exact start time relative to the announced start time of the display or the announced start time of the segment in the MPD to the segment in the sense that the segment contains this information. This can be added to the new block dedicated to adaptive streaming. This block may also include information that is provided by the TIDX or SIDX block. The result of this third approach is that when searching to a specific position near the beginning of one of the segments, the client, based on MPD, can select the next segment after containing the desired search point. A simple answer in this case could be to move the search point forward to the beginning of the extracted segment (i.e., to the next random access point after the search point). Typically, random access points are provided at least every few seconds (and often there is a small coding gain if they are made less frequent), and therefore, in the worst case, the search point can be moved a few seconds later than specified. Alternatively, the client can determine when retrieving the header information for the segment that the requested search point is actually in the previous segment and request that segment instead. This can lead to an accidental increase in the time required to complete the search operation.

LIST OF AVAILABLE SEGMENTS

A multimedia view contains a set of mappings, each of which provides some other coding version for the original multimedia content. Mappings themselves primarily contain information about different display options compared to other options. They also contain an explicit or implicit list of available segments.

Segments can be differentiated into timeless segments containing only metadata and multimedia segments, which mainly contain multimedia data. A multimedia presentation description (“MPD”) primarily identifies and assigns different attributes to each of the segments, either implicitly or explicitly. Attributes predominantly assigned to each segment contain a period during which a segment is available, resources and protocols over which segments are available. In addition, multimedia segments are predominantly assigned attributes such as the start time of the segment in the multimedia view and the duration of the segment in the multimedia view.

Where the multimedia presentation is of type “on demand”, which is mainly indicated by the attribute in the description of the multimedia presentation, for example, OnDemandAvailabilityInterval, the multimedia presentation description usually describes all segments and also provides an indication of when segments are available and when segments are not available. The start times of the segments are mainly expressed relative to the start of the multimedia presentation, so that two clients starting to play the same multimedia presentations, but at different times, can use the same, same multimedia presentation description, and the same multimedia segments. This advantageously increases the ability to cache segments.

Where the multimedia presentation is of the “live broadcast” type, which is mainly indicated by the attribute in the description of the multimedia presentation, for example, the Service attribute, segments containing the multimedia presentation after the actual time of the day are usually not formed or at least not accessible, despite the fact that segments are fully described in MPD. However, by indicating that the multimedia presentation service is of the “live broadcast” type, the client can create a list of available segments along with time distribution attributes for the NOW client’s internal time by the wall clock based on the information contained in the MPD and the download time of the MPD. The server mainly works in the sense that it makes the resource available so that the reference client working with the MPD instance by the NOW wall clock can access the resources.

In particular, the reference client generates a list of available segments along with time distribution attributes for the NOW client’s internal time according to the wall clock time based on the information contained in the MPD and the MPD download time. When ahead of time, the client will use the same MPD and form a new list of available segments, which can be used for continuous playback of multimedia presentations. Therefore, the server can declare segments in MPD before these segments are actually available. This is beneficial because it reduces the frequent updating and loading of MPD.

Suppose that a list of segments, each with a start time of tS, is described either explicitly by a playlist in elements, such as FileInfoStatic, or implicitly using an element such as FileInfoDynamic. An advantageous way to create a list of segments using FileInfoDynamic is described below. Based on this compilation rule, the client has access to a list of URIs for each mapping r, in this document called FileURI (r, i), and the start time tS (r, i) for each segment with index i.

The use of information in MPD to form an available window of time segments can be performed using the following rules.

For a “on demand” type of service, which is mainly indicated by an attribute, for example, Service, if the current wall clock time on the NOW client is in any availability range, mainly expressed by an MPD element, for example, OnDemandAvailabilityInterval, then all the described segments of this view are available on Demand. If the current wall clock time on the NOW client is outside of any availability range, then none of the described segments in this view are available on demand.

For a service of the “live broadcast” type, which is mainly indicated by an attribute, for example, Service, the start time tS (r, i) mainly expresses the time of availability of wall clocks. The availability start time can be displayed as a combination of the time of the live broadcast service of the event and some turnaround time on the server for capturing, encoding and publishing. The time for this process can be set, for example, in the MPD, using the guard interval tG, set, for example, as SafetyGuardIntervalLiveService in the MPD. This would ensure a minimal difference between UTC and the availability of data on the HTTP streaming server. In another embodiment, the MPD explicitly sets the time that a segment exists in the MPD without providing turnaround time as the difference between the live event broadcast time and turnaround time. In the following description, it is assumed that any global times are defined as availability times. A person skilled in the field of live multimedia broadcasting can derive this information from the appropriate information in the description of the multimedia presentation after reading this description.

If the current wall clock time on the NOW client is outside any range of the live presentation interval predominantly expressed by the MPD element, such as the LivePresentationInterval, then none of the described segments of this live view are available. If the current wall clock time on the NOW client is in the interval of the live view, then at least some segments in the described segments of this live view can be available.

The limit of available segments is controlled by the following values:

Wall clock NOW time (as available for customer).

The allowed tTSB lazy buffer depth, for example, specified as TimeShiftBufferDepth in the media view description.

The client in the relative event time tl can only be allowed to request segments with start times tS (r, i) in the interval (NOW-tTSB) and NOW or in such an interval that the end time of the segment with duration d is also turned on, leading to the interval (NOW- tTSB -d) and NOW.

MPD UPDATE

In some embodiments, the server does not know the pointer to the file or segment and the start times of the segments in advance, since, for example, the server’s location will change, or the media presentation includes some advertising from another server, or the duration of the media presentation is unknown, or the server wants hide pointer for next segments.

In such embodiments, the server can only describe segments that are already available or become available shortly after this MPD instance is published. In addition, in some embodiments, the client predominantly consumes multimedia in the vicinity of the multimedia described in the MPD, so that the user perceives the contained multimedia program as close to generating multimedia content as possible. As soon as the client expects it to reach the end of the described media segments in the MPD, it predominantly requests a new instance of MPD to continue to continuously play, waiting for the server to publish a new MPD describing the new media segments. The server primarily generates new MPD instances and updates the MPD so that clients can rely on procedures for continuous updates. The server can adapt its MPD update procedures along with the formation and publication of segments to the procedures of the reference client, which acts in the way that the general client can act.

If a new instance of MPD describes only a short lead, then clients often need to request new instances of MPD. This can lead to scalability issues and unnecessary uplink and downlink traffic due to unnecessary frequent requests.

Therefore, on the one hand, it is appropriate to describe segments as far as possible in the future, without necessarily making them available, and on the other hand, to allow unexpected updates in MPD to express new server locations, allow the insertion of new content, such as advertising, or provide changes to the codec settings.

In addition, in some embodiments, the duration of the multimedia segments may be short, for example in the range of several seconds. The duration of the multimedia segments is advantageously flexible in order to adapt to suitable segment sizes that can be optimized for transmission or caching properties to compensate for end-to-end delay in live broadcast services or other aspects that relate to the storage or transmission of segments, or for other reasons. Especially in cases where the segments are small compared to the duration of the multimedia presentation, a significant amount of multimedia segment resources and start times must be described in the description of the multimedia presentation. As a result, the size of the description of the multimedia presentation can be large, which can adversely affect the loading time of the description of the multimedia presentation, and therefore affect the delay in launching the multimedia presentation, as well as the use of bandwidth on the access communication line. Therefore, it is useful not only to allow the description of the list of multimedia segments using playlists, but also to allow the description using patterns or rules for creating URLs. URL patterns and rules are used synonymously in this description.

In addition, templates can be used primarily to describe segment pointers in cases of live broadcast after the current time. In such cases, MPD updates, in fact, are not needed, since the pointers, as well as the list of segments are described by templates. However, unanticipated events may still occur that require changes to the description of the mappings or segments. Changes to the description of the multimedia presentation in adaptive HTTP streaming may be needed when content from several different sources merges together, for example, when an ad is inserted. Content from different sources may differ in various ways. Another reason during live streaming presentations is that it may be necessary to change the URLs used for content files to allow for control interception in the event of a failure from one original live broadcast server to another.

In some embodiments, it is useful that if the MPD is updated, then the updates in the MPD are made so that the updated MPD is compatible with the previous MPD in the sense that the reference client, and therefore any implemented client, forms an equally functional list of available segments from the updated MPD for any time up to the validity time of the previous MPD, as he would have made from a previous MPD instance. This requirement ensures that (a) customers can immediately start using the new MPD without synchronization with the old MPD, as it is compatible with the old MPD until the update time; and (b) the update time does not need to be synchronized with the time at which the actual change in the MPD occurs. In other words, MPD updates can be announced in advance, and the server can replace the old instance of MPD as soon as new information is available, without having to store different versions of MPD.

Two possibilities may exist for the time distribution of media for updating the MPD for a set of mappings or all mappings. Either (a) the existing global timeline continues throughout the MPD update (referred to in this document as “continuous MPD update”), or (b) the current timeline ends and a new timeline begins with the segment following the change (in this document called "intermittent MPD update").

The difference between these options may be obvious, taking into account that the tracks of the multimedia fragment, and therefore the segment, usually do not start and end at the same time due to the different degree of sampling into tracks. During the normal presentation, samples of one track in a fragment can be played before some samples of another track of the previous fragment, i.e. there is some kind of overlap between fragments, although overlap in a single track may not be present.

The difference between (a) and (b) is whether this overlap can be applied across the entire MPD update. When the MPD update occurs due to the docking of completely separate content, such overlap is usually difficult to achieve, since the new content needs new coding to dock with the previous content. Therefore, it is useful to enable intermittent updates to the multimedia presentation by resuming the timeline for some segments, and if possible also to specify a new set of displays after the update. Also, if the content is independently encoded and segmented, adjusting the timestamps to fit the global timeline of the previous portion of the content is also avoided.

When an update occurs for less significant reasons, for example, only adding new multimedia segments to the list of described multimedia segments, or if the location of the URL changes, then overlap and continuous updates can be enabled.

In the case of intermittent MPD updates, the timeline of the last segment in the previous display ends at the last time the presentation of any sample in the segment ended. The timeline of the next display (or, more precisely, the first time the first multimedia segment is presented, the new part of the multimedia presentation, also called the new period) usually and mainly starts at the same moment as the end of the last period presentation, so as to guarantee smooth and continuous playback.

Two cases are illustrated in FIG. eleven.

It is preferable and advantageous to limit MPD updates to segment boundaries. The rationale for restricting such changes or updates to segment boundaries is as follows. First, changes in binary metadata for each display, typically in the movie title, can occur at least at the boundaries of segments. Secondly, the description of the multimedia presentation may contain pointers (URLs) to segments. In a sense, MPD is an umbrella data structure that groups together all the segment files associated with a multimedia presentation. To preserve this inclusion relation, a single MPD can refer to each segment, and when the MPD is updated, it is mainly updated only at the segment boundary.

The boundaries of the segments usually do not need to be aligned, however, for the case of content docked from different sources, and for interrupted MPD updates in general, it makes sense to align the boundaries of the segments (in particular, so that the last segment of each display can end in the same video frame and not contain audio samples with the start time of the presentation later than the presentation time of that frame). An intermittent update can then start a new set of mappings at a common point in time, called a period. The start time of the validity of this new set of mappings is provided, for example, using the start time of the period. The relative start time of each display is reset to zero, and the start time of the period puts the set of displays in this new period on the global media presentation timeline.

Continuous MPD updates do not need to align segment boundaries. Each segment of each alternative display can be controlled by a single description of the multimedia presentation, and accordingly update requests for new instances of the description of the multimedia presentation, usually triggered by the expectation that no additional multimedia segments are described in the current MPD, may occur at different times depending on the consumed set of displays, including a set of displays that are expected to be consumed.

In order to support updates to MPD elements and attributes in a more general case, any elements, not just mappings or a set of mappings, can be associated with a validity time. Therefore, if some MPD elements need to be updated, for example, when the number of mappings changes or the rules for creating URLs are changed, then these elements can be updated individually at specified times by providing multiple copies of the element with non-overlapping validity times.

Reality is mainly associated with global multimedia time, so the described element associated with the reality time is valid in the period of the global multimedia presentation timeline.

As discussed above, in one embodiment, the validity times are only added to the full set of mappings. Each complete set then forms a period. The time of reality then forms the time of the beginning of the period. In other words, in the particular case of using the reality element, the complete set of mappings can be valid for a period of time indicated by the global reality time for the set of mappings. The validity time of a set of mappings is called a period. At the beginning of a new period, the validity of the previous set of mappings expires, and the new set of mappings is valid. Again, note that times of validity of periods are preferably not intersected.

As noted above, changes to the description of the multimedia presentation occur at the boundaries of the segments, and therefore, for each display, the change of the element actually occurs at the next boundary of the segment. The client can then generate a valid MPD including a list of segments for each point in time within the media presentation time.

Interrupted docking of blocks may be suitable in cases where the blocks contain multimedia data from different displays, or from different content, for example, from a segment of content and advertising, or in other cases. In a streaming system at the request of blocks, it may be necessary that changes in the presentation metadata occur only at the boundaries of the blocks. This can be beneficial for implementation reasons, because updating the parameters of a multimedia decoder in a block may be more difficult than updating them only between blocks. In this case, it can be predominantly specified that the intervals of reality described above can be interpreted as approximate, so that the element is considered valid from the first border of the block not earlier than the beginning of the specified interval of reality to the first border of the block not earlier than the end of the specified interval of reality.

An exemplary embodiment of the foregoing, which describes new improvements to the block-based streaming system, is described in the section below entitled, “Changes in Multimedia Views”.

SEGMENT DURATION ALARM

Intermittent updates effectively divide the view into a series of disjoint intervals, called a period. Each period has its own timeline for distributing multimedia samples over time. The time distribution of multimedia in a display within a period can be indicated mainly by specifying a separate compact list of segment durations for each period or for each display in a period.

An attribute, for example called a period start time, associated with items in the MPD, can set the validity time of some items within the media presentation time. This attribute can be added to any MPD elements (attributes to which reality can be assigned can change to elements).

For interrupted MPD updates, segments of all mappings may end at the break point. This usually at least implies that the last segment before the break point has a different duration from previous segments. A segment duration alarm may include an indication that either all segments have the same duration, or an indication of a separate duration for each segment. It may be desirable to have a compact display for the list of segment durations, which is effective when many of them have the same duration.

The durations of each segment in one display or set of displays can advantageously be transmitted in a single line, which sets all segment durations for a single interval from the beginning of an interrupted update, i.e. the beginning of the period, until the last multimedia segment described in the MPD. In one embodiment, the format of this element is a text string corresponding to a product that contains a list with records of segment durations, each record containing a duration attribute dur and an optional multiplier mult attribute indicating that this mapping contains <mult> segments of the first record with duration < dur> of the first record, then <mult> of segments of the second record with a duration of <dur> of the second record, etc.

Each duration entry specifies the duration of one or more segments. If the value <dur> is accompanied by the symbol "*" and a number, then this number sets the number of consecutive segments with this duration, in seconds. If the multiplier sign “*” is absent, then the number of segments is equal to one. If “*” is present without a subsequent number, then all subsequent segments have a specified duration, and there can be no additional entries in the list. For example, the string “30 *” means that all segments are 30 seconds long. The line "30 * 12 10.5" indicates 12 segments with a duration of 30 seconds, followed by a segment of 10.5 seconds.

If the segment durations are set separately for each alternative display, then the sum of the segment durations in each interval may be the same for each display. In the case of video tracks, the interval may end on the same frame in each alternate display.

Those skilled in the art, upon reading this disclosure of the invention, may find similar and equivalent methods for expressing segment durations in a compact manner.

In another embodiment, the segment duration is signaled constant for all segments in the display except the last using the <duration> duration signaling attribute. The duration of the last segment before the interrupted update may be shorter, provided that the starting point of the next interrupted update or the beginning of a new period is provided, which then implies the duration of the last segment reaching the beginning of the next period.

CHANGES AND UPDATES TO DISPLAY METADATA

Indication of changes to binary encoded display metadata, such as changes to the title of the movie “moov”, can be done in different ways: (a) there can be one moov block for the entire display in a separate file mentioned in the MPD, (b) there can be one moov block for each alternative display in a separate file referred to in each alternative display, (c) each segment may contain a moov block and therefore is independent, (d) a moov block may be present for the entire display in one 3GP file along with the MPD.

Note that in case (a) and (b), a single ‘moov’ can mainly be combined with the concept of reality from the above in the sense that more блоков moov ’blocks can be mentioned in the MPD, provided that their reality is disjoint. For example, by defining the boundary of a validity period, ‘moov’ in the old period may expire at the beginning of a new period.

In the case of option (a), a reference to a single moov block may be assigned to a reality element. Multiple View headers may be allowed, but only one may be valid at a time. In another embodiment, the validity time of the entire set of mappings in a period or the entire period, as defined above, can be used as the validity time for these display metadata, typically provided as a moov header.

In the case of option (b), a reference to the moov block of each display may be assigned to a reality element. Multiple display headers may be allowed, but only one may be valid at a time. In another embodiment, the validity time of the entire display or the entire period, as defined above, can be used as the validity time for these display metadata, typically provided as a moov header.

In the case of option (c), no signaling can be added to the MPD, but additional signaling in the multimedia stream can be added to indicate whether the moov block will change for any of the upcoming segments. This is further explained below in relation to the section “Signaling updates in segment metadata”.

SIGNAL UPDATES IN SEGMENT METADATA

To avoid frequent updates to the media presentation description to obtain information about potential updates, it is useful to signal any such updates along with the multimedia segments. An additional element or elements may be provided in the multimedia segments themselves, which may indicate that updated metadata, such as a description of the multimedia presentation, is available and must be accessed within a certain amount of time in order to successfully continue compiling lists of available segments. Moreover, such elements may provide a file identifier, such as a URL, or information that can be used to generate a file identifier for the updated metadata file. The updated metadata file may include metadata identical to those provided in the original metadata file for the modified view to indicate validity intervals along with additional metadata also followed by validity intervals. Such an indication may be provided in the multimedia segments of all available displays for presenting multimedia. A client accessing the streaming system upon request of the blocks, upon detection of such an indication in the multimedia block, can use the file upload protocol or other means to retrieve the updated metadata file. The client through this is provided with information about changes in the description of the multimedia presentation and the time at which they occur or have already occurred. It is preferable that each client requests an updated description of the multimedia presentation only once when such changes occur, instead of “polling” and receiving the file many times for possible updates or changes.

Examples of changes include adding or removing displays, changes in one or more displays, for example, a change in data rate, resolution, aspect ratio, included tracks or codec parameters, and changes in URL design rules, for example, another original advertising server. Some changes can only affect the initialization segment, for example the movie title atom (“moov”) associated with the display, while other changes can affect the description of the multimedia presentation (MPD).

In the case of on-demand content, these changes and their timing may be known in advance and may be signaled in the description of the multimedia presentation.

For live content, changes may not be known until the moment they occur. One solution is to let you dynamically update the description of the multimedia presentation that is available at a specific URL, and require customers to regularly request this MPD to detect changes. This solution has a disadvantage in terms of scalability (load on the original server and cache efficiency). In a scenario with a large number of viewers, caches can accept many MPD requests after the previous version has expired in the cache and before the new version is accepted, and all these requests can be redirected to the original server. The original server may need to constantly process cache requests for each updated version of MPD. Also, updates may not just be time-aligned with changes to the presentation of multimedia.

Since one of the benefits of HTTP Streaming is the ability to use standard web infrastructure and services for scalability, the preferred solution may include only “static” (ie, cached) files and not rely on clients “polling” files to see whether they have changed.

Discussed and proposed solutions for changing metadata updates, including a description of the multimedia presentation, and binary display metadata, such as moov atoms, in the multimedia view in Adaptive Streaming HTTP.

For the case of live content, the points at which the MPD or “moov” may change may not be known when the MPD is being formed. Since you should usually avoid frequent “polling” of MPD to check for updates for bandwidth and scalability reasons, MPD updates can be indicated “in the band” in the segment files themselves, i.e. each media segment may be able to indicate updates. Depending on the format of the segments (a) through (c) above, a different update may be signaled.

Typically, the following indication can advantageously be provided in a signal in a segment: an indicator that the MPD can be updated before requesting the next segment in this display or any next segment that has a start time greater than the start time of the current segment. An update may be announced in advance, indicating that the update should only occur in any segment later than the next. This MPD update can also be used to update binary display metadata, such as movie titles, if the pointer of a multimedia segment changes. Another signal may indicate that at the end of this segment no segments should be requested that are ahead of time.

If the segments are formatted according to the format of the segment (s), i.e. each multimedia segment may contain self-initialization metadata, for example a movie title, another signal may be added indicating that the subsequent segment contains an updated movie title (moov). This mainly allows you to include the movie title in the segment, but the client needs to request the movie title only if the previous segment indicates the update of the movie title or in the case of search or random access when switching displays. In other cases, the client may issue a request for a byte range to a segment that excludes the movie title from the download, thereby saving bandwidth predominantly.

In yet another embodiment, if an indication of MPD Update is signaled, the signal may also comprise a pointer, such as a URL, for an updated description of the multimedia presentation. The updated MPD can describe the presentation before and after the update using attributes of reality, for example, the new and old period in the case of interrupted updates. This can be used primarily to enable delayed viewing, which is further described below, but also mainly allows you to signal MPD update at any time before the changes that it contains take effect. The client can immediately download the new MPD and apply it to the current view.

In a specific implementation, the signaling of any changes to the description of the multimedia presentation, the moov headers, or the end of the presentation may be contained in a block of streaming information that is formatted according to the rules of the segment format using the block structure in the basic format of the ISO multimedia file. This block can provide a separate signal for any of the various updates.

Streaming Information Block

Definition

Block Type: ‘sinf’

Container: No

Mandatory: No

Quantity: Zero or One.

The streaming information block contains information about the streaming presentation of which the file is a part.

Syntax

aligned (8) class StreamingInformationBox

extends FullBox (‘sinf’) {

unsigned int (32) streaming_information_flags;

/// The following are optional fields

string mpd_location

}

Semantics

streaming_information_flags contains a logical OR of zero or more of the following:

0x00000001 movie title update will follow

0x00000002 update of the description of the view

0x00000004 ending presentation

mpd_location is present if and only if the presentation description update flags are set and provide a unified resource locator for the new multimedia presentation description.

Example use case for MPD Updates for Live Broadcast Services

Suppose a service provider wants to provide live coverage of a football event using the enhanced block-stream streaming described in this document. Millions of users may want to access the presentation of this event. A live event is interrupted from time to time by pauses when there is a break or other lull in action during which an ad may be added. As a rule, it is absent, or there is a small preliminary warning about the exact distribution of pauses in time.

The service provider may need to provide redundant infrastructure (for example, encoders and servers) to ensure smooth switching if any components fail during a live event.

Suppose that user Anna accesses the service on the bus using her mobile device, and the service is immediately available. Next to her is another user, Paul, who is watching the event on a laptop computer. They score a goal, and both celebrate this event at the same time. Paul tells Anna that the first goal in the game was even more exciting, and Anna uses the service so that she can watch the event 30 minutes earlier in time. Upon seeing the goal, she returns to the live event.

To respond to this use case, the service provider must be able to update the MPD, signal to the customers that an updated MPD is available, and allow customers to access the streaming service so that it can present data close to real time.

Updating the MPD is feasible asynchronously to segment transfer, as explained elsewhere in this document. The server can provide guarantees to the receiver that the MPD has not been updated for some time. The server can rely on the current MPD. However, no explicit signaling is needed when the MPD is updated before a certain minimum update period.

Fully synchronized playback is hardly achievable, since the client can work with different instances of the MPD update, and therefore clients can have a shift. Using MPD updates, the server can report changes, and clients can be notified of changes, even during submission. Segment-based in-band signaling can be used to indicate MPD updates, so updates can be limited to segment boundaries, but this should be acceptable in most applications.

An MPD element can be added that provides MPD publication time by wall clock time, as well as an optional MPD update unit, which is added at the beginning of segments to signal that an MPD update is required. Updates can be performed hierarchically, as is the case with MPD.

The “Publish Time” of the MPD provides a unique identifier for the MPD and when the MPD was released. It also provides binding for update procedures.

The MPD update block can be found in MPD after the “styp” block, with the specified block type = “mupe”, which does not require a container, is optional and has a quantity of zero or one. The MPD updater contains information about the media presentation of which the segment is a part.

An example syntax is as follows:

aligned (8) class MPDUpdateBox

extends FullBox (‘mupe’) {

unsigned int (3) mpd information flags;

unsigned int (l) new-location flag;

unsigned int (28) latest_mpd_update time;

/// The following are optional fields

string mpd_location

}

The semantics of various objects of the MPDUpdateBox class may look like this:

mpd_information_flags: boolean OR zero or more of the following:

0x00 update multimedia presentation description now

0x01 update of media presentation description in front

0x02 ending presentation

0x03-0x07 reserved

new_location_flag: if set to 1, then a new description of the media view is available in the new location specified by mpd_location.

latest_mpd_update time: sets the time (in ms) by which the MPD needs to be updated, relative to the MPD release time of the last MPD. The client can choose to upgrade MPD at any time between the present.

mpd_location: is present if and only if new_location_flag is set, and if so, mpd_location provides a unified resource locator for the new media presentation description.

If the bandwidth used by the updates is a problem, then the server may offer MPD for some features of the device, so only those parts are updated.

Delayed viewing and network PVR

When deferred viewing is supported, it may happen that two or more MPDs or movie titles are valid for the duration of the session. In this case, by updating the MPD if necessary, but with the addition of the concept of a mechanism or a validity period, a valid MPD can exist for the entire time window. This means that the server can ensure that any MPD and Movie title are declared over any period of time that falls within the valid time window for deferred viewing. The client must ensure that the available MPD and metadata for its current submission time is valid. It can also support the transition of a live broadcast session to a network PVR session using only minor MPD updates.

SPECIAL MULTIMEDIA SEGMENTS

When the ISO / IEC 14496-12 file format is used in a block-demand streaming system, the problem is, as described above, which may be advantageous, to store multimedia data for a single version of the presentation in several files configured in sequential time segments. In addition, it may be useful to configure so that each file starts at a random access point. Additionally, it can be advantageous to select the positions of the search points during the video encoding process and segment the presentation into several files, each starting with a search point, based on the choice of search points that was made during the encoding process, with each random access point being placed or Do not fit at the beginning of the file, but each file starts at a random access point. In one embodiment with the properties described above, presentation metadata, or a description of a multimedia presentation, may contain the exact duration of each file, the duration being perceived to mean, for example, the difference between the start time of the video multimedia file and the start time of the multimedia video of the next file. Based on this information in the presentation metadata, the client is able to build a correlation between the global timeline for representing multimedia and the local timeline for multimedia in each file.

In another embodiment, the size of the presentation metadata can advantageously be reduced by specifying instead that each file or segment has the same duration. However, in this case, and where multimedia files are generated in accordance with the method described above, the duration of each file may not be exactly the duration specified in the description of the multimedia presentation, because the random access point may not exist at the point located at exactly the specified duration from the beginning of the file.

Now will be described an additional embodiment of the invention to ensure the correct operation of the streaming system upon request of the blocks, despite the aforementioned discrepancy. In this method, an element can be provided in each file that defines the correlation of the local multimedia timeline in the file (by which we mean the timeline starting from the zero time stamp, with respect to which the time stamps of decoding and composition of multimedia samples in the file are set in accordance with ISO / IEC 14496-12) with a global presentation timeline. This mapping information may contain a single global timestamp of the presentation that corresponds to a zero timestamp on the local timeline of the file. Alternatively, the correlation information may include an offset value that specifies the difference between the global presentation time corresponding to a zero time stamp on the local timeline of the file and the global presentation time corresponding to the beginning of the file in accordance with the information provided in the presentation metadata.

An example of such blocks may be, for example, a track fragment decoding time block (‘tfdt’) or a track fragment adjustment block (‘tfad’) together with a multimedia track fragment control block (‘tfma’).

An example client, including the formation of a list of segments

An example client will now be described. It can be used as a reference client for the server to ensure proper formation and updating of MPD.

The HTTP streaming client is controlled by the information provided in the MPD. It is assumed that the client has access to the MPD, which he accepted at time T, i.e. the moment when he could successfully accept MPD. The successful reception determination may include a client receiving an updated MPD, or a client verifying that the MPD has not been updated since a previous successful reception.

The behavior of the sample client appears. To provide the user with a continuous streaming service, the client first analyzes the MPD and generates a list of available segments for each display for the local time of the client at the current system time, taking into account the list of segments that are described in detail below, using playlists or URL creation rules, if possible. The client then selects one or more mappings based on the information in the display attributes and other information, for example, the available bandwidth and capabilities of the client. Depending on the grouping, the mappings may be shown autonomously or in conjunction with other mappings.

For each display, the client receives binary metadata, for example, a “moov” header to display, if present, and multimedia segments of the selected displays. The client accesses the multimedia content by requesting segments or byte ranges of segments, possibly using a list of segments. The client may first buffer the media before starting the presentation, and as soon as the presentation has begun, the client continues to consume the multimedia content, constantly requesting segments or parts of segments, taking into account the MPD update procedures.

The client can switch displays, taking into account the information of the updated MPD and / or the updated information from its environment, for example, changing the available bandwidth. With any multimedia segment request containing a random access point, the client can switch to another display. When moving forward, i.e. ahead of the current system time (called “NOW time” to represent the time relative to the view), the client consumes the available segments. Each time ahead of NOW time, the client whenever possible expands the list of available segments for each display in accordance with the procedures specified in this document.

If the end of the multimedia presentation has not yet been reached, and if the current playback time falls within the threshold for which the client expects the end of the multimedia in the multimedia described in the MPD for any consumed display or display that will be consumed, then the client may request an MPD update with a new time reading the reception time T. Once it is accepted, the client whenever possible takes into account the updated MPD and the new time T when forming lists of available segments. FIG. 29 illustrates a procedure for live streaming services at various times on a client.

FORMING A LIST OF AVAILABLE SEGMENTS

Assume that the HTTP streaming client has access to MPD and might want to list the available segments for the NOW wall clock. The client synchronizes with reference to global time with some accuracy, but mainly does not require direct synchronization with the HTTP streaming server.

The list of available segments for each display is preferably set in the form of a list of pairs from the start time of the segment and the segment pointer, wherein the start time of the segment can be set relative to the start of the display without loss of generality. The beginning of the display can be aligned with the beginning of the period, if this concept is applied. Otherwise, the start of the display may be at the beginning of the multimedia view.

The client uses the rules for creating URLs and time distribution, which, for example, are further defined in this document. Once a list of the described segments is obtained, this list is further limited to the available segments, which may be a subset of the segments of the full multimedia presentation. The creation is controlled by the current NOW wall clock time on the client. As a rule, segments are only available for any NOW time in the set of moments of availability. For NOW moments, no segments are available outside this window. In addition, for live streaming services, suppose that checktime provides information about how far multimedia is described in the future. Checktime is set on the media time axis documented in MPD; when the client’s playing time reaches checktime, it predominantly requests a new MPD.

Then, the list of segments is further limited to checktime along with the MPD attribute TimeShiftBufferDepth, so that the available multimedia segments are only those for which the sum of the start time of the multimedia segment and the start time of the display falls in the interval between NOW minus timeShiftBufferDepth minus the duration of the last segment described, and shorter value from either checktime or NOW.

SCALABLE BLOCKS

Sometimes the available bandwidth is reduced so low that the block or blocks currently being received at the receiver are unlikely to be fully received on time for playback without temporarily stopping the presentation. The receiver can detect such situations in advance. For example, a receiver may determine that it receives blocks by encoding 5 units of media for every 6 units of time, and contains a buffer of 4 units of media, so that the receiver may assume that it needs to stop the presentation, or pause, after about 24 units of time. With sufficient notification, the receiver can respond to this situation, for example, by rejecting the current flow of blocks and starting to request a block or blocks from another content display, for example, one that uses less bandwidth per unit playback time. For example, if the receiver switches to display where blocks are encoded for at least 20% longer video time for the same block size, then the receiver may eliminate the need to stop until the bandwidth situation improves.

However, it may not be economical to force the receiver to completely discard data already received from the abandoned display. In the embodiment of the block streaming system described in this document, the data in each block can be encoded and configured so that some data prefixes in the block can be used to continue the presentation without the received remainder of the block. For example, well-known scalable video coding techniques may be used. Examples of such video encoding methods include H.264 Scalable Video Encoding (SVC) or H.264 Advanced Video Signal Encoding (AVC) time scalability. Advantageously, this method allows the presentation to continue based on the portion of the block that is received even when the reception of the block or blocks could be left, for example, due to changes in the available bandwidth. Another advantage is that a single data file can be used as a source for several different content displays. This is possible, for example, by using partial HTTP GET requests that select a subset of the block that matches the desired mapping.

One improvement detailed in this document is an improved segment, a scalable segment map. A scalable segment map contains locations of different levels in a segment, so that the client can access parts of the segments accordingly and retrieve the level. In another embodiment, the multimedia data in the segment is ordered so that the quality of the segment increases as the data downloads gradually from the beginning of the segment. In another embodiment, a gradual increase in quality is applied to each block or fragment enclosed in the segment, so fragment requests can be performed to implement a scalable approach.

FIG. 12 is a drawing showing an aspect of scalable blocks. In this figure, transmitter 1200 outputs metadata 1202, scalable layer 1 (1204), scalable layer 2 (1206), and scalable layer 3 (1208), the latter being delayed. Then, receiver 1210 may use metadata 1202, scalable layer 1 (1204), and scalable layer 2 (1206) to display multimedia presentation 1212.

INDEPENDENT LEVELS OF SCALABILITY

As explained above, it is undesirable for the streaming system upon request of the blocks to stop when the receiver is not able to receive the requested blocks of a certain display of multimedia data in time for playback, as this often leads to poor user experience. Stops can be avoided, reduced, or mitigated by limiting the data rate of the mappings selected much lower than the available bandwidth, so that it is unlikely that any given part of the presentation did not arrive on time, but this strategy has the disadvantage that the quality of multimedia is necessarily much lower than in principle can be provided by the available bandwidth. A presentation of lower quality than possible can also be interpreted as a poor user experience. Thus, the developer of a streaming system at the request of blocks is faced with a choice when executing client procedures, programming a client or configuring hardware, or requesting a version of the content that has a much lower data rate than the available bandwidth, in which case the user may suffer from poor quality multimedia, or request a version of the content that has a data transfer rate close to the available bandwidth, and in this case Tea user may suffer from a high probability of pauses during a presentation when changing existing available bandwidth.

To cope with such situations, the block streaming systems described in this document can be configured to independently handle multiple levels of scalability, so that the receiver can execute multi-level requests, and the transmitter can respond to multi-level requests.

In such embodiments, the encoded multimedia data for each block may be divided into several disjoint fragments, referred to in this document as “layers”, so that the combination of layers contains all the multimedia data for the block, and so that a client that has received some subsets of the layers may perform decoding and presentation of content display. In this approach, the ordering of the data in the stream is such that adjacent ranges are increasing in quality, and metadata reflects this.

An example of a technique that can be used to form layers with the aforementioned property is a scalable video encoding technique, for example, which is described in the ITU-T H.264 / SVC standards. Another example of a technique that can be used to form layers with the aforementioned property is the temporal scalability layer technique that is provided in the ITU-T H.264 / AVC standard.

In these embodiments, metadata can be provided in the MPD or in the segment itself, which allows you to generate queries regarding individual levels of any given block and / or combinations of levels and / or a given level from several blocks and / or a combination of levels of several blocks. For example, levels containing a block may be stored in a single file, and metadata specifying byte ranges in the file corresponding to individual levels may be provided.

A file upload protocol that allows the specification of byte ranges, such as HTTP 1.1, can be used to request individual levels or several levels. In addition, as will be appreciated by one of ordinary skill in the art when considering this disclosure, the techniques described above with respect to generating, requesting, and loading variable sized blocks and variable block combinations can be applied with the same success in this context.

COMBINATIONS

A number of embodiments will now be described that may be used predominantly by a streaming client upon request of blocks in order to achieve improved user experience and / or reduced capacity requirements of the serving infrastructure compared to existing techniques by using multimedia data divided into levels, as described above.

In the first embodiment, well-known techniques in a block-based streaming system can be applied with a modification in that different versions of the content are in some cases replaced by different combinations of levels. In other words, where the existing system can provide two separate content displays, the improved system described here can provide two levels, where one content display in the existing system is similar in transmission speed, quality and possibly other indicators to the first level in the improved system, and the second the display of content in the existing system is similar in terms of transmission speed, quality, and possibly in other indicators, a combination of two levels of an improved system. As a result, the storage capacity needed in an improved system is reduced compared to what is needed in an existing system. In addition, while clients of an existing system may issue requests for blocks of one display or another display, clients of an improved system may issue requests for either the first or both block levels. As a result, the perception of the user in the two systems is similar. In addition, improved caching is provided, since even for different qualities shared segments are used, which are then cached with great credibility.

In a second embodiment, a client in an improved block request streaming system using the layer method described herein may maintain a separate data buffer for each of several media coding layers. As will be clear to those skilled in the art of managing data in client devices, these “separate” buffers can be implemented by allocating physically or logically separate memory areas to individual buffers or using other techniques in which buffered data is stored in a single or multiple memory areas, and separation of data from different levels is achieved logically by using data structures that contain links to locations in the data warehouse from individual levels, and therefore ther term "individual buffers" should be understood to include any method by which the individual data levels can be identified separately. The client issues requests for individual levels of each block based on the occupancy of each buffer, for example, the levels can be ordered in priority order, so that data request from one level may not be issued if the occupancy of any buffer for the lower level in priority order is lower than the threshold quantities for that lower level. In this method, priority is given to receiving data from the lower levels in priority order, so if the available bandwidth falls below the upper levels necessary to receive also in priority order, only the lower levels are requested. In addition, the threshold values associated with different levels may differ, so that, for example, lower levels have higher threshold values. If the available bandwidth is changed so that the data for the upper level cannot be received before the playing time of the block, then the data for the lower levels will be accepted already, and therefore the presentation can continue using one lower level. Thresholds for buffer occupancy can be specified in terms of data bytes, duration of playback of data contained in the buffer, number of blocks, or any other suitable measure.

In the third embodiment, the methods of the first and second embodiments can be combined so that multiple multimedia displays comprising a subset of levels are provided (as in the first embodiment), and so the second embodiment is applied to a subset of levels in a display.

In the fourth embodiment, the methods of the first, second and / or third embodiments may be combined with an embodiment in which several independent displays of content are provided, so that, for example, at least one of the independent displays contains several levels to which the techniques from the first , second and / or third embodiments.

ADVANCED BUFFER CONTROLLER

In combination with the buffer tracking unit 126 (see FIG. 2), an advanced buffer manager can be used to optimize the client-side buffer. On-demand streaming systems want to ensure that multimedia playback can start quickly and continue smoothly while maximizing multimedia quality to the user or destination device. This may require the client to request blocks that have the highest quality multimedia, but which can also quickly start and be taken on time to play without causing pauses in the presentation.

In embodiments that use the advanced buffer manager, the manager determines which media blocks to request and when to make these requests. An advanced buffer manager could, for example, be provided with a set of metadata for the content to be presented, which metadata includes a list of displays available for the content and metadata for each display. The display metadata may contain information about the data transfer rate in the display and other parameters, for example video codecs, audio or other codecs and codec parameters, video resolution, decoding complexity, audio language and any other parameters that may affect the choice of display on the client.

The metadata for mapping may also contain identifiers for the blocks onto which the mapping is segmented, these identifiers providing the information the client needs to request the block. For example, when the request protocol is HTTP, the identifier may be an HTTP URL, if possible together with additional information identifying a byte range or time period in a file identified by URL, and this byte range or time period identifying a specific block in a file identified by URL .

In a specific implementation, the advanced buffer manager determines when the receiver makes a request for new blocks, and can handle sending requests. In a new aspect, the advanced buffer manager requests new blocks in accordance with the value of the balancing ratio, which balances the use of too much bandwidth and the end of multimedia during streaming.

The information received by the buffer tracking unit 126 from the block buffer 125 may include indications of each event when the multimedia data is received, how much is received when the multimedia data playback has started or stopped, and the multimedia playback speed. Based on this information, the buffer tracking unit 126 may calculate a variable representing the current buffer size, B _current . In these examples, B _current represents the amount of media contained in the buffer or buffers of the client or other device, and can be measured in units of time, so that B _current represents the amount of time that would take to play all the media represented by blocks or partial blocks stored in the buffer or buffers if no additional blocks or partial blocks were received. Thus, B _current represents the “playback time” at the normal playback speed of multimedia data available on the client but not yet played.

Over time, the value of B _current will decrease, as the media is played, and may increase each time new data is received for the block. Note that for the purposes of this explanation, it is assumed that a block is received when all the data of that block is available in the block interrogator 124, but other means can be used instead, for example, to take into account the reception of partial blocks. In practice, block reception can occur over a period of time.

FIG. 13 illustrates a change in the value of B _current with time when multimedia is played and blocks are received. As shown in FIG. 13, the value of B _current is zero for moments earlier than t ₀ , indicating that no data has been received. At time t ₀ , the first block is received, and the value of B _{current is} increased to equal the playing time of the received block. At the same time, playback has not yet begun, and therefore the value of B _current remains constant until t ₁ , at which the second block arrives, and B _current increases by the size of this second block. At the same time, playback starts, and the value of B _current begins to decrease linearly until t ₂ , when the third block arrives.

The progression of B _current continues in this “sawtooth” manner, increasing stepwise each time a block is received (at times t ₂ , t ₃ , t ₄ , t ₅ and t ₆ ), and gradually decreasing when data is played between them. Note that in this example, the reproduction takes place at the normal reproduction speed for the content, and therefore the slope of the curve between receiving blocks is exactly -1, meaning that one second of multimedia data is reproduced for every one second of real time. With frame multimedia reproduced at a given number of frames per second, for example, 24 frames per second, a tilt of -1 will be approximately expressed by small step functions that indicate the playback of each individual data frame, for example, in steps of -1/24 seconds when it is being played every frame.

FIG. 14 shows another example of the development of B _current over time. In that example, the first block arrives at t ₀ , and playback starts immediately. The arrival of blocks and playback continues until t ₃ , at which the value of B _current reaches zero. When this happens, no additional multimedia data is available for playback, causing a pause in the multimedia view. At time t ₄ , the fourth block is received, and playback can resume. This example, therefore, shows the case where the reception of the fourth block occurred later than necessary, leading to a pause in playback and, accordingly, poor user experience. Thus, the goal of an advanced buffer manager and other features is to reduce the likelihood of this event, while ensuring high quality multimedia.

Buffer monitor 126 can also calculate another metric, B _ratio (t), which is the ratio of multimedia received in a given time period to the length of this time period. More precisely, B _ratio (t) is equal to T _received / (T _now -t), where T _received is the amount of multimedia (measured by its playback time), taken in the period from t, some point earlier than the current time, to the current moment T _now .

B _ratio (t) can be used to measure the rate of change of B _current . B _ratio (t) = 0 is the case when no data has been received since t; B _current will be reduced by (T _now -t) from then on, assuming media is playing. B _ratio (t) = 1 is the case when multimedia is taken in the same volume as it is reproduced in time (T _now -t); B _current will have the same value at time T _now , as at time t. B _ratio (t)> 1 is the case when more data is received than is necessary for playback in time (T _now -t); B _current will be increased from moment t to moment T _now .

Buffer monitor 126 additionally calculates a State value that can take a discrete number of values. Buffer monitor 126 is additionally equipped with a function NewState (B _current , B _ratio ), which, taking into account the current value of B _current and B _ratio values for t <T _now , provides a new State value as output. Whenever B _current and B _ratio cause this function to return a value other than the current State value, a new value is assigned to State, and this new State value is indicated to the block selector 123.

The NewState function can be estimated relative to the space of all possible values of the pair (B _current , B _ratio (T _now -T _x )), where T _x can be a fixed (configured) value or can be derived from B _current , for example, using the configuration table, which correlates the values of B _current with the values of T _x , or may depend on the previous value of State. A buffer monitor 126 is provided by one or more partitions of this space, each partition containing sets of disjoint regions, each region being annotated with a State value. The evaluation of the NewState function then includes the operation of identifying the partition and determining the region into which the pair falls (B _current , B _ratio (T _now -T _x )). The return value is then the annotation associated with that area. In the simple case, only one partition is provided. In more complex cases, the partitioning may depend on the pair (B _current , B _ratio (T _now -T _x )) in the previous evaluation time of the NewState function, or on other factors.

In a specific embodiment, the partition described above may be based on a configuration table containing a number of thresholds for B _current and a number of thresholds for B _ratio . In particular, let the threshold values for B _current be B _thresh (0) = 0, B _thresh (1), ..., B _thresh (n ₁ ), B _thresh (n ₁ +1) = ∞, where n ₁ is the number of non-zero thresholds for B _current . Let the threshold values for B _ratio be B _r-thresh (0) = 0, B _r-thresh (1), ..., B _r-thresh (n ₂ ), B _r-thresh (n ₂ +1) = ∞ where n ₂ is the number of thresholds for the B _ratio . These thresholds specify a partition containing a grid of (n ₁ +1) into (n ₂ +1) cells, where the i-th cell of the j-th row corresponds to the region in which B _thresh (i-1) <= B _current <B _thresh (i) and B _r-thresh (j-1) <= B _ratio <B _r-thresh (j). Each cell of the grid described above is annotated with a State value, for example, associated with specific values stored in memory, and then the NewState function returns the State value associated with the cell indicated by the values of B _current and B _ratio (T _now -T _x ).

In a further embodiment, a hysteresis value may be associated with each threshold value. In this improved method, the evaluation of the NewState function may be based on a time partition formed using a set of temporarily changed threshold values, as follows. For each B _{current, a} threshold value that is less than the B _current range corresponding to the selected cell in the last NewState estimate, the threshold value is reduced by subtracting the hysteresis value associated with that threshold value. For each B _{current, a} threshold value that is greater than the B _current range corresponding to the selected cell in the last NewState estimate, the threshold value is increased by adding the hysteresis value associated with that threshold value. For each B _{ratio, a} threshold value that is less than the B _ratio range corresponding to the selected cell in the last NewState estimate, the threshold value is reduced by subtracting the hysteresis value associated with that threshold value. For each B _{ratio, a} threshold value that is greater than the B _ratio range corresponding to the selected cell in the last NewState estimate, the threshold value is increased by adding the hysteresis value associated with that threshold value. Modified threshold values are used to evaluate the NewState value, and then the threshold values return to their original values.

Other methods for defining partitions of space will be apparent to those skilled in the art upon reading this disclosure. For example, a partition can be specified using inequalities based on linear combinations of B _ratio and B _current , for example, threshold values of a linear inequality of the form α1 • B _ratio + α2 • B _current ≤α0 for real-valued α0, α1 and α2 to define half-spaces in the common space , and the assignment of each disjoint set as the intersection of a certain number of such half-spaces.

The above description is illustrative of the main process. As it will be clear to experts in the field of real-time programming when reading this disclosure, effective implementations are possible. For example, each time new information is provided to the buffer tracking unit 126, it is possible to calculate the future time at which the NewState will change to a new value if, for example, no additional data is received for the blocks. Then a timer is set for this time, and in the absence of additional input, the expiration of this timer will cause a new State value to be sent to the block selector 123. As a result, the calculations need to be performed only when new information is transmitted to the buffer tracking unit 126, or when the timer expires, rather than continuously.

Suitable values for State include Low, Sustainable, and Full. An example of a suitable set of threshold values and a resulting grid of cells are shown in FIG. fifteen.

In FIG. 15, the threshold values of B _current are shown on the horizontal axis in milliseconds with the hysteresis values shown below as “+/- value”. The threshold values of B _ratio are shown on the vertical axis in ppm (ie, multiplied by 1000) with the hysteresis values shown below as “+/- value”. State values are annotated in the grid cells as “L”, “S” and “F” for “Low”, “Stable” and “Full”, respectively.

The block selector 123 receives notifications from the block interrogator 124 whenever it is possible to request a new block. As described above, the block selector 123 is provided with information regarding a plurality of available blocks and metadata for those blocks, including, for example, information about the speed of the multimedia data in each block.

Information about the speed of the multimedia data of the block may contain the actual speed of the multimedia data of a particular block (i.e., the size of the block in bytes divided by the playback time in seconds), the average speed of the multimedia display data to which the block belongs, or a measure of the necessary available bandwidth , on a long-term basis, in order to play without pauses the display to which the block belongs, or a combination of the above.

The block selector 123 selects the blocks based on the State value last specified by the buffer tracking block 126. When this State value is Sustainable, the block selector 123 selects a block from the same display as the previous selected block. The selected block is the first block (in playback order) containing multimedia data for a period of time in a view for which no multimedia data has been requested before.

When the State value is Low, the block selector 123 selects a block from the display at a lower media speed than the previously selected block. A number of factors can affect the exact display choice in this case. For example, a block selector 123 may be provided by indicating an aggregated input data rate and may select a display with a multimedia data rate that is less than this value.

When the State value is “Full”, the block selector 123 selects a block from the display at a higher media speed than the previously selected block. A number of factors can affect the exact display choice in this case. For example, a block selector 123 may be provided by indicating an aggregated input data rate and may select a display with a multimedia data rate that does not exceed this value.

A number of additional factors may further affect the operation of the block selector 123. In particular, the frequency with which the speed of multimedia data of the selected unit is increased may be limited even if the buffer tracking unit 126 continues to indicate a “Full” state. In addition, it is possible that the block selector 123 accepts the “Full” status indication, but there are no blocks with a higher available multimedia data rate (for example, because the last selected block was already for the highest available multimedia data rate). In this case, the block selector 123 may delay the selection of the next block by the time selected so that the total amount of multimedia data buffered in the block buffer 125 is limited from above.

Additional factors may influence the set of blocks that are considered during the selection process. For example, the available blocks may be limited to blocks from mappings whose encoding resolution falls within a certain range provided by the block selector 123.

The block selector 123 may also receive input from other components that monitor other aspects of the system, such as the availability of computing resources for multimedia decoding. If such resources become scarce, then the block selector 123 may select blocks whose decoding is indicated in the metadata as having less computational complexity (for example, displays with a lower resolution or frame rate usually have less decoding complexity).

The embodiment described above provides a significant advantage in that the use of the B _ratio value when evaluating the NewState function in the buffer tracking unit 126 allows a faster increase in quality at the beginning of the presentation compared to the method that only B _{current considers} . Without taking into account the B _ratio , a large amount of buffered data can accumulate before the system can select blocks with a higher speed of multimedia data, and, therefore, higher quality. However, when the B _ratio value is large, this indicates that the available bandwidth is much higher than the speed of the multimedia data of previously received blocks, and that even with relatively small buffered data (i.e., a low value for B _current ) it remains safe request blocks with a higher speed of multimedia data, and therefore higher quality. Equally, if the value of B _{ratio is} low (for example, <1), then this indicates that the available bandwidth has dropped below the speed of multimedia data for previously requested blocks, and accordingly, even if B _{current is} large, the system will switch to lower media speed and therefore lower quality, for example, to avoid reaching the point where B _current = 0, and media playback stops. This improved behavior can be especially important in environments where network conditions, and therefore transmission speeds, can change quickly and dynamically, for example, when users stream to mobile devices.

Another advantage is provided by using configuration data to specify a partition of the value space (B _current , B _ratio ). Such configuration data may be transmitted to the buffer tracking unit 126 as part of the presentation metadata, or by other dynamic means. Since in practical applications the behavior of user network connections can be very variable between users and over time for a single user, it can be difficult to predict the partitions that will be applicable to all users. The ability to provide such configuration information to users dynamically allows you to develop good configuration settings over time in accordance with experience.

REQUEST VARIABLE SIZE OF REQUEST

A high request rate may be necessary if each request refers to a single block, and if each block encodes a short multimedia segment. If the multimedia blocks are short, then video playback quickly moves from block to block, which more often allows the receiver to adjust or change the selected data rate by changing the display, increasing the likelihood that playback can continue without stopping. However, the disadvantage of the high frequency of requests is that they can be unstable in some networks in which the available bandwidth is limited in the network from the client to the server, for example, in wireless WAN networks, for example, wireless WAN 3G and 4G, where the data link capacity is The client’s network connectivity is limited or may become limited for short or long periods of time due to changes in radio conditions.

High request frequency also implies a high load on the serving infrastructure, which introduces associated costs in the form of bandwidth requirements. Thus, it is desirable to have some of the benefits of a high query frequency without all the disadvantages.

In some embodiments of a block streaming system, the flexibility of a high request frequency is combined with less frequent requests. In this embodiment, blocks may be formed as described above and aggregated into segments containing several blocks, as described above. At the beginning of the presentation, the processes described above are applied in which each request refers to a single block, or several simultaneous requests are made to request parts of the block to guarantee fast channel switching time and therefore a good user experience at the beginning of the presentation. Subsequently, when a certain condition, which will be described below, is fulfilled, the client may issue requests that include several blocks in a single request. This is possible because blocks are aggregated into larger files or segments and can be requested using byte or time ranges. Sequential byte or time ranges can be aggregated into a single larger byte or time range, resulting in a single request for multiple blocks, and even interrupted blocks can be requested in a single request.

One basic configuration, which may be determined by the decision whether to request a single block (or partial block) or to request several consecutive blocks, has the basis of the configuration for deciding whether the requested blocks are likely to play. For example, if it is likely that there will soon be a need to switch to another display, then it is better for the client to request single blocks, i.e. small amounts of multimedia data. One reason for this is that if you are querying multiple blocks when switching to another display could be unavoidable, switching can be done before the last few query blocks are played. Thus, downloading these last few blocks may delay the transmission of multimedia display data to which switching is performed, which may cause multimedia playback to stop.

However, single block requests result in a higher frequency of requests. On the other hand, if it is unlikely that there will soon be a need to switch to another display, then it may be preferable to generate requests for several blocks, since all of these blocks will probably be played, and this leads to a lower frequency of requests, which can significantly reduce losses requests, especially if it’s typical that there is no close change in the display.

In traditional block combining systems, the quantity requested in each request is not dynamically adjusted, i.e. typically, each request is for the entire file, or each request is for approximately the same size of the display file (sometimes measured in time, sometimes in bytes). Thus, if all requests are smaller, then the loss on requests is high, whereas if all requests are larger, this increases the chances of media stop events, and / or providing lower quality media playback if lower quality displays are selected to avoid the need for fast Change displays when network conditions change.

An example of a condition that, when executed, can cause subsequent requests to access multiple blocks is a buffer size threshold, B _current . If B _{current is} below the threshold value, then each issued request refers to a single block. If B _{current is} greater than or equal to the threshold value, then each issued request refers to several blocks. If a request is issued that refers to several blocks, then the number of requested blocks in each single request can be determined in one of several possible ways. For example, the amount may be constant, for example, two. Alternatively, the number of blocks requested in a single request may depend on the state of the buffer, and in particular, on B _current . For example, you can set a number of thresholds, while the number of blocks requested in a single request is derived from the largest among several thresholds that is less than B _current .

Another example of a condition that, when executed, can cause queries to access multiple blocks, is the variable State value described above. For example, when State is “Stable” or “Full”, then requests may be issued for several blocks, but when State is “Low”, then all requests may be for one block.

Another embodiment is shown in FIG. 16. In this embodiment, when it is necessary to issue the next request (determined in step 1300), the current value of State and B _{current is} used to determine the size of the next request. If the current State value is “Low” or the current State value is “Full” and the current display is not the highest available (determined at 1310, the answer is “Yes”), then the next request is selected short, for example, for the next block exactly (the block is determined and the request is completed at block 1320). The rationale for this is that these are the conditions under which it is likely that switching of mappings will be pretty soon. If the current State value is “Sustainable” or the current State value is “Full” and the current display is the highest available (determined at step 1310, the answer is “No”), then the duration of consecutive blocks requested in the next request is selected proportional to α -blocks B _current for some fixed α <1 (blocks are determined at block 1330, the query is executed at block 1340), for example, for α = 0.4, if B _current = 5 seconds, then the next query may refer to approximately 2 seconds of blocks whereas if B _current = 10 seconds m, the following query can relate to approximately 4 seconds blocks. One logical rationale for this is that under these conditions it could be unlikely that switching to a new display would be performed in an amount of time that is proportional to B _current .

FLEXIBLE CONVEYOR PROCESSING

Block streaming systems can use the file request protocol, which is based on a specific transport protocol, such as TCP / IP. At the beginning of a TCP / IP or other transport protocol connection, it may take some considerable time to reach the full available bandwidth. This can lead to a “loss on connection start” each time a new connection is started. For example, in the case of TCP / IP, a connection startup loss occurs due to both the time taken to initially acknowledge the TCP connection to establish the connection, and the time taken by the congestion tracking protocol to fully utilize the available bandwidth.

In this case, it may be desirable to issue multiple requests using a single connection in order to reduce the frequency with which loss is introduced when the connection starts. However, some file transfer protocols, such as HTTP, do not provide a mechanism for canceling a request, except generally closing the transport layer connection and thereby introducing a loss on connection start when a new connection is established instead of the old one. It may be necessary to cancel the issued request if it is determined that the available bandwidth has changed and a different multimedia data rate is needed instead, i.e. there is a decision to switch to another display. Another reason for canceling a issued request may be if the user requested that the multimedia presentation end and a new presentation begin (possibly the same content item at a different presentation point or possibly a new content item).

As you know, connection startup losses can be avoided by keeping the connection open and reusing the same connection for subsequent requests, and, as is also known, a connection can be fully used if several requests are issued simultaneously on the same connection (a technique known as "pipelining" in the context of HTTP). However, the drawback of issuing several requests at the same time, or in a more general sense, so that several requests are issued before the previous connection requests are completed, may be that the connection is then intended to transmit a response to those requests, and therefore, if they become if changes to which requests should be made are desirable, then the connection may be closed if it becomes necessary to cancel already issued requests that are no longer needed.

The probability that the issued request needs to be canceled may partially depend on the length of the time interval between issuing the request and the playback time of the requested block in the sense that when this time interval is large, it is likely that the issued request needs to be canceled (because it’s likely that the available available bandwidth will change during the interval).

As you know, some file upload protocols have the property that a single underlying transport layer connection can be used primarily for multiple download requests. For example, HTTP has this property because reusing a single connection for multiple requests eliminates the “loss on connection start” described above for requests other than the first. However, the drawback of this approach is that the connection is designed to transport the requested data in each issued request, and therefore, if the request or requests need to be canceled, either the connection can be closed, causing a loss when the connection starts, when a replacement connection is established, or the client may expect to receive data that is no longer needed, introducing a delay in receiving subsequent data.

We will now describe an embodiment that retains the advantages of reusing a compound without introducing this disadvantage, and which also further increases the frequency with which the compounds can be reused.

The embodiments of block streaming systems described in this document are configured to reuse a connection for multiple requests without having to first allocate the connection to a specific set of requests. Essentially, a new request is issued over an existing connection when already issued connection requests are not yet completed, but are close to completion. One reason not to wait for the completion of existing requests is that if previous requests are completed, then the connection speed may deteriorate, i.e. the underlying TCP session may go into an idle state, or the cwnd TCP variable may decrease significantly, thereby significantly reducing the initial download speed of a new request issued on that connection. One reason for waiting almost to completion before issuing an additional request is that if a new request is issued much earlier than the completion of previous requests, then the new issued request may not even start for some substantial period of time, and it may be that during this period of time Before a new issued request begins, the decision to form a new request is no longer valid, for example, because of the decision to switch the display. Thus, an embodiment of clients that implement this technique will issue a new connection request as late as possible without slowing down connection loading capabilities.

The method comprises monitoring the number of bytes received on a connection in response to the last request issued on that connection, and applying a check to that number. This can be accomplished by having a receiver (or transmitter, if applicable) configured for tracking and verification.

If the check passes, then an additional request can be issued on the connection. One example of a suitable check is whether the number of bytes received is a fixed fraction of the size of the requested data. For example, this proportion may be 80%. Another example of a suitable check is based on the following calculation, as illustrated in FIG. 17. In the calculation, let R be an estimate of the data rate of the connection, T be an estimate of the round-trip period (“RTT”), and X be a numerical coefficient, which, for example, could be a constant set to a value between 0.5 and 2, where the estimates of R and T are updated on a regular basis (updated at step 1410). Let S be the size of the data requested in the last request, B be the received number of bytes in the requested data (calculated at step 1420).

A suitable check would be to instruct the receiver (or transmitter, if applicable) to perform the inequality assessment procedure (S-B) <X • R • T (checked at step 1430), and if “Yes”, then perform the action. For example, a check may be conducted to see if there is another request ready to be issued for the connection (checked at step 1440), and if “Yes”, then issue that connection request (step 1450), and if “No”, then the process returns to step 1410 to continue updating and validating. If the result of the check in step 1430 is “No,” then the process returns to step 1410 to continue updating and checking.

Checking the inequality at step 1430 (performed, for example, by appropriately programmed elements) causes each subsequent request to be issued when the amount of remaining data to be received is equal to X times the amount of data that can be received at the current estimated receive rate in one RTT. A number of methods are known in the art for estimating a data rate, R, at step 1410. For example, a data rate may be estimated as Dt / t, where Dt is the number of bits received in the previous t seconds, and where t may be, for example, 1 s or 0.5 s or some other interval. Another method is an exponential weighted average, or a filter with an Infinite Impulse Response (IIR) of the first order at the incoming data rate. A number of methods are known in the art for estimating RTT, T, at 1410.

The check in step 1430 can be applied to the aggregation of all active connections on the interface, as explained in more detail below.

The method further comprises forming a list of potential requests, associating each potential request with a set of suitable servers to which a request can be made, and arranging the list of potential requests in priority order. Some entries in the list of potential queries may have the same priority. Servers in the list of matching servers associated with each potential request are identified by hostnames. Each host name corresponds to a set of Internet Protocol addresses that can be obtained from a domain name system that is well known. Therefore, each possible request in the list of potential requests is associated with a set of Internet protocol addresses, in particular, by combining sets of Internet protocol addresses associated with host names associated with servers associated with a potential request. Whenever the check described in step 1430 passes for the connection and no new request has yet been issued on that connection, the request with the highest priority is selected in the lists of potential requests with which the Internet protocol address is associated with the destination of the connection, and this request is issued by connection. The request is also removed from the list of potential requests.

Potential requests can be deleted (canceled) from the list of potential requests, new requests can be added to the list of candidates with a priority that is higher than already existing requests in the list of candidates, and existing requests in the list of candidates can change their priority. The dynamic nature of the queries that are in the list of potential queries, and the dynamic nature of their priority in the candidate list can change which queries may be issued next, depending on when the type check described in step 1430 passes.

For example, it is possible that if the response to the verification described in step 1430 is “Yes” at some point t, then the next query issued would be query A, whereas if the response to the verification described in step 1430 is not “Yes” until some moment t '> t, then the next query issued would instead be query B, because either query A was removed from the list of potential queries between time t and t', or because query B was added to the list of potential queries with more higher priority than query A, between time t and t ', or because then the request B was on the list of candidates at time t, but with a lower priority than the request A, and between the time t and t 'B priority request made higher than the query A.

FIG. 18 illustrates an example of a query list in a query candidate list. In this example, there are three connections, and there are six queries in the candidate list, indicated by A, B, C, D, E, and F. Each of the queries in the candidate list can be issued on a subset of the connections as indicated, for example, query A can be issued on the connection 1, while request F may be issued on connection 2 or connection 3. The priority of each request is also indicated in FIG. 18, and a lower priority value indicates that the request has a higher priority. Thus, requests A and B with priority 0 are requests of the highest priority, while request F with a priority value of 3 is the lowest priority among requests in the candidate list.

If at this point in time t connection 1 passes the test described in step 1430, then either request A or request B is issued on connection 1. If instead, connection 3 passes the test described in step 1430 at this time t, then request D is issued on connection 3, since the request D is the request with the highest priority that can be issued on connection 3.

Assume that for all connections, the response to the test described in step 1430, from time t to some later point t 'is "No", and between time t and t', request A changes its priority from 0 to 5, request B is deleted from the list of candidates, and a new query G with priority 0 is added to the list of candidates. Then, at time t ′, the new candidate list may be as shown in FIG. 19.

If at time t ′, connection 1 passes the test described in step 1430, then request C with priority 4 is issued on connection 1, since this is the highest priority request in the candidate list that can be issued on connection 1 at a given time.

Assume in the same situation that instead, a request A would be issued for connection 1 at time t (which was one of the two options with the highest priority for connection 1 at time t, as shown in Fig. 18). Since the response to the check described in step 1430, from time t to some later point t ′ is “No” for all connections, connection 1 still transmits data at least until time t ′ for requests issued before time t, and accordingly, request A would not start at least until t '. Issuing a request C at time t 'is a better solution than issuing a request A at time t, because request C starts at the same time after t' when request A started, and since by that time request C has a higher priority than request A.

As another alternative, if the type check described in step 1430 is applied to the aggregation of active connections, then a connection that has a destination whose Internet Protocol address is associated with the first request in the list of potential requests or with another request with the same priority, as mentioned first request.

A number of ways are possible for forming a list of possible queries. For example, a candidate list may contain n queries representing queries of the next n pieces of data of the current display in a time-order view, where a query of the earliest data part has the highest priority and a request of the latest data part has the lowest priority. In some cases, n may be one. The value of n may depend on the size of the buffer B _current , the State variable, or another measure of the state of the busy buffer of the client. For example, you can specify a number of threshold values for B _current and the value associated with each threshold value, and then the value n is taken as the value associated with the largest threshold value that is less than B _current .

The embodiment described above guarantees flexible distribution of requests across connections, ensuring that reuse of an existing connection is preferred even if the highest priority request is not suitable for that connection (because the destination IP address of the connection is not the one allocated to any of the host names associated with request). The dependence of n on B _current or State or another measure of client buffer busyness ensures that such requests are not “out of order of priority” when the client urgently needs to issue and complete a request associated with the next piece of data that needs to be reproduced in a time sequence.

These methods can advantageously be combined with joint HTTP and FEC.

AGREED SERVER SELECTION

As is well known, files for downloading using the file upload protocol are usually identified by an identifier containing the host name and file name. For example, this is the case for the HTTP protocol, in which case the identifier is a Unified Resource Identifier (URI). A host name can correspond to several hosts identified by Internet Protocol addresses. For example, this is a general way to distribute the load of requests from multiple clients across multiple physical machines. In particular, this approach is typically applied by Content Transfer Networks (CDNs). In this case, the request issued on the connection to any of the physical hosts is assumed to have achieved the goal. A number of methods are known by which a client can select from Internet Protocol addresses associated with a host name. For example, these addresses are typically provided to a client through a domain name system and are provided in order of priority. The client can then select the Internet Protocol address with the highest priority (first). However, there is usually no coordination between clients regarding how this choice is made, as a result of which different clients may request the same file from different servers. This can result in the same file being stored in the cache of several nearby servers, which reduces the efficiency of the cache infrastructure.

This can be handled by a system that predominantly increases the likelihood that two clients requesting the same block will request this block from the same server. The new method described in this document contains a choice from among the available Internet protocol addresses in a manner determined by the identifier of the file to be requested, and so that different clients who are presented with the same or similar choices of Internet protocol addresses and file identifiers will make one and the same choice.

A first embodiment of the method is described with reference to FIG. 20. The client first receives a set of Internet Protocol addresses IP ₁ , IP ₂ , ..., IP _n , as shown in step 1710. If there is a file for which requests should be issued, as decided in step 1720, then the client determines the Internet address a protocol for issuing queries regarding a file as determined in steps 1730-1770. Given the set of Internet Protocol addresses and the identifier for the file to be requested, the method comprises arranging the Internet Protocol addresses in a manner determined by the file identifier. For example, for each Internet Protocol address, a byte string is generated containing the concatenation of the Internet Protocol address and the file identifier, as shown in step 1730. A hash function is applied to this byte string, as shown in step 1740, and the resulting values of the hash function are configured in accordance with a fixed ordering, as shown in step 1750, for example, increasing the numbering order, forming an ordering in the addresses of the Internet Protocol. The same hash function can be used by all clients, thereby ensuring that the same result is generated by the hash function over the given input from all clients. The hash function can be statically configured in all clients in the client set, or all clients in the client set can receive a partial or full description of the hash function when clients receive a list of Internet protocol addresses, or all clients in the client set can receive a partial or full description hash functions when clients receive a file identifier, or the hash function may be determined in another way. The Internet Protocol address that is the first in this ordering is selected, and this address is then used to establish a connection and issue requests for all or parts of the file, as shown in steps 1760 and 1770.

The above method can be applied when a new connection is established to request a file. It can also be used when a number of established connections are available, and one of these connections can be selected to issue a new request.

In addition, when an established connection is available and the request can be selected from a set of potential requests with equal priority, the ordering of potential requests is formed, for example, using the same method of hash function values described above, and a potential request is selected that appears first in this streamlining. The methods can be combined to select both a connection and a potential request from a set of connections and requests of equal priority, again by calculating the hash for each combination of connection and request, ordering these hash values in accordance with a fixed ordering, and choosing the combination that occurs first in ordering formed in a set of combinations of queries and connections.

This method is advantageous for the following reason: a typical approach taken by the serving block infrastructure, such as shown in FIG. 1 (BSI 101) or FIG. 2 (BSI 101), and in particular the approach commonly used by CDNs, is to provide multiple caching proxies that accept client requests. The caching proxy server may not be provided with the file requested in the given request, in which case such servers usually redirect the request to another server, receive a response from that server, usually including the requested file, and redirect the response to the client. A caching proxy server can also store (cache) the requested file so that it can respond immediately to subsequent file requests. The general approach described above has the property that the set of files stored on a given caching proxy server is largely determined by the set of requests received by the caching proxy server.

The method described above has the following advantage. If all clients in a set of clients are provided with the same list of Internet protocol addresses, then these clients will use the same Internet protocol address for all requests issued for the same file. If there are two different lists of Internet Protocol addresses, and each client is supplied with one of these two lists, then clients will use no more than two different Internet Protocol addresses for all requests issued for the same file. In general, if the lists of Internet Protocol addresses provided to clients are similar, then clients will use a small set of secured Internet Protocol addresses for all requests issued for the same file. Since immediate clients seek similar Internet Protocol address lists, it is likely that nearby customers only issue file requests from a small fraction of the caching proxies available to those clients. Thus, there will be only a small fraction of the caching proxies that cache the file, which mainly minimizes the amount of caching resources used to cache the file.

Preferably, the hash function has the property that a very small fraction of different input data correlates with the same result, and that different input data correlates with highly random results to ensure that for a given set of Internet protocol addresses the proportion of files for which the specified one of the Internet Protocol addresses is the first in the sorted list using step 1750, is approximately the same for all Internet Protocol addresses in the list. On the other hand, it is important that the hash function is deterministic in the sense that for a given input, the result of the hash function is the same for all clients.

Another advantage of the above method is the following. Suppose all clients in a client set are provided with the same list of Internet protocol addresses. Due to the hash functions just described, it is likely that requests for different files from these clients will be evenly distributed over a set of Internet protocol addresses, which in turn means that requests will be distributed evenly across caching proxies. Thus, the caching resources used to store these files are distributed evenly across the caching proxies, and file requests are distributed evenly across the caching proxies. Thus, the method provides both storage balancing and load balancing throughout the caching infrastructure.

A number of variations to the above approach are known to those skilled in the art, and in many cases these variations retain the property that the set of files stored on a given broker is at least partially determined by the set of requests that the caching proxy server has accepted. In the general case, in which a given hostname is split into several physical caching proxies, it will be common for all of these servers to ultimately store a copy of any given file that is often requested. Such duplication may be undesirable because the storage resources on the caching proxies are limited, and as a result, files may be accidentally deleted from the cache. The new method described here ensures that requests for a given file are sent to caching proxies so that this duplication is reduced, thereby reducing the need to delete files from the cache, and thereby increasing the likelihood that any given file is present in the intermediary’s cache (t .e. not removed from it).

When a file is present in the intermediary’s cache, the response sent to the client passes faster, which has the advantage of reducing the likelihood that the requested file arrives later, which can lead to a pause in multimedia playback and therefore poor user experience. Moreover, when the file is not present in the broker's cache, the request can be sent to another server, causing additional load on both the serving infrastructure and network connections between the servers. In many cases, the server to which the request is sent may be located in a remote location, and transferring a file from that server back to the caching proxy server may result in transmission costs. Therefore, the new method described here leads to a reduction in these transmission costs.

PROBABILITY REQUESTS FOR THE WHOLE FILE

The specific issue when HTTP is used with range requests is the behavior of cache servers, which are typically used to provide scalability in the serving infrastructure. Although it may be common for HTTP cache servers to support an HTTP range header, the exact behavior of different HTTP cache servers varies from implementation. Most cache server implementations serve range requests from the cache when the file is available in the cache. A common implementation of HTTP cache servers always redirects downstream HTTP requests containing the range header to the upstream node until the cache server has a copy of the file (cache server or original server). In some implementations, the upstream response to the range request is the entire file, and this entire file is cached, and the response to the downstream range request is retrieved from this file and sent. However, in at least one implementation, the upstream response to the range request is just the data bytes in the range request itself, and these data bytes are not cached, but instead sent as a response to the downstream range request. As a result, the use of range headers by clients can result in the file itself never being cached and the desired scalability properties of the network being lost.

The operation of caching proxies was described above, and a method for requesting blocks from a file, which is an aggregation of several blocks, was also described. For example, this can be achieved by using the HTTP range request header. Such queries are hereinafter referred to as “partial queries”. An additional embodiment will now be described, which is advantageous when the serving block infrastructure 101 does not fully support the HTTP range header. Typically, servers in a serving block infrastructure, such as a content transfer network, support partial requests, but may not store the response to partial requests in a local storage (cache). Such a server can perform a partial request by forwarding the request to another server until the entire file is stored in local storage, in which case a response can be sent without forwarding the request to another server.

A block-request streaming system that applies the new block aggregation enhancement described above may not work well if the serving block infrastructure exhibits this behavior, since all requests that are partial requests will be redirected to another server and no requests will be served by caching proxies -servers, interfering, first of all, with the goal of providing caching proxy servers. During the streaming process at the request of blocks, which is described above, the client may at some point request a block, which is located at the beginning of the file.

In accordance with the new method described in this document, whenever a certain condition is met, such requests can be converted from requests for the first block in a file to requests for the entire file. When the request for the entire file is accepted by the caching proxy server, the proxy server usually saves the response. Therefore, the use of these requests forces the file to be cached by the local caching proxy servers, so that subsequent requests of either the full file or partial requests can be served directly by the caching proxy server. The condition may be such that among the set of requests associated with the given file, for example, the set of requests generated by the set of clients viewing the content item in question, the condition will be satisfied at least for the secured share of these requests.

An example of a suitable condition is that a randomly selected number is greater than a predetermined threshold. This threshold value can be set so that the conversion of a single block request to the request of the whole file occurs on average for a secured share of requests, for example, once out of ten (in this case, a random number can be selected from the interval [0, 1], and the threshold value maybe 0.9). Another example of a suitable condition is that the hash function calculated from some information associated with the block and some information associated with the client takes one of the provided set of values. This method has the advantage that for a file that is often requested, the file will be cached by the local proxy server, however, the operation of the streaming system upon request of blocks does not change significantly from the standard work, in which each request is intended for a single block. In many cases, when a request is converted from a request for a single block to a request for an entire file, client procedures would otherwise continue to request other blocks in the file. If so, then such requests can be stopped, because the blocks in question will be accepted in any case as a result of requesting the entire file.

Creating a URL, creating a list of segments and searching for it

Creating a list of segments is faced with the problem of how a client can create a list of segments from MPD at the local NOW time of a particular client for a specific display that starts at some starttime starttime, either relative to the beginning of the multimedia presentation for cases on demand, or expressed by the wall clock time. The list of segments can contain a pointer, for example, a URL, to the optional source display metadata, as well as a list of multimedia segments. Each media segment can be assigned starttime, duration, and a pointer. Starttime usually expresses the approximate value of the media time contained in the media segment, but not necessarily the exact sampling time. Starttime is used by the HTTP streaming client to issue a download request at the appropriate time. Creating a list of segments, including the start time of each, can be performed in different ways. URLs can be provided as a playlist, or a URL rule can be used primarily to compactly display a list of segments.

A list of segments based on URL compilation can, for example, be executed if the MPD signals this using a specific attribute or element, for example FileDynamicInfo or an equivalent signal. A generalized way to create a list of segments from URL compilation is given in the section called “URL Compilation Overview” below. Playlist-based compilation may, for example, be signaled by another signal. Searching through a list of segments and getting into the exact time of multimedia is also predominantly implemented in this context.

URL DESIGN OVERVIEW

As described above, in one embodiment of the present invention, a metadata file may be provided that contains URL composition rules that allow client devices to compile file identifiers for presentation units. Now an additional new improvement will be described in the block-based streaming system, which includes changes to the metadata file, including changes to the rules for creating URLs, changes to the number of encodings available, changes to the metadata associated with available encodings, for example data transfer rate, aspect ratio, resolution, audio or video codec or codec parameters, or other parameters.

In this new enhancement, additional data may be provided associated with each element of the metadata file indicating a time interval in the general representation. In this time interval, the element can be considered valid, otherwise the element can be ignored. In addition, the syntax of metadata can be improved so that elements that were previously allowed to appear only once or no more than once can appear several times. In this case, an additional restriction may be applied, which provides that for such elements, the specified time intervals must be disjoint. At any given point in time, consideration only of elements whose time interval contains a given point in time leads to a metadata file that corresponds to the original metadata syntax. We call such time intervals intervals of reality. Therefore, this method provides for signaling in a single metadata file of changes of the type described above. Advantageously, such a method can be used to provide a multimedia presentation that supports changes to the described view at specified points in the presentation.

URL CONSTRUCTOR

As described in this document, a common feature of block-based streaming systems is the need to issue “metadata” to the client that identifies the available multimedia encodings and provides the information the client needs to request blocks from those encodings. For example, in the case of HTTP, this information may contain URLs for files containing multimedia blocks. A playlist file may be provided that lists URLs for blocks for a given encoding. Several playlist files are provided, one for each encoding, together with a main playlist of playlists that lists playlists corresponding to different encodings. The disadvantage of this system is that the metadata can become quite large and therefore take some time to request when the client starts the flow. An additional drawback of this system is obvious in the case of live broadcast content, when files corresponding to multimedia data blocks are generated “on the fly” from a multimedia stream that is captured in real time (live broadcast), for example, a sports live broadcast or a news program. In this case, the playlist files can be updated every time a new block is available (for example, every few seconds). Client devices can repeatedly call a playlist file to determine if new blocks are available and retrieve their URLs. This can create a significant burden on the service infrastructure and, in particular, means that metadata files cannot be cached longer than the update interval, which is equal to the size of the block, which is usually on the order of several seconds.

One important aspect of a block-request streaming system is the method used to inform clients about file identifiers, such as URLs that should be used, along with a file upload protocol, to request blocks. For example, a method in which for each display in a view a playlist file is provided that lists the URLs of files containing blocks of multimedia data. The disadvantage of this method is that at least part of the playlist file itself needs to be downloaded before playback can begin, increasing the channel switching time and therefore causing a poor user experience. For a long presentation of multimedia with multiple or many displays, the list of file URLs can be large, and therefore the playlist file can be large, further increasing the channel switching time.

Another disadvantage of this method occurs in the case of live content. In this case, the full list of URLs does not become available in advance and the playlist file is periodically updated when new blocks become available, and clients periodically request a playlist file to accept the updated version. Because this file is frequently updated, it cannot be stored on caching proxies for a long time. This means that a lot of requests for this file will be redirected to other servers and ultimately to the server that generates the file. In the case of the popular presentation of multimedia, this can lead to a high load on this server and the network, which in turn can lead to a long response time, and therefore to a large time switching channels and poor user experience. In the worst case, the server becomes overloaded, and this leads to the fact that some users can not see the view.

When executing a streaming system upon request of blocks, it is advisable to avoid imposing restrictions on the type of file identifiers that can be used. The reason is that a number of considerations may encourage the use of identifiers of a particular kind. For example, in the case where the serving infrastructure of the blocks is a content transfer network, there may be file or storage naming conventions related to the desire to distribute the storage or maintenance load across the network, or other requirements that lead to specific types of file identifier that cannot be predicted during system design.

An additional embodiment will now be described that mitigates the aforementioned disadvantages while maintaining the flexibility to select suitable file identification conventions. In this method, metadata for each display in a multimedia representation may be provided containing a rule for compiling a file identifier. A rule for compiling a file identifier may contain, for example, a text string. In order to determine a file identifier for a given presentation block, a method may be provided for interpreting a rule for compiling a file identifier, this method including determining input parameters and evaluating a rule for compiling a file identifier, together with input parameters. The input parameters may include, for example, the index of the file to be identified, the first file having an index of zero, the second having an index of one, the third having an index of two, and so on. For example, if each file takes the same length of time (or approximately the same length of time), then you can easily determine the index of the file associated with any given time in the view. Alternatively, the time in the view occupied by each file may be provided in the view or version of the metadata.

In one embodiment, the rule for compiling a file identifier may contain a text string that may contain some special identifiers corresponding to input parameters. A method for evaluating a rule for compiling a file identifier comprises determining the positions of special identifiers in a text string and replacing each such special identifier with a string representation of the value of the corresponding input parameter.

In another embodiment, the rule for compiling the file identifier may comprise a text string corresponding to some expression language. The expression language contains a definition of the syntax that expressions in the language can correspond to, and a set of rules for evaluating the string corresponding to the syntax.

A specific example will now be described with reference to FIG. 21 and subsequent drawings. An example of syntax definition for a suitable expression language defined in the Backus-Naur augmented form is shown in FIG. 21. An example of rules for evaluating a string corresponding to the output rule <expression> in FIG. 21 contains a recursive conversion of a string obeying an output rule <expression> (<expression>) to a string obeying an output rule <literal> as follows:

<expression>, which obeys the <literal> inference rule, remains unchanged.

An <expression> that obeys the <variable> inference rule is replaced by the value of the variable identified by the <token> line in the <variable> inference rule.

An <expression> that obeys the <function> inference rule is evaluated by evaluating each of its arguments in accordance with these rules and applying a transformation to these arguments, depending on the <token> element in the <function> inference rule, as described below.

An <expression> that obeys the last alternative of the <expression> inference rule is evaluated by evaluating the two <expression> elements and applying an operation to these arguments, depending on the <operator> element of the last alternative of the <expression> inference rule, as described below.

The method described above assumes that the assessment occurs in a context in which a plurality of variables can be defined. A variable is a pair (name, value), where “name” is a string obeying the <token> output rule, and “value” is a string obeying the <literal> output rule. Some variables may be set outside the evaluation process before the evaluation begins. Other variables may be set in the evaluation process itself. All variables are “global” in the sense that only one variable exists with every possible “name”.

An example of a function is the printf function. This function takes one or more arguments. The first argument may obey the output rule <string> (hereinafter referred to as the "string"). The printf function evaluates the converted version of the first argument. The conversion used is the same as the “printf” function in the C standard library, with additional arguments included in the <function> output rule, which supplies the additional arguments expected by the printf function in the C standard library.

Another example of a function is the hash function. This function takes two arguments, the first of which can be a string, and the second can obey the output rule <number> (hereinafter “number"). The hash function applies a hash algorithm to the first argument and returns results that are a non-negative integer less than the second argument. An example of a suitable hash function is defined in a C function shown in FIG. 22, whose arguments are the input string (excluding surrounding quotes) and a numeric input value. Other examples of hash functions are well known to those skilled in the art.

Another example of a function is the Subst function, which takes one, two, or three string arguments. In the case where one argument is supplied, the result of the Subst function is the first argument. In the case when two arguments are supplied, the result of the “Subst” function is calculated by removing any occurrences of the second argument (excluding surrounding quotes) in the first argument and returning the first argument changed in this way. In the case when three arguments are supplied, the result of the “Subst” function is calculated by replacing any occurrences of the second argument (excluding surrounding quotes) in the first argument with the third argument (excluding surrounding quotes) and returning the first argument changed in this way.

Some examples of operators are the addition, subtraction, division, multiplication and modulo operators, identified by the output rules <operator> ‘+’, ‘-‘, ‘/’, ‘*’, ‘%’, respectively. These operators require that the inference rules <expression> or the side of the inference rule <operator> be evaluated with numbers. The operator estimate contains the application of a suitable arithmetic operation (addition, subtraction, division, multiplication and modulo division respectively) to these two numbers in the usual way and returning the result in the form corresponding to the output rule <number>.

Another example of an operator is an assignment operator identified by an output rule <operator> ‘=’. This statement requires that the left argument be evaluated by a string whose contents match the <token> output rule. The contents of the string are specified by the string of characters in the surrounding quotation marks. The equality operator causes the assignment of a variable whose name is <token> to be equal to the contents of the left argument, a value equal to the result of evaluating the right argument. This value is also the result of evaluating the operator expression.

Another example of a statement is a sequence statement, identified by the output rule <operator> ‘;’. The result of evaluating this operator is the right argument. Note that, as with all operators, both arguments are evaluated, and the left argument is evaluated first.

In one embodiment of this invention, the file identifier can be obtained by evaluating the rule for compiling the file identifier in accordance with the above rule with a specific set of input variables that identify the desired file. An example of an input variable is a variable with the name “index” and a value equal to the numeric index of the file in the view. Another example of an input variable is a variable with the name “bitrate” and a value equal to the average data rate of the desired version of the view.

FIG. 23 illustrates some examples of the rules for compiling a file identifier, the input variables being “id”, which identifies the identifier to display in the desired representation, and “seq”, which defines the serial number for the file.

As will be clear to those skilled in the art upon reading this disclosure, numerous variations of the above method are possible. For example, not all functions and operators described above may be provided, or additional functions or operators may be provided.

URL Mapping Rules and Timing

This section provides the basic rules for creating a URI for assigning a URI to a file or segment, as well as the start time for each segment in the media display and presentation.

For this item, a description of the media presentation on the client is assumed.

Assume that the HTTP streaming client plays media that is loaded as part of the media view. The actual presentation time of the HTTP client can be relative to where the presentation time is relative to the start of the presentation. During initialization, the presentation time t = 0 may be allowed.

At any time t, the HTTP client can load any data with tP playback time (also relative to the start of the presentation) no more than MaximumClientPreBufferTime ahead of the actual presentation time t and any data that is necessary due to user interaction, for example, search, fast forward, etc. In some embodiments, the MaximumClientPreBufferTime may not even be set in the sense that the client can load data ahead of the current tP play time without restrictions.

The HTTP client can avoid loading unnecessary data, for example, any segments from mappings that are not intended for playback can usually not be downloaded.

The main process in providing streaming services can be downloading data by generating suitable requests for downloading all files / segments or a subset of files / segments, for example, using GET HTTP requests or partial GET HTTP requests. This description discusses how to access data for a specific tP play time, but typically a client can download data for a longer play time range to avoid inefficient requests. An HTTP client can minimize the number / frequency of HTTP requests when providing a streaming service.

To gain access to multimedia data during tP playback, or at least close to tP playback time in a particular display, the client determines the URL to the file that contains this playback time, and also determines the byte range in the file to access this playback time.

A media presentation description can assign a display id to each display, r, for example, using the RepresentationID attribute. In other words, the contents of the MPD, when recorded by the capture system or when read by the client, will be interpreted so that an assignment exists. To load data for a specific playback time tP for a specific display with id r, the client can make a suitable URI for the file.

The media presentation description can assign the following attributes to each file or segment of each display r:

(a) the serial number i of the file in the mapping r, and i = 1, 2, ..., Nr, (b) the relative start time of the file with the display id r and the file index i relative to the presentation time, defined as ts (r, i), (c) The file URI for the file / segment with the display id r and file index i, designated as FileURI (r, i).

In one embodiment, the start time of the file and the file URI can be provided explicitly for display. In another embodiment, the list of file URIs can be provided explicitly, with each file URI receiving an inherently assigned index i according to the position in the list, and the segment start time is displayed as the sum of all segment durations for segments 1 through i-1. The duration of each segment can be provided in accordance with any of the above rules. For example, any specialist in mathematics can use other methods of deriving a methodology to easily infer the start time from a single element or attribute and the position / index of the file URI in the display.

If a dynamic rule for creating URIs is provided in the MPD, then the start time of each file and each file URI can be generated dynamically using the composition rule, index of the requested file and, if possible, some additional parameters provided in the description of the multimedia presentation. Information can be provided, for example, in attributes and MPD elements, for example FileURIPattern and FileInfoDynamic. FileURIPattern provides information on how to compose a URI based on the index number i of the file index and the display id r. FileURIFormat is composed as:

FileURIFormat = sprintf ("% s% s% s% s% s.% S", BaseURI, BaseFileName,

RepresentationIDFormat, SeparatorFormat,

FileSequenceIDFormat, FileExtension);

FileURI (r, i) is composed as

FileURI (r, i) = sprintf (FileURIFormat, r, i);

The relative start time ts (r, i) for each file / segment can be displayed using some attribute contained in the MPD that describes the duration of the segments in this display, for example, the FileInfoDynamic attribute. An MPD may also contain a sequence of FileInfoDynamic attributes that is global for all displays in a media view, or at least for all displays in a period, just as defined above. If multimedia data is requested for a specific playback time tP in the display r, then the corresponding index i (r, tP) can be displayed as i (r, tp), so that the playback time of this index is in the interval of the start time ts (r, i ( r, tP)) and ts (r, i (r, tP) +1). Access to the segment may be further limited by the above cases, for example, the segment is not available.

Access to the exact playback time tP, as soon as the index and URI of the corresponding segment are obtained, depends on the actual format of the segment. In this example, suppose the multimedia segments have a local timeline that starts at 0 without loss of generality. In order to access and present the data at the tP playback time, the client can download the data corresponding to the local time from a file / segment, which can be accessed using the URI FileURI (r, i), moreover, i = i (r, t _p ) .

Typically, clients can download the entire file and can then access the tP play time. However, it is not necessary to download all the information of the 3GP file, since the 3GP file provides structures for correlating the local time distribution with the byte range. Therefore, only certain byte ranges for access to the tP playback time may be sufficient for multimedia playback, provided that sufficient random access information is available. Also, sufficient information about the structure and correlation of the byte range with the local time distribution of the multimedia segment can be provided in the initial part of the segment, for example, using the segment index. Having access to the initial, for example, 1200 bytes of the segment, the client can get enough information for direct access to the byte range necessary for the playback time tP.

In a further example, suppose that the segment index, possibly given as the tidx block below, can be used to identify the byte offsets of the required fragment or fragments. Partial GET requests can be generated for the required fragment or fragments. There are other alternatives, for example, the client can issue a standard file request and cancel it when the first tidx block is received.

SEARCH

The client may try to go to a specific presentation time t _p in the display. Based on MPD, the client has access to the start time of the multimedia segment and the URL of the multimedia segment for each segment in the display. The client can get the segment index segment_index from the segment most likely containing multimedia samples for the presentation time t _p , as the maximum index of the segment i, for which the start time tS (r, i) is less than or equal to the presentation time t _p , i.e. segment_index = max {i | tS (r, i) <= t _p }. The segment URL is obtained as FileURI (r, i).

Note that the time distribution information in the MPD may be approximate due to problems related to the location of the Random Access Points, the alignment of the media tracks and the shift in the distribution of time over the media. As a result, the segment identified by the aforementioned procedure may begin at the moment just after t _p , and the multimedia data for the presentation time t _p may be in the previous multimedia segment. In the case of a search, either the search time can be updated to equal the first sampling time of the extracted file, or the previous file can be extracted instead. However, note that during continuous playback, including cases where there is a switch between alternative displays / versions, multimedia data is still available for the time between t _p and the beginning of the extracted segment.

For an accurate search regarding the presentation time t _{p, the} HTTP streaming client needs to access a random access point (RAP). To determine the random access point in the multimedia segment in the case of Adaptive Streaming HTTP 3GPP, the client can use, for example, the information in the 'tidx' or 'sidx' block, if any, to determine the location of the random access points and the corresponding presentation time in the multimedia view. In cases where a segment is a 3GPP movie fragment, the client can also use the information in the 'moof' and 'mdat' blocks, for example, to find the RAP and obtain the necessary presentation time from the information in the movie fragment and the segment start time derived from the MPD. If there is no RAP with a presentation time before the requested presentation time t _p , then the client can either access the previous segment, or can use only the first random access point as a search result. When multimedia segments begin with RAP, these procedures are simple.

Also note that it is not necessary to download all the information of the multimedia segment in order to access the presentation time t _p . The client may, for example, first request a 'tidx' or 'sidx' block from the beginning of the media segment using byte range requests. By using the 'tidx' or 'sidx' blocks, the time distribution of the segment can be correlated with the byte range of the segment. Constantly using partial HTTP requests, you need to access only the relevant parts of the multimedia segment for improved user experience and low startup delays.

FORMING A LIST OF SEGMENTS

As described in this document, it should be obvious how to implement an HTTP direct streaming client that uses the information provided by MPD to form a list of segments for display, which has an approximate signal duration of the dur segment. In some embodiments, the client may assign consecutive indices i = 1, 2, 3, ... to the multimedia segments in the display, i.e. the first multimedia segment is assigned the index i = 1, the second multimedia segment is assigned the index i = 2, and so on. Then, startTime [i] is assigned to the list of multimedia segments with indexes of i segments, and the URL [i] is generated, for example, as follows. First, the index i is set to 1. The start time of the first multimedia segment is obtained as 0, startTime [1] = 0. The URL of media segment i, URL [i], is obtained as FileURI (r, i). The process continues for all described multimedia segments with index i, and startTime [i] of multimedia segment i is obtained as (i-1) * dur, and URL [i] is obtained as FileURI (r, i).

SIMULTANEOUS HTTP / TCP REQUESTS

A problem in a block-based streaming system is the desire to always request the highest quality blocks that can be received in full on time for playback. However, the data arrival rate may not be known in advance, and therefore, it may happen that the requested unit does not arrive on time for playback. This leads to the need to pause multimedia playback, which leads to poor user experience. This problem can be mitigated by using client algorithms that take a conservative approach to selecting blocks for a query by requesting blocks of lower quality (and therefore smaller size), which are more likely to be received on time, even if the data rate decreases during block reception. However, this conservative approach has the disadvantage of possibly transmitting lower quality playback to the user or destination device, which is also a poor user experience. The problem can be exacerbated when several HTTP connections are used simultaneously to load different blocks, as described below, since the available network resources are shared between the connections and, accordingly, are used simultaneously for blocks with different playback times.

It may be beneficial for the client to issue requests for several blocks at the same time, and in this context, “simultaneously” means that answers to requests occur at overlapping time intervals, and it does not necessarily happen that the requests are executed exactly or even approximately at the same time time. In the case of HTTP, this approach can improve the use of available bandwidth due to the behavior of the TCP protocol (which is well known). This can be especially important for improving the switching time of the content, since when new content is first requested, the corresponding HTTP / TCP connections for which data is requested for the blocks can start slowly, and accordingly, the use of several HTTP / TCP connections at this point can significantly speed up data transfer time of the first blocks. However, requesting different blocks or fragments on different HTTP / TCP connections can also lead to poor performance, since the requests of the blocks that need to be played first compete with the requests of subsequent blocks, competing HTTP / TCP downloads vary significantly in transmission speed, and accordingly the completion time the request can be very variable, and it is usually not possible to control which HTTP / TCP downloads will complete quickly and which are slower, and accordingly it is possible that at least part of the time Loading the HTTP / TCP first few blocks are completed last, respectively leading to a large and variable channel switch time.

Suppose that each block or fragment of a segment is downloaded over a separate HTTP / TCP connection, and that the number of parallel connections is n, and the playback time of each block is t seconds, and that the streaming speed of the content associated with the segment is S. When the client first starts streaming content, requests for the first n blocks representing n * t seconds of multimedia data may be issued.

As is known to those skilled in the art, there is a large deviation in the data rate of TCP connections. However, to simplify this discussion, suppose that ideally all connections occur in parallel, so that the first block will be received completely at about the same time as the other n-1 requested blocks. To further simplify the discussion, suppose that the aggregate bandwidth used by n download connections is fixed at B for the entire download duration, and that the streaming speed S is constant throughout the display. Additionally, suppose that the structure of the multimedia data is such that reproduction of the block can be performed when the entire block is available on the client, i.e. playback of a block can begin only after the entire block is received, for example, because of the structure of the underlying video encoding, or because encryption is used to encrypt each fragment or block separately, and accordingly the entire fragment or block must be received before it can be decrypted. Thus, in order to simplify the discussion below, we assume that the entire block must be adopted before any of the blocks can be played. Then the time required before the first block is received and can be played is approximately n * t * S / B.

Since it is desirable to minimize the switching time of the content, it is desirable to minimize n * t * S / B for this purpose. The value of t can be determined by factors such as the underlying video encoding structure and which capture methods are used, and accordingly t can be quite small, but very small values of t lead to an overly complicated segment map and may not be compatible with efficient video encoding and decryption, if used. The value of n can also affect the value of B, i.e. B may be greater for a larger number of n connections, and accordingly, reducing the number of connections, n, has a negative side effect of possibly reducing the amount of available bandwidth that is used, B, and therefore may be ineffective in achieving the goal of reducing content switching time. The value of S depends on which display is selected for download and playback, and ideally, S should be as close to B as possible in order to maximize the quality of multimedia playback for given network conditions. Thus, to simplify this discussion, suppose that S is approximately equal to B. Then the channel switching time is proportional to n * t. Thus, using more connections to load different fragments can degrade channel switching times if the aggregate bandwidth used by the connections is sublinearly proportional to the number of connections, which is usually the case.

As an example, suppose t = 1 second, and for n = 1 the value is B = 500 Kbit / s, for n = 2 the value is B = 700 Kbit / s, and for n = 3 the value is B = 800 Kbit / s. Suppose that a mapping with S = 700 Kbps is selected. Then, with n = 1, the loading time for the first block is 1 * 700/500 = 1.4 seconds, with n = 2 the loading time for the first block is 2 * 700/700 = 2 seconds, and n = 3 the loading time for the first block equal to 3 * 700/800 = 2.625 seconds. In addition, as the number of connections increases, it is likely that the variability of the individual download speeds of the connections increases (although even with one connection, some significant variability is likely). Thus, in this example, the channel switching time and the variability of the channel switching time increase as the number of connections increases. Intuitively, the blocks that are transmitted have different priorities, i.e. the first block has the earliest transmission deadline, the second block has the second earliest deadline, etc., while the boot connections on which the blocks are transmitted compete for network resources during transmission, and therefore the blocks with the earliest deadlines are longer delayed when more competing blocks are requested. On the other hand, even in this case, the use of, ultimately, more than one download connection allows you to maintain a consistently higher streaming speed, for example, three connections can support streaming speeds up to 800 Kbps in this example, whereas with Only 500 Kbps streams can be supported per connection.

In practice, as noted above, the data transfer speed of a connection can be very variable both within the same connection over time and between connections, and as a result of n requested blocks usually do not complete at the same time, and in fact in most cases one block may end at half the time of another block. This effect leads to unpredictable behavior, because in some cases the first block can end much earlier than other blocks, and in other cases the first block can end much later than other blocks, and as a result, playback can start relatively quickly in some cases, and in other cases may occur late. This unpredictable behavior can be frustrating to the user and therefore can be considered a poor user experience.

Therefore, methods are needed in which several TCP connections can be used to improve channel switching time and channel variability, while providing good quality and possible streaming speed. Also needed are methods that allow for adjusting the proportion of the available bandwidth allocated to each block when the reproduction time of the block approaches, so that a large proportion of the available bandwidth to the block with the closest playback time can be allocated, if necessary.

JOINT HTTP / TCP REQUEST

We now describe how to use concurrent HTTP / TCP requests together. The receiver can use several simultaneous joint HTTP / TCP requests, for example, using multiple HTTP byte range requests, each such request is for a part of a fragment in the source segment, or a whole fragment in the original segment, or a part or fragment of a recovery in a recovery segment, or all recovery fragments in the recovery segment.

The benefits of collaborative HTTP / TCP requests along with the use of FEC recovery data can be especially important for providing consistent, fast channel switching times. For example, during channel switching, it is possible that TCP connections have either just begun or have been idle for some period of time, in which case the congestion window, cwnd, is at a minimum value for connections, and accordingly, the transmission speed of these TCP connections will be required several round-trip periods (RTT) to increase, and there will be high variability in the transmission rates over different TCP connections during this increase time.

Now we will give an overview of the non-FEC method, which is a HTTP / TCP joint request method in which only the multimedia data of the source blocks is requested using several simultaneous HTTP / TCP connections, i.e. No FEC recovery data is requested. Using the non-FEC method, parts of the same fragment are requested over different connections, for example, using HTTP byte range requests for fragment parts, and accordingly each HTTP byte range request relates, for example, to the part of the byte range indicated in the segment map for fragment. It may be that a separate HTTP / TCP request increases its transmission rate in order to make full use of the available bandwidth over several RTT (round-trip periods), and accordingly there is a relatively long period of time when the transmission rate is less than the available bandwidth , and accordingly, if a single HTTP / TCP connection is used to download, for example, the first piece of content that needs to be played, then the channel switching time may be large. Using a method without FEC, downloading different parts of the same fragment over different HTTP / TCP connections can significantly reduce the time of switching channels.

An overview will now be given of a method with FEC, which is a joint HTTP / TCP request method in which source segment multimedia data and FEC recovery data generated from multimedia data are requested using multiple simultaneous HTTP / TCP connections. Using the FEC method, parts of the same fragment and FEC recovery data generated from that fragment are requested on different connections using HTTP byte range requests for fragment parts, and accordingly each HTTP byte range request relates, for example, to a byte part range specified in the segment map for the fragment. It may be that a separate HTTP / TCP request increases its transmission rate in order to make full use of the available bandwidth over several RTTs (round-trip periods), and accordingly there is a relatively long period of time when the transmission rate is less than the available bandwidth, and accordingly, if a single HTTP / TCP connection is used to download, for example, the first piece of content to be reproduced, then the channel switching time may be large. Using the method with FEC has the same advantages as the method without FEC, and has the additional advantage that not all the requested data must arrive before the fragment can be restored, thereby further reducing the channel switching time and channel switching time variability. By making requests on different TCP connections and redundantly requesting, by requesting also FEC recovery data for at least one of the connections, the amount of time it takes to transfer enough data, for example, to recover the first requested fragment that allows you to start playing multimedia, can be significantly reduce and make it much more consistent than if you didn’t use shared TCP connections and FEC recovery data.

FIG. 24 (a) - (e) show an example of fluctuations in the transmission speed of 5 TCP connections operating on the same communication line to the same client from the same HTTP web server in an emulated Development Network with Data Optimization ( EVDO). In FIG. 24 (a) - (e) the X axis shows the time in seconds, and the Y axis shows the speed for each connection at which bits are received on the client for each of the 5 TCP connections, measured at 1 second intervals. In this particular emulation, there were a total of 12 TCP connections operating on this link, and accordingly the network was relatively busy during the time shown, which could be typical when more than one client is streaming in the same cell in the mobile network. Note that although bit rates are partly correlated with time, there is a big difference in bit rates for 5 connections at a multitude of times.

FIG. 25 shows a possible request structure for a fragment that has a size of 250,000 bits (approximately 31.25 kilobytes), and there are 4 requests for HTTP byte ranges made in parallel for different parts of the fragment, i.e. the first HTTP connection requests the first 50,000 bits, the second HTTP connection requests the next 50,000 bits, the third HTTP connection requests the next 50,000 bits, and the fourth HTTP connection requests the next 50,000 bits. If FEC is not used, i.e. method without FEC, then in this example there are only 4 fragment requests. If FEC is used, i.e. method with FEC, in this example there is one additional HTTP connection that requests an additional 50,000 bits of FEC recovery data in the recovery segment formed from the fragment.

FIG. 26 is an increase in the first pair of seconds in the 5 TCP connections shown in FIG. 24 (a) to (e), wherein in FIG. 26, the X axis shows time in 100 millisecond intervals, and the Y axis shows the rate at which bits are received on the client for each of the 5 TCP connections, measured at 100 millisecond intervals. One line shows the aggregated number of bits that are received on the client for a fragment from the first 4 HTTP connections (excluding the HTTP connection for which FEC data is requested), i.e. what comes in using a non-FEC method. Another line shows the aggregated number of bits that is received on the client for a fragment from all 5 HTTP connections (including the HTTP connection over which FEC data is requested), i.e. what comes in using the FEC method. For the FEC method, it is assumed that a fragment can be decoded with FEC from receiving any 200,000 bits out of 250,000 requested bits, which can be implemented if, for example, the Reed-Solomon FEC code is used, and which can mainly be realized if, for example, the code is used RaptorQ described in Luby IV. For the FEC method, in this example, enough data is received to recover a fragment using FEC decoding after 1 second, providing a channel switching time of 1 second (assuming that data for subsequent fragments can be requested and received before the first fragment is fully reproduced). For the method without FEC in this example, all the data for 4 queries must be received before the fragment can be restored, which occurs after 1.7 seconds, leading to a channel switching time of 1.7 seconds. Thus, in the embodiment shown in FIG. In the 26th example, the method without FEC is 70% worse in terms of channel switching time than the method with FEC. One of the reasons for the advantage shown by the method with FEC in this example is that for a method with FEC, receiving any 80% of the requested data allows you to recover a fragment, while for a method without FEC you need to receive 100% of the requested data. Thus, a method without FEC needs to wait for the slowest TCP connection to complete the transfer, and due to natural changes in the TCP transmission rate, there is a tendency for the slowest TCP connection to have a large difference in transmission speed compared to the average TCP connection. In the FEC method in this example, one slow TCP connection does not determine when the fragment is recoverable. Instead, for a method with FEC, the transmission of a sufficient amount of data is much more dependent on the average TCP transmission rate than on the worst case TCP transmission rate.

There are many variations of the non-FEC method and the FEC method described above. For example, joint HTTP / TCP requests can only be used for the first few fragments after a channel switch occurs, and after that only a single HTTP / TCP request is used to load additional fragments, several fragments, or full segments. As another example, the number of used HTTP / TCP joint connections may depend on the urgency of the requested fragments, i.e. how close the playback time for these fragments, and from the current network conditions.

In some variations, multiple HTTP connections can be used to request recovery data from recovery segments. In other varieties, different amounts of data may be requested over different HTTP connections, for example, depending on the current size of the media buffer and the speed at which data is received on the client. In another variation, the source mappings are not independent of each other, but instead represent multilevel coding of multimedia, when, for example, the improved source mappings may depend on the underlying source mappings. In this case, there may be a recovery map corresponding to the base source map and another recovery map corresponding to the combination of the base and enhanced source maps.

Additional common elements are added to the benefits that can be realized using the methods described above. For example, the number of HTTP connections used may vary depending on the current amount of multimedia in the multimedia buffer, and / or the speed of reception in the multimedia buffer. Joint HTTP requests using FEC, i.e. the FEC method described above and variations of that method can be used actively when the media buffer is relatively empty, for example, more joint HTTP requests are executed in parallel for different parts of the first fragment requesting the entire source fragment and a relatively large fraction of the recovery data from the corresponding recovery fragment, and then moving on to a reduced number of concurrent HTTP requests requesting larger portions of multimedia data per request and requesting a smaller fraction of the data x reduction, for example by passing 1, 2 or 3 simultaneous HTTP request, proceeds to inquiry complete fragments or several successive tracks on the request, and proceeding to the rejection of the request recovery data when a clipboard of multimedia increases.

As another example, the amount of FEC recovery data may vary depending on the size of the media buffer, i.e. when the media buffer is small, you can request more FEC recovery data, and when the media buffer increases, the amount of requested FEC recovery data may decrease, and at some point, when the media buffer is large enough, you can not request any FEC recovery data, only data from source segments of source mappings. The advantages of such improved techniques are that they can provide shorter and more consistent channel switching times, and more flexibility for possible multimedia stuttering or shutdowns, while minimizing the amount of extra bandwidth used beyond the amount that would only be consumed by multimedia transmission in source segments, by reducing both query traffic and FEC recovery data, while at the same time providing support for the highest multimedia speeds possible for these network conditions ovium.

ADDITIONAL IMPROVEMENTS FOR USING SIMULTANEOUS HTTP CONNECTIONS

An HTTP / TCP request can be canceled if a suitable condition is met, and another HTTP / TCP request can be formed to load data that can replace the data requested in the canceled request, and the second HTTP / TCP request can request exactly the same data as in the initial request, for example, source data; or overlapping data, for example, part of the same source data and recovery data that were not requested in the first request; or completely disjoint data, for example, recovery data that was not requested in the first request. An example of a suitable condition is when the request fails due to the lack of a response from the serving block infrastructure (BSI) at the appointed time, or a failure to establish a transport connection with the BSI, or the receipt of an explicit server failure message, or other failure condition.

Another example of a suitable condition is when the reception of data is unusually slow, in accordance with a comparison of the connection speed (the speed at which the data is received in response to the request in question) with the expected connection speed, or with an estimate of the connection speed necessary to receive the response until the playback time of the contained multimedia data or other time dependent on that time.

This approach is advantageous when the BSI sometimes shows failures or poor performance. In this case, the aforementioned approach increases the likelihood that the client may continue to reliable playback of multimedia data, despite failures or poor performance in the BSI. Note that in some cases it may be preferable to design the BSI in such a way that it exhibits such failures or poor performance randomly, for example, such a design may be less expensive than an alternative design that does not exhibit such failures or poor performance or exhibits them less often. In this case, the method described in this document has the additional advantage that it allows the use of such a less expensive performance for the BSI without a natural degradation of the user's perception.

In another embodiment, the number of data requests issued corresponding to a given block may depend on whether a suitable condition for the block is fulfilled. If the condition is not met, then the client can be limited from generating additional block requests, if the successful completion of all currently incomplete data requests to the block would allow the block to be restored with a high probability. If the condition is met, then more block requests may be issued, i.e. the above limitation does not apply. An example of a suitable condition is when the time before the scheduled playing time of the unit or another time, dependent on that time, decreases below a predetermined threshold. This method is advantageous because additional data requests for the block are issued when the reception of the block becomes more urgent, because the playback time of multimedia data containing the block is approaching. In the case of generally accepted transport protocols, such as HTTP / TCP, these additional requests have the effect of increasing the proportion of available bandwidth allocated to the data, which contributes to the reception of the block in question. This reduces the time required to receive enough data to restore the block completely, and therefore reduces the likelihood that the block cannot be restored before the scheduled playback time of multimedia data containing this block. As described above, if the unit cannot be restored to the scheduled playback time of the multimedia data containing the unit, then the playback may stop, resulting in a poor user experience, and therefore the method described in this document mainly reduces the likelihood of this user poor experience.

It should be understood that throughout this description of the invention, references to the scheduled block playing time refer to the time at which encoded multimedia data containing the block may first be available on the client in order to achieve presentation playback without stopping. As will be clear to experts in the field of multimedia presentation systems, this time in practice is slightly earlier than the actual time the multimedia containing the block appeared in the physical transducers used for playback (screen, speaker, etc.), since several conversion functions may be required to multimedia data containing the block in order to actually play back that block, and these functions may take some amount of time to complete. For example, multimedia data is usually transported in a compressed form, and conversion may be used for decompression.

METHODS FOR FORMING FILE STRUCTURES SUPPORTING JOINT METHODS OF HTTP / FEC

An embodiment will now be described for generating a file structure that can be used primarily by a client using joint HTTP / FEC methods. In this embodiment, for each source segment there is a corresponding recovery segment formed as follows. The R parameter indicates on average how much FEC recovery data is generated for the source data in the source segments. For example, R = 0.33 indicates that if the source segment contains 1,000 kilobytes of data, then the corresponding recovery segment contains approximately 330 kilobytes of recovery data. The S parameter indicates the character size in bytes used for encoding and decoding FEC. For example, S = 64 indicates that the source data and the recovery data contain characters with a size of 64 bytes each for FEC encoding and decoding purposes.

The recovery segment can be formed for the source segment as follows. Each fragment of the source segment is considered the source block for the purposes of FEC encoding, and accordingly, each fragment is considered as a sequence of source symbols of the source block, from which recovery symbols are formed. The number of recovery symbols generated as a result for the first i fragments is calculated as TNRS (i) = ceiling (R * B (i) / S), where ceiling (x) is a function that outputs the smallest integer with a value that is at least least equal to x. Thus, the number of recovery symbols generated for fragment i is NRS (i) = TNRS (i) -TNRS (i-1).

The recovery segment contains concatenation of recovery symbols for fragments, when the order of recovery symbols in the recovery segment is the order of fragments from which they are formed, and in the fragment the recovery symbols are in the order of their encoding symbol identifier (ESI). The structure of the recovery segment corresponding to the structure of the initial segment is shown in FIG. 27, including a 2700 recovery segment generator.

Note that by setting the number of recovery symbols for the fragment, as described above, the total number of recovery symbols for all previous fragments, and accordingly the byte index in the recovery segment, depends only on R, S, B (i-1) and B (i) , and does not depend on any previous or subsequent structure of fragments in the original segment. This is useful because it allows the client to quickly calculate the position of the start of the recovery unit in the recovery segment, as well as quickly calculate the number of recovery symbols in that recovery unit, using only local information about the structure of the corresponding fragment of the original segment from which the recovery unit is formed. Thus, if a client decides to start downloading and playing a fragment from the middle of the original segment, then he can also quickly generate and access the corresponding recovery unit from the corresponding recovery segment.

The number of source symbols in the source block corresponding to fragment i is calculated as NSS (i) = ceiling ((B (i) -B (i-1)) / S). The last source character is padded with zero bytes for the purpose of encoding and decoding FEC if B (i) -B (i-1) is not a multiple of S, i.e. the last source character is padded with zero bytes, so it is S bytes in size for FEC encoding and decoding, but these zero bytes are not stored as part of the source segment. In this embodiment, the ESIs for the source symbol are 0, 1, ..., NSS (i) -1, and the ESIs for the recovery symbols are NSS (i), ..., NSS (i) + NRS (i) -1.

The URL for the recovery segment in this embodiment can be formed from the URL for the corresponding source segment simply by adding, for example, the suffix “.repair” to the URL of the source segment.

The recovery indexing information and the FEC information for the recovery segment are implicitly set by the indexing information for the corresponding source segment and from the values of R and S, as described herein. Temporal displacements and the fragment structure containing the recovery segment are determined by temporal displacements and the structure of the corresponding initial segment. The byte offset to the end of the recovery symbols in the recovery segment corresponding to fragment i can be calculated as RB (i) = S * ceiling (R * B (i) / S). The number of bytes in the recovery segment corresponding to fragment i is then equal to RB (i) -RB (i-1), and accordingly, the number of recovery symbols corresponding to fragment i is calculated as NRS (i) = (RB (i) -RB (i -1)) / S. The number of source symbols corresponding to fragment i can be calculated as NSS (i) = ceiling ((B (i) -B (i-1)) / S). Thus, in this embodiment, the recovery indexing information for the recovery unit in the recovery segment and the corresponding FEC information can be implicitly output from R, S and the indexing information for the corresponding fragment of the corresponding source segment.

As an example, consider the example shown in FIG. 28, showing fragment 2, which begins at the byte offset B (1) = 6,410 and ends at the byte offset B (2) = 6,770. In this example, the character size is S = 64 bytes, and the dashed vertical lines show the byte offsets in the source segment that correspond to multiple numbers S. The total size of the recovery segment as a fraction of the size of the source segment is set to R = 0.5 in this example. The number of source symbols in the source block for fragment 2 is calculated as NSS (2) = ceiling ((6,770-6,410) / 64) = ceil (5.625) = 6, and these 6 source symbols have ESI 0, ..., 5, respectively, with the first the source character is the first 64 bytes of fragment 2, which starts at the byte index of 6.410 in the source segment, the second source character is the next 64 bytes of fragment 2, which begins at the index of byte 6,474 in the source segment, etc. The final byte offset of the recovery unit corresponding to fragment 2 is calculated as RB (2) = 64 * ceiling (0.5 * 6.770 / 64) = 64 * ceiling (52.8 ...) = 64 * 53 = 3.392, and the initial byte offset of the recovery unit corresponding to fragment 2 is calculated as RB (1) = 64 * ceiling (0.5 * 6.410 / 64) = 64 * ceiling (50.07 ...) = 64 * 51 = 3.264, and accordingly in this example there are two recovery characters in the recovery block, corresponding to fragment 2 with ESI 6 and 7, respectively, starting at a byte offset of 3.264 in the recovery segment and ending at a byte offset of 3.392.

Note that in the embodiment shown in FIG. In the 28th example, even if R = 0.5 and there are 6 source symbols corresponding to fragment 2, the number of recovery symbols is not equal to 3, as can be assumed if you simply use the number of source symbols to calculate the number of recovery symbols, but instead it turned out to be 2 in according to the methods described in this document. In contrast to simply using the number of source symbols of a fragment to determine the number of recovery symbols, the embodiments described above allow one to calculate the positioning of a recovery unit in a recovery segment solely from the index information associated with the corresponding source block corresponding to the source segment. In addition, when the number, K, of source characters in the source block increases, the number of recovery characters, KR, of the corresponding recovery block is approximately expressed with K * R, since usually KR does not exceed ceil (K * R) and KR at least equals floor ((K-1) * R), where floor (x) is the largest integer of at most x.

There are many variations of the above embodiments for generating a file structure that can be used primarily by a client using joint HTTP / FEC methods, as one skilled in the art will recognize. As an example of an alternative embodiment, the initial segment for display can be divided into N> 1 parallel segments, where for i = 1, ..., N, the given fraction Fi of the original segment is contained in the i-th parallel segment, and where the sum Fi for i = 1, ..., N is 1. In this embodiment, there can be one main segment map that is used to obtain segment maps for all parallel segments, similar to how the recovery segment map is obtained from the map of the original segment in the above embodiment e implementation. For example, the main segment map can indicate the fragment structure if all the original multimedia data was not divided into parallel segments, but instead was contained in the same initial segment, and then the segment map for the ith parallel segment can be obtained from the main segment map by calculating that if the amount of multimedia data in the first prefix of fragments of the initial segment is L bytes, then the total number of bytes of this prefix in the aggregation among the first i parallel segments is ceil (L * Gi), where Gi is the sum of F _j for j = 1, ..., i. As another example of an alternative embodiment, the segments may consist of combining the source multimedia data for each fragment with the immediately following recovery data for that fragment, obtaining a segment that contains a mixture of the source multimedia data and the recovery data generated using the FEC code from those source multimedia data. As another example of an alternative embodiment, a segment that contains a mixture of source media and recovery data can be divided into several parallel segments containing a mixture of source media and recovery.

WAYS OF PROCESSING THE STREAM TRANSFER WITH A LOW DELAY

In some deployment scenarios, low latency streaming for a live broadcast service may be desirable. For example, in the case of a local event at the place of distribution, such as a sporting event or concert, it is desirable that the delay between the live broadcast and the live broadcast service on the client be as short as possible. For example, a maximum delay of 1 second may be desirable.

As described above, it may be useful to configure so that each file containing the media presentation segment starts at random access points (RAPs). Some profiles, in particular the live profile of the base ISO media file format, require each media segment to start in RAP.

However, in conditions where transmission with a low end-to-end delay is required, the duration of each segment should be short to minimize the delay between the action of the live broadcast and the presentation of the live broadcast event on the client. It is advisable to avoid inserting RAP into each segment that you want to use for low latency streaming. For example, RAPs in a video are typically implemented by IDR frames. Coding efficiency can be improved by avoiding the use of IDR frames in short segments, desirable for low-latency streaming.

According to an embodiment, a display compatible with the live profile and a low-delay display of the multimedia presentation are generated. Display compatible with the live profile has relatively long durations of multimedia segments. Each multimedia segment display compatible with the profile of the live broadcast has a RAP at the beginning of the multimedia segment. The low-latency display has relatively shorter segments (which may be referred to as “multimedia fragments”), which may not contain RAP. Clients that support low-latency streaming receive multimedia fragments formed to display low-latency multimedia presentations, while clients that do not support low-latency streaming may be able to receive multimedia segments configured to display a compatible forward profile multimedia presentations.

FIG. 30 illustrates the relationship between multimedia fragments for low-latency streaming and multimedia fragments. A media segment 3002 configured to stream a live profile contains RAP 3004 at the beginning of the multimedia data (“mdat”). In contrast, of the multimedia fragments 3004, 3306, and 3008 formed for low-delay streaming, only the multimedia fragment 3004 contains an RAP.

Multimedia fragments are generated on the fly and are available for download by clients via HTTP. Multimedia fragments can be accumulated in multimedia segments compatible with the live profile of the basic format of the ISO multimedia file without requiring any modifications to the multimedia fragments. For example, multimedia fragments may be concatenated into multimedia segments.

Both multimedia segments and multimedia fragments can be formed using the same encoding process. In this way, multimedia can be efficiently encoded for consumption by clients working in environments requiring low end-to-end latency, and by clients using a protocol requiring RAP in each segment.

In some embodiments, a segment index (SIDX) is generated for each media fragment. SIDX may include a display time range in a multimedia segment and a corresponding byte range of the multimedia segment occupied by the multimedia fragment. In some embodiments, SIDX indicates whether a RAP is present in the fragment. In FIG. 30, the contains_RAP field of the SIDX block of the multimedia fragment 3004 is set to 1, indicating that the multimedia fragment 3004 contains an RAP. The contains_RAP field of the SIDX block of multimedia fragments 3006 and 3008 is set to 0, indicating that the multimedia fragments 3006 and 3008 do not contain RAP. SIDX may optionally indicate the presentation time of the first RAP in the fragment.

According to an embodiment, the media server can form fragments for streaming with a low delay and push the fragments into the cache. The cache can link fragments to form multimedia segments compatible with the live profile. After the multimedia segments are formed, the cache can remove the multimedia fragments that were linked to form the multimedia segment.

A single multimedia presentation description (MPD) may store information about a first display with multimedia segments compatible with a live view profile of a multimedia presentation and a second display with multimedia fragments of a low-delay stream. Deferred viewing can be provided using multimedia segments for buffering delayed viewing and multimedia fragments for processing viewing at a border close to the live stream. The client can switch between these displays, for example, starting in the annealed viewing buffer and moving to the border of the live stream, skipping sections of the multimedia presentation. Each MOD display can be assigned an attribute to express an array of mappings available for a single multimedia view.

In the MPD, which stores information about the first display with multimedia segments and the second display with multimedia fragments, it may be useful to provide information indicating which multimedia fragments of the second display begin with RAP. For example, an MPD may include an attribute for indicating the frequency of occurrence of RAP in a plurality of multimedia fragments. In one embodiment, the MPD includes an attribute indicating the frequency in units of the number of fragments (i.e., each x-th multimedia fragment contains RAP). In another embodiment, the attribute indicates the frequency in units of time distance between adjacent RAPs.

Alternatively, information about multimedia fragments can be stored in the first MPD, and information about multimedia segments can be stored in the second MPD.

In some embodiments, the MPD may signal specific parameters applicable to a particular display, for example, the maximum duration of a multimedia segment or a multimedia fragment of a display.

Additional embodiments may be provided by one of ordinary skill in the art after reading this disclosure. In other embodiments, predominantly combinations or sub-combinations of the invention disclosed above may be formed. Exemplary component configurations are shown for purposes of illustration, and it should be understood that combinations, additions, reconfigurations, and the like are contemplated in alternative embodiments of the present invention. Thus, although the invention has been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible.

For example, the processes described herein may be implemented using hardware components, software components, and / or any combination thereof. In some cases, software components may be provided on tangible, permanent storage media for execution on hardware that is provided on storage media, or they are separate from the storage media. The description of the invention and the drawings, respectively, should be considered in an explanatory and not limiting sense. However, it will become apparent that various modifications and changes can be made to them without departing from the broad essence and scope of the invention as set forth in the claims, and that the invention is intended to encompass all modifications and equivalents within the scope of the following claims.

Claims (31)

1. A method of structuring data content to be serviced using a multimedia server, comprising the steps of: receive content subject to service; generating a plurality of multimedia segments representing the content and encoded in accordance with an encoding protocol that includes one or more multimedia presentation frames encoded in each multimedia segment, wherein there is a random access point in each multimedia segment; and forming a plurality of multimedia fragments encoded in accordance with a coding protocol, wherein the multimedia segment includes the plurality of multimedia fragments, and at least some of the plurality of multimedia fragments include random access points and at least some do not include random access points, and the random access point includes a position in the segment at which the decoder can decode fragments that are the last with respect to said random access point, irrespective of fragments that are prior to said random access point; and forming a segment index for a multimedia segment, wherein the segment index includes a presentation time range for each multimedia fragment in the multimedia segment, a corresponding byte range in the multimedia segment occupied by each multimedia fragment, and an indicator of the presence of a random access point that indicates whether a random access point is present access in each multimedia fragment. 2. The method of claim 1, wherein the multimedia segment is formed by combining a plurality of multimedia fragments. 3. The method according to claim 2, further comprising the step of forming a media segment in the cache, and after the media segment is formed in the cache, a plurality of media fragments used to form the media segment are deleted from the cache. 4. The method of claim 1, further comprising generating a single multimedia presentation description (MPD) file that stores information about a first multimedia presentation display containing a plurality of multimedia segments and a second multimedia presentation display display containing a plurality of multimedia fragments. 5. The method according to claim 4, in which the MPD file contains an attribute for indicating the frequency of occurrence of random access points in the second display. 6. The method of claim 5, wherein the frequency is a period of time. 7. The method of claim 5, wherein the frequency is the number of multimedia fragments. 8. The method of claim 1, further comprising determining the positions for the random access points among the plurality of multimedia fragments, the random access points being located at variable non-fixed points among the plurality of multimedia fragments. 9. A media server containing: a processor configured to: receiving content to be serviced; forming a plurality of multimedia segments representing the content and encoded in accordance with an encoding protocol that includes one or more multimedia presentation frames encoded in each multimedia segment, wherein there is a random access point in each multimedia segment; and generating a plurality of multimedia fragments encoded in accordance with an encoding protocol, wherein the multimedia segment includes the plurality of multimedia fragments, and at least some of the multimedia fragments include random access points and at least some do not include random access points, and the random access point includes a position in the segment at which the decoder can decode fragments that are after exploring with respect to said random access point, irrespective of fragments that are preceding with respect to said random access point; and generating a segment index for the multimedia segment, wherein the segment index includes a presentation time range for each multimedia fragment in the multimedia segment, a corresponding byte range in the multimedia segment occupied by each multimedia fragment, and an indicator of the presence of a random access point that indicates whether a random access point is present access in each multimedia fragment. 10. The multimedia server according to claim 9, in which the multimedia segment is formed by combining many multimedia fragments. 11. The multimedia server according to claim 10, wherein the processor is further configured to form a multimedia segment in the cache, and after the multimedia segment is formed in the cache, a plurality of multimedia fragments used to form the multimedia segment are deleted from the cache. 12. The multimedia server according to claim 9, wherein the processor is further configured to generate a single multimedia presentation description file (MPD) that stores information about a first multimedia presentation display containing a plurality of multimedia segments and a second multimedia presentation display display containing a plurality of multimedia fragments . 13. The multimedia server according to claim 12, in which the MPD file contains an attribute for indicating the frequency of occurrence of random access points in the second display. 14. The multimedia server according to claim 13, in which the frequency is a period of time. 15. The multimedia server according to claim 13, in which the frequency is the number of multimedia fragments. 16. The multimedia server according to claim 9, wherein the processor is further configured to determine positions for random access points among a plurality of multimedia fragments, wherein the random access points are located at variable non-fixed points among the plurality of multimedia fragments. 17. A permanent computer-readable medium on which computer-executable instructions are stored that, when executed by one or more computing devices, cause one or more computing devices: receive content subject to service; generate a plurality of multimedia segments representing the content and encoded in accordance with an encoding protocol that includes one or more multimedia presentation frames encoded in each multimedia segment, wherein there is a random access point in each multimedia segment; and to form a plurality of multimedia fragments encoded in accordance with an encoding protocol, wherein the multimedia segment includes said plurality of multimedia fragments, and at least some of the plurality of multimedia fragments include random access points and at least some do not include random access points, and the random access point includes a position in the segment at which the decoder can decode fragments that are the last traveling with respect to said random access point, regardless of fragments that are preceding with respect to said random access point; and generate a segment index for the multimedia segment, wherein the segment index includes a presentation time range for each multimedia fragment in the multimedia segment, a corresponding byte range in the multimedia segment occupied by each multimedia fragment, and an indicator of the presence of a random access point that indicates whether a random access point is present access in each multimedia fragment. 18. The computer-readable medium of claim 17, wherein the instructions upon execution cause one or more computing devices to determine positions for random access points among the plurality of multimedia fragments, wherein the random access points are located at variable non-fixed points among the plurality of multimedia fragments.

Images