TWI492598B - Enhanced block-request streaming system for handling low-latency streaming - Google Patents

Description

Enhanced block request streaming system for handling low latency streams

The present invention relates to improved media streaming systems and methods, and more particularly to self-adjusting to network and buffer conditions to optimize the presentation of streaming media and allowing efficient concurrent or time-distributed delivery of streaming media material. Systems and methods.

Streaming media delivery may become increasingly important as delivering high quality audio and video over packet-based networks such as the Internet, cellular and wireless networks, power line networks, and other types of networks becomes More and more common. The quality at which the delivered streaming media can be presented may depend on several factors, including the resolution (or other attributes) of the original content, the encoding quality of the original content, the ability of the receiving device to decode and render the media, at the receiver. The timeliness and quality of the received signals. In order to generate a perceived good streaming media experience, the transmission and timeliness of the signals received at the receiver may be particularly important. A good transmission can provide fidelity of the stream received at the receiver relative to the stream sent by the sender, and timeliness can represent how quickly the receiver can begin broadcasting the content after initially requesting the content.

A media delivery system can be characterized as a system having media sources, media destinations, and separate (time and/or spatial) channels for source and destination. Typically, the source includes a transmitter that can access media in an electronically manageable form, and has the ability to electronically control the receipt of media (or an approximation of the media) and provide the media to the media consumer (eg, having A receiver that is coupled in some manner to a user of the receiver, storage device or component, display device of another channel, or the like.

While many variations are possible, in a typical example, a media delivery system has one or more servers that can access media content in electronic form, and one or more client systems or devices make a pair to the server. The media requests, and the server uses the transmitter that is part of the server to transmit to the receiver at the client to deliver the media so that the received media can be consumed by the client in some manner. In a simple example, there is a server and a client for a given request and response, but this is not required.

Traditionally, media delivery systems can be characterized as "download" models or "streaming" models. The "download" model can be characterized by the time base independence between the delivery of media material and the broadcast of the media to the user or recipient device.

As an example, the media is downloaded sufficiently before it is needed or will be used, and when the media is used, there is already a generally large amount of media available at the recipient. Delivery in the context of downloading is often performed using a file transfer protocol such as HTTP, FTP, or file delivery on one-way transmission (FLUTE), and the delivery rate can be controlled by underlying traffic and/or congestion control protocols ( Such as TCP / IP) to decide. The flow or congestion control agreement The operation may be independent of the media's playout to the user or destination device, while the playout may occur concurrently with the download or at some other time.

The "streaming" mode can be characterized by the tight coupling between the delivery of media material and the time base of the media to the broadcast of the user or recipient device. Delivery in this context is often performed using a streaming protocol, such as Real Time Streaming Protocol (RTSP) for control and Instant Transfer Protocol (RTP) for media material. The delivery rate can be determined by the streaming server and usually matches the broadcast rate of the data.

Some of the shortcomings of the "download" model may be that the media material may not be available due to the time base independence between delivery and playout (eg, because the available bandwidth is less than the media data rate) , causing the broadcast to temporarily stop ("stagnation"), and this will result in a bad user experience; or it may be required to download the media material long before the broadcast (for example, because the available bandwidth is greater than the media data rate), thereby consuming The storage resources that may be scarce on the receiving device are lost, and valuable network resources are consumed for delivery, which is wasted if the content is not finally broadcast or otherwise used.

The advantage of the Download model is that the technology required to perform such downloads (for example, HTTP) is very mature, widely deployed, and fully applicable to a wide range of applications. Download servers and solutions for implementing large-scale scalability of such file downloads (eg, HTTP web servers and content delivery networks) may be readily available, making service deployment based on this technology simple and cost effective low.

Some of the shortcomings of the "streaming" model may be that, in general, the media The delivery rate of volume data is not adapted to the available bandwidth from the server to the client connection, and requires a dedicated streaming server or a more complex network architecture that provides bandwidth and latency guarantees. Although there are streaming systems that support changing the delivery rate based on available bandwidth (eg, Adobe Flash self-tuning streams), these systems are generally inferior to download traffic control protocols such as TCP in terms of utilizing all available bandwidth. Generally efficient.

Recently, a new media delivery system based on a combination of "streaming" and "downloading" models has been developed and deployed. An example of such a model is referred to herein as a "block request stream" model in which a media client uses a download protocol, such as HTTP, to request media material chunks from a service infrastructure. The focus in such systems may be the ability to begin streaming, such as using a personal computer to decode and render incoming audio and video streams and display the video on a computer screen and play the audio via a built-in speaker. Or as another example, a set-top box is used to decode and render the received audio and video streams and display the video on a television display device and play the audio via a live sound system.

Other concerns such as being able to decode the source block fast enough to keep up with the source stream rate, minimize decoding latency, and reduce the use of available CPU resources are also a problem. Another concern is to provide a robust and scalable streaming delivery solution that allows components of the system to fail without adversely affecting the quality of the streams delivered to the receiver. Other problems may occur based on information about the rapid change of the presentation while the presentation is being distributed. Therefore, it is desirable to have improved processes and equipment.

A block request streaming system typically uses an ingestion system to provide improved user experience and bandwidth efficiency for such systems, which will be supplied by a conventional file server (eg, HTTP, FTP, or the like). In the form of data, wherein the ingestion system ingests content and prepares the content as a file or material element to be supplied by the file server that may or may not include cache memory.

According to an embodiment, a media server of a block request streaming system allows for low latency streaming of content to the media. Relatively large media segments for live profile streaming can be aggregated from relatively small media segments for low latency streaming. Media segments and media segments are encoded according to the same encoding protocol.

A better understanding of the nature and advantages of the present invention will be set forth in the <RTIgt;

100‧‧‧block streaming system

101‧‧‧ Block supply infrastructure

101 (1) ‧ ‧ block supply infrastructure

102‧‧‧Ingestion

103‧‧‧Ingestion system

104‧‧‧HTTP Streaming Server

106‧‧‧HTTP cache memory

108‧‧‧Client

110‧‧‧Content storage

112‧‧‧Request

114‧‧‧Respond

122‧‧‧Network

123‧‧‧block selector

124‧‧‧block requester

125‧‧‧block buffer

126‧‧‧ buffer monitor

127‧‧‧Media Decoder

128‧‧‧Media transducer

270‧‧‧Repair segment generator

300‧‧‧ busbar

302‧‧‧Ingestion processor

304‧‧‧ memory

306‧‧‧Disk storage

308‧‧‧Video display unit

310‧‧‧number input device

312‧‧‧Network interface device

400‧‧‧ busbar

402‧‧‧Client Processor

404‧‧‧ memory

406‧‧‧Disk storage

408‧‧‧Video display unit

410‧‧‧number input device

412‧‧‧Network interface device

500‧‧‧MPD

501‧‧‧time record

502‧‧‧ indicates record

503‧‧‧Information

504‧‧‧Initialization section

505(1)‧‧‧Media section

510‧‧‧ source section

512‧‧‧Repair section

700‧‧‧Simple index

702‧‧‧ Hierarchical index

900‧‧‧Relay data sheet

902‧‧‧HTTP Streaming Client

904‧‧‧Media block

906‧‧‧HTTP Streaming Server

1000‧‧‧Video Streaming

1002‧‧‧ Block

1004‧‧‧RAP

1200‧‧‧transmitter

1202‧‧‧Relay information

1204‧‧‧Scalable layer 1

1206‧‧‧Scalable layer 2

1208‧‧‧Scalable layer 3

1210‧‧‧ Receiver

1212‧‧‧Media presentation

1300‧‧ steps

1310‧‧‧Steps

1320‧‧‧Steps

1330‧‧‧Steps

1340‧‧ steps

1410‧‧‧Steps

1420‧‧‧Steps

1430‧‧‧Steps

1440‧‧‧Steps

1450‧‧‧Steps

1710‧‧‧Steps

1720‧‧‧Steps

1730‧‧‧Steps

1740‧‧‧Steps

1750‧‧ steps

1760‧‧ steps

1770‧‧‧Steps

3002‧‧‧Media section

3004‧‧‧Media clips

3006‧‧‧Media clip

3008‧‧‧Media clips

FIG. 1 illustrates elements of a block request stream system in accordance with an embodiment of the present invention.

2 illustrates the block request streaming system of FIG. 1, illustrating more details of elements of a client system coupled to a block provisioning infrastructure ("BSI") to receive material processed by a content ingestion system.

Figure 3 illustrates the hardware/software implementation of the ingestion system.

Figure 4 illustrates the hardware/software implementation of the client system.

Figure 5 illustrates a possible structure of the content store shown in Figure 1, including segment and media presentation descriptor ("MPD") files, and segmentation, time base, and other structures within the MPD file.

FIG. 6 illustrates details of a typical source segment as may be stored in the content store illustrated in FIGS. 1 and 5.

Figures 7a and 7b illustrate a simple index and a hierarchical index within a file.

Figure 8(a) illustrates variable block size control with aligned view points over a plurality of versions of the media stream.

Figure 8(b) illustrates variable block size control with non-aligned view points over a plurality of versions of the media stream.

Figure 9(a) illustrates a relay data table.

Figure 9(b) illustrates the transfer of block and relay data tables from the server to the client.

Figure 10 illustrates a block that is independent of the RAP boundary.

Figure 11 illustrates the continuous time base and discontinuous time base across the segments.

Figure 12 is a diagram illustrating an aspect of a scalable block.

Figure 13 illustrates a graphical representation of the evolution of certain variables within a block request stream system over time.

Figure 14 illustrates another graphical representation of the evolution of certain variables within a block request stream system over time.

Figure 15 depicts a cell grid of states as a function of threshold.

16 is a flow diagram of a process that can request a single block and multiple blocks per request that can be performed in a receiver.

Figure 17 is a flow chart of a flexible pipeline process.

Figure 18 illustrates a set of candidate requests at a certain time, a prioritization of the set of candidate requests, and an instance of which requests can be issued on which connections.

Figure 19 illustrates a set of candidate requests that have evolved over time, The priority order of the set of candidate requests, and the instances on which the connections can be issued.

Figure 20 is a flow diagram of a consistent cache memory server proxy selection based on a file identifier.

Figure 21 illustrates a syntactic definition of a suitable arithmetic language.

Figure 22 illustrates an example of a suitable hash function.

Figure 23 illustrates an example of a file identifier construction rule.

24(a) to 24(e) illustrate the bandwidth fluctuation of the TCP connection.

Figure 25 illustrates multiple HTTP requests for source and repair materials.

Figure 26 illustrates an example channel change time with and without FEC.

Figure 27 illustrates details of a repair segment generator that generates a repair segment from a source segment and control parameters as part of the ingestion system shown in Figure 1.

Figure 28 illustrates the relationship between a source block and a repair block.

Figure 29 illustrates a procedure for live service at different times at the client.

Figure 30 illustrates the relationship between media segments and media segments for low latency streaming.

In the figures, like items are referred to by like reference numerals and subscripts are provided in parentheses to indicate multiple instances of similar or identical items. Unless otherwise stated, the final subscript (eg, "N" or "M") is not intended to be limited to any particular value, and the number of instances of one item may differ from the number of instances of another item, even if illustrated. The same reference numerals are used when the sub-labels are repeatedly used.

As described herein, the goal of a streaming system is to move media from a storage location of the media (or the location from which the media is being generated) to a location where the media is being consumed, ie presented to the user or otherwise by humans or Electronic consumers "run out". Ideally, the streaming system can provide uninterrupted replay (or more generally, uninterrupted "consumption") at the receiving end and can begin shortly after the user requests the stream or the streams. Play a stream or stream collection. For efficiency reasons, it is also desirable that each stream is once the user indicates that the stream is no longer needed, such as when the user is switching from one stream to another or the stream is subject to, for example, "subtitle" streaming. The presentation of such streams is stopped. If media components such as video continue to be presented, but different streams are selected to present the media components, it is often better to have the new stream occupy a limited bandwidth and stop the old stream.

The block request streaming system in accordance with the embodiments described herein provides a number of benefits. It should be understood that a viable system need not include all of the features described herein, as some applications may provide a satisfactory level of experience with features that are less than all of the features described herein.

HTTP streaming

HTTP streaming is a specific type of streaming. Under HTTP streaming, the source can be a standard web server and a content delivery network (CDN) and standard HTTP can be used. This technique can involve streaming segments and using multiple streams, all of which fall within the context of standardized HTTP requests. Media such as video can be encoded with multiple bit rate to form different versions or representations. The terms "version" and "representation" are used synonymously herein. Each version or The representation can be broken down into smaller pieces to form segments, which may be on the order of a few seconds. Each segment can then be stored as a separate file on a web server or CDN.

On the client side, requests for individual segments can then be made using HTTP, which are seamlessly stitched together by the client. The client can switch to different data rates based on the available bandwidth. The client may also request multiple representations, each representation presenting different media components, and may present the media in the representations jointly and synchronously. The triggering of the handover may include, for example, buffer occupancy and network measurements. When operating in steady state, the client can adjust the pace requested to the server to maintain the target buffer occupancy.

Advantages of HTTP streaming can include bit element rate self-adjustment, quick start and view, and minimal unnecessary delivery. These advantages stem from controlling delivery to only a short time ahead of playout, maximizing the available bandwidth (via variable bit rate media), and optimizing streaming segmentation and smart client program.

The media presentation description can be provided to the HTTP streaming client to enable the client to provide the streaming service to the user using a collection of files (eg, in the format specified by 3GPP, referred to herein as a 3gp segment). The media presentation description and possibly the update of the media presentation description describes the media presentation of the set of structured segments, each segment containing media components to enable the client to present the included media in a synchronized manner and to provide advanced features. , such as viewing, switching bit element rates, and jointly presenting media components in different representations. The client can use the media presentation description information in different ways to get the service. Specifically, according to the media presentation description, the HTTP streaming client can decide which segment in the set can be accessed, so that the data in the streaming service is for the client capability and the user. It is useful.

In some embodiments, the media presentation description can be static, but the segments can be dynamically created. The media presentation description can be as compact as possible to minimize access and download time for the service. Other dedicated server connectivity can be minimized, such as general or frequent time base synchronization between the client and the server.

Media presentations can be constructed to allow access by terminals with different capabilities - such as access to different access network types, different current network conditions, display size, access bit rate, and transcoder support. The client can then extract the appropriate information to provide streaming services to the user.

The media presentation description can also allow deployment flexibility and compactness as required.

In the simplest case, each replacement representation can be stored in a single 3GP file, ie in accordance with a file as defined in 3GPP TS 26.244 or in accordance with ISO as defined in ISO/IEC 14496-12 or derivative specifications. Any other file of the base media file format (such as the 3GP file format described in 3GPP Technical Specification 26.244). In the remainder of this document, when referring to 3GP files, it should be understood that ISO/IEC 14496-12 and derivative specifications may map all of the described features to more general terms as defined in ISO/IEC 14496-12 or any derivative specification. Sexual ISO base media file format. The client can then request the initial portion of the file to learn the media relay material (which is typically stored in a movie header package, also referred to as a "moov" package) along with the movie clip time and byte offset. Transfer amount. The client can then issue an HTTP partial access request to obtain the requested movie fragment.

In some embodiments, it may be desirable to split each representation into segments, where the segments. In the case where the segment format is based on the 3GP file format, the segment contains a non-overlapping time slice of the movie segment, which is called "split by time". Each of the segments may contain multiple movie segments and each segment itself may be a valid 3GP file. In another embodiment, the representation is split into an initial segment containing the relay material (typically a movie header "moov" packet) and a set of media segments, each media segment containing media material, and the initial segment and The cascading of any media segment constitutes a valid 3GP file, and the initial segment of one representation and the cascading of all media segments also constitute a valid 3GP file. The entire presentation can be constructed by sequentially broadcasting each segment, mapping the local time stamp within the file to the global presentation time according to the start time of each representation.

It should be noted that the reference to "segment" throughout this specification should be understood to include any material item constructed or read or otherwise obtained from the storage medium as a result of a file download agreement request (including, for example, an HTTP request). . For example, in the case of HTTP, the data item can be stored in an actual file resident on a disk or other storage medium connected to the HTTP server or forming part of the HTTP server, or the data item can be executed in response to the HTTP request. Constructed by CGI scripts or other dynamically executed programs. Unless otherwise indicated, the terms "file" and "segment" are used synonymously herein. In the case of HTTP, a segment can be thought of as the entity body of an HTTP request response.

The terms "presentation" and "content item" are used synonymously herein. In many instances, the presentation is an audio, video or other media presentation with a defined "broadcast" time, but other variations are also possible.

Unless otherwise stated, the terms "block" and "fragment" are in this section. It is used synonymously in the text and usually represents the smallest data aggregate with an index. Based on the available indexing methods, the client can request different portions of the fragment in different HTTP requests, or can request one or more consecutive fragments or fragment portions in an HTTP request. In the case where a segment based on an ISO base media file format or a segment based on a 3GP file format is used, the segment typically represents a combination defined as a movie clip header ('moof') packet and a media material ('mdat') packet. Movie clips.

In this document, in order to simplify the description herein, it is assumed that the network carrying the data is based on packets. It should be appreciated after reading this disclosure that those skilled in the art can apply the embodiments of the invention described herein to other types of Transmission network, such as a continuous bit stream network.

In this document, in order to simplify the description herein, it is assumed that the FEC code provides protection against long data delivery time and variable problems. It should be appreciated after reading this disclosure that those skilled in the art can apply the embodiments of the present invention to others. Types of data transfer problems, such as bit flipping of data. For example, in the absence of FEC, if the last portion of the requested segment is much later than the previous portion of the segment or the arrival time of the last portion of the requested segment varies greatly, the content swap time may be long and available. In the case of FEC and parallel requests, only the majority of the data requested for the segment needs to be recovered after the arrival, thereby reducing the variability in content switching time and content switching time. In this description, it can be assumed that the data to be encoded (ie, the source material) has been decomposed into equal-length "symbols" that can have any length (small to a single bit), but for different parts of the data, the symbols Can have different lengths, for example, different characters can be used for different data blocks Number size.

In this description, in order to simplify the description herein, it is assumed that FEC is applied to a data "block" or "slice" each time, that is, a "block" is a "source block" for FEC encoding and decoding purposes. The client device can use the segment indexing method described herein to illustrate the source block structure of the decision segment. One skilled in the art can apply embodiments of the present invention to other types of source block structures, such as a source block that can be part of a segment, or can encompass one or more segments or segment portions.

An FEC code that is considered to be used in conjunction with a block request stream is typically a system FEC code, i.e., the source symbol of the source block can be included as part of the encoding of the source block, and thus the source symbol is transmitted. As will be appreciated by those skilled in the art, the embodiments described herein are equally applicable to non-systematic FEC codes. The system FEC encoder generates a number of repair symbols from the source block formed by the source symbols, and the combination of at least some of the source symbols and the repair symbols is the encoded symbol representing the source block transmitted on the channel. Some FEC codes may be useful for efficiently generating as many repair symbols as are required, such as "information additive code" or "fountain code", and examples of such codes include "chain reaction codes" and "Multi-level chain reaction code". Other FEC codes, such as Reed-Solomon codes, may actually only produce a limited number of repair symbols per source block.

In many instances of the examples, it is assumed that the client is coupled to a media server or a plurality of media servers, and the client requests streaming media to the media server or the plurality of media servers on a channel or a plurality of channels. . However, more complicated arrangements are also possible.

Benefit example

The media client maintains the coupling between the time base of the block request and the time base for media broadcast to the user via the block request stream. This model preserves the advantages of the "download" model described above while avoiding some of the disadvantages that are often decoupled between media playout and data delivery. The block request stream model utilizes the rate and congestion control mechanisms available in transport protocols such as TCP to ensure that the maximum available bandwidth is used for media material. In addition, partitioning the media presentation into blocks allows each of the encoded media data blocks to be selected from a set of multiple available codes.

The selection can be based on any number of criteria, including a match of the media data rate and the available bandwidth - even when the available bandwidth changes over time, the media resolution or decoding complexity matches the client capabilities or configuration, or A match of user preferences such as language. The selection may also include downloading and rendering of auxiliary components such as accessibility components, closed captioning, subtitles, sign language video, and the like. Use existing block request streaming system model include mobile network (Move Network ^TM), Microsoft Smooth Stream (Microsoft Smooth Streaming) and ^TM streaming protocols Apple iPhone.

Typically, each media material block can be stored on the server as an individual file, and then requested by the HTTP server software executing on the server in cooperation with a protocol such as HTTP. Typically, the client is provided with a relay profile, which may be, for example, an Extensible Markup Language (XML) format or a playlist text format or a binary format. The relay profile describes what is commonly referred to in this document. Features presented for "represented" media, such as available encodings (eg, required bandwidth, resolution, encoding parameters, Media type, language), and the way to divide the code into blocks. For example, the relay material may include a uniform resource locator (URL) for each block. The URL itself may provide a scheme such as prepending with the string "http://" to indicate that the protocol that will be used to access this recorded resource is HTTP. Another example is "ftp://" to indicate that the protocol to be used is FTP.

In other systems, the media block may be constructed "in execution" by a server in response to a request from the client to indicate the requested portion of the media presentation in time. For example, in the HTTP scenario using the scheme "http://", a request response is provided for execution of the request for the URL, and some specific material is included in the entity body of the request response. The implementation of how the response to the request is generated in the network can be quite different, depending on the implementation of the server that serves such requests.

Typically, each block can be independently decodable. For example, in the case of video media, each block can start at the "view point." In some coding schemes, the view point is called a "random access point" or "RAP", although not all RAPs are designated as view points. Similarly, in other coding schemes, the view point starts in the "independent data refresh" frame or "IDR" in the case of H.264 video coding, although not all IDRs are designated as view points. The view point is the location in the video (or other) media where the decoder does not need to know about the previous frame or data or sampled data, and the frame or sample being decoded is not in a self-sustaining manner but for example as current The situation in which the difference between the frame and the previous frame is encoded may be the case of information about the previous frame or data or sampled.

The focus in such systems may be the ability to start streaming. For example, a personal computer is used to decode and render the received audio stream and video stream and display the video on a computer screen and play the audio via the built-in speaker, or as another example, use the set-top box to decode and render the received signal. The audio stream and the video stream are displayed on the television display device and played through the live sound system. The main concern may be to allow the user to decide to view the new content delivered as a stream and take action to express his decision (eg, the user clicks on a link in the browser window or a play button on the remote control). The delay between the time when the content starts to be played on the user's screen (hereinafter referred to as "content change time") is minimized. Each of these points of interest can be addressed by elements of the enhanced system described herein.

An example of a content swap is that the user is viewing the first content delivered via the first stream, and then the user decides to view the second content posted via the second stream and initiates an action to begin viewing the second content. The second stream may be sent from a set of servers that are the same or different than the first stream. Another example of a content swap is that the user is visiting the website and decides to begin viewing the first content delivered via the first stream by clicking on a link within the browser window. In a similar manner, the user may decide not to start from the beginning, but to play the content from some time within the stream. The user instructs the user's client device to view the time location and the user may expect the selected time to be rendered immediately. Minimizing content swap time is important for video viewing, allowing users to get a high-quality, fast content surfing experience while searching and sampling a wide range of available content.

Recently, it has become customary to consider the use of Forward Error Correction (FEC) codes to protect streaming media during transmission. When in the packet network (the packet network instance package) When transmitted over the Internet and on such wireless networks as are standardized by groups such as 3GPP, 3GPP2, and DVB, the source stream is placed into the packet when it is generated or becomes available, and thus such The packet can be used to carry the source stream or content stream to the receiver in the order in which the source stream or the content stream is generated or becomes available.

In a typical scenario where an FEC code is applied to such types of scenarios, the encoder may use the FEC code to establish a repair packet, which is then sent as a supplement to the original source packet containing the source stream. The repair packet has the following properties: When the source packet is lost, the received repair packet can be used to recover the data contained in the lost source packet. The repair packet can be used to recover the contents of the completely lost source packet, but can also be used to recover from partial packet loss, either from a fully received repair packet or even from a partially received repair. The package is carried out. Thus, a whole or partially received repair packet can be used to recover an entire or partially lost source packet.

In other instances, other types of corruption may occur in the transmitted material, such as bit values may be flipped, and thus the repair packet may be used to correct such corruption and provide as accurate recovery as possible of the source packet. In other examples, the source stream is not necessarily transmitted as an individual packet, but instead may be sent, for example, as a continuous bit stream.

There are many examples of FEC codes that can be used to provide protection for source streams. The Reed-Solomon code is a well-known code for error and erasure correction in a communication system. For the erasure correction on the packet data network, for example, the well-known and efficient implementation of the Reed-Solomon code is used in L.Rizzo's "Effective Erasure Codes for Reliable Computer Communication". Protocols (Efficient Erase Codes for Reliable Computer Protocols), Computer Communications Review 27(2): 24-36 (April 1997) (hereinafter referred to as "Rizzo") and Bloemer et al. "An XOR" -Based Erasure-Resilient Coding Scheme, Technical Report TR-95-48, International Computer Science Association, Berkeley, California (1995) (hereafter referred to as "XOR-Reed -Cauchy matrix or Vandermonde matrix described in or elsewhere.

Other examples of FEC codes include LDPC codes, chain reaction codes such as those described in Luby I, and multi-level chain reaction codes such as those in Shokrollahi I.

Examples of FEC decoding procedures for variants of Reed-Solomon codes are described in Rizzo and XOR-Reed-Solomon. In these examples, the decoding may be applied after sufficient source data packets and repair data packets have been received. The decoding process may be computationally intensive, and depending on the available CPU resources, the decoding process may take quite a considerable amount of time in proportion to the length of time spanned by the media in the block. The receiver can account for the length of time required for decoding when the calculus begins to receive the delay between the media stream and the broadcast of the media. This delay due to decoding is perceived by the user as a delay between the user's request for a particular media stream and the start of replay. It is therefore desirable to minimize this delay.

In many applications, the packet can be further subdivided into symbols that apply the FEC process to the symbol. A packet may contain one or more symbols (or less than one symbol, but typically the symbols will not be split across the packet group unless the error conditions between the packet groups are known to be highly correlated). Symbols can have any size, but symbols The size is often at most equal to the size of the packet. The source symbols are the symbols that encode the material to be transmitted. A repair symbol is a supplemental symbol that is generated directly or indirectly from a source symbol as a source symbol (ie, if all source symbols are available and no repair symbols are available, the data to be transmitted can be completely recovered).

Some FEC codes may be block based because the encoding operation depends on one or more symbols in the block and may be independent of the symbols that are not in the block. Via block-based coding, the FEC encoder may generate the repair symbols for the block from the source symbols in the block, and then proceed to the next block without reference to the source symbols other than the current block being encoded. Other source symbols. In transmission, a source block including a source symbol may be represented by an encoded block that includes encoded symbols (the encoded symbols may be some source symbols, some repair symbols, or both). In the presence of repair symbols, not all source symbols are required in each coded block.

For some FEC codes, especially Reed-Solomon codes, the encoding and decoding time may increase to an impractical level as the number of encoded symbols per source block increases. Therefore, in practice, there is often a realistic upper bound on the total number of encoded symbols that can be generated per source block (255 is the approximate practical upper limit for some applications), especially when Reed-Solomon encoding is performed by custom hardware. Or in the typical case of the decoding process, for example, using the Reed-Solomon code included as part of the DVB-H standard to protect the stream against packet loss, the MPE-FEC process is implemented in specialized hardware within the cellular phone. The specialized hardware is limited to a total of 255 Reed-Solomon coded symbols per source block. Since the symbols are often required to be placed in separate packet payloads, the maximum length of the source block being encoded is set. A realistic upper limit has been set. For example, if the packet payload is limited to 1024 bytes or less and each packet carries one encoded symbol, the encoded source block can be up to 255 kilobytes, and of course the size of the source block itself is also It is the upper limit.

Such as being able to decode the source block fast enough to keep up with the source stream rate, minimize the decoding latency introduced by FEC decoding, and use only a small portion of the available CPU on the receiving device at any point during FEC decoding, etc. Other concerns are addressed by the elements described in this article.

There is a need to provide a robust and scalable streaming delivery solution that allows components of the system to fail without adversely affecting the quality of the streams delivered to the receiver.

The block request stream system needs to support changes to the structure or relay material of the presentation, such as changes to the number of available media codes or parameters encoding the media (such as bit rate, resolution, aspect ratio, audio or A change in the video transcoder or codec parameter, or a change to other relay material associated with the content file, such as a URL. Such changes may be necessary for a number of reasons, including editing together large rendered content from different sources, such as advertisements or different segments, as for example due to configuration changes, equipment failure, or recovery from equipment failure. Modifications to URLs or other parameters that become necessary as a result of changes in service infrastructure caused by other reasons.

There is a method by which the presentation can be controlled by a continuously updated playlist file. Since the file is continuously updated, at least some of the changes described above can be made in the updates. A disadvantage of the conventional method is that the client device must constantly retrieve (also called "polling") the playlist file, thereby setting the service infrastructure. The load is applied, and the file may not be cached longer than the update interval, making the task of the service infrastructure much more difficult. This is addressed by the elements herein to provide such updates as described above without the need for the client to continuously poll the relay profile.

Another problem that is typically known from broadcast distributions, particularly in live services, is the lack of the ability for the user to view content that is broadcast earlier than when the user joined the program. Typically, local personal recording consumes unnecessary local storage, or local recording is not possible when the client does not tune to the program or is prohibited by content protection regulations. Network recording and time-shift viewing are preferred, but require individual connections to the server and separate delivery protocols and infrastructure from live services, resulting in duplicate infrastructure and significant server costs. This is also addressed by the elements described herein.

System overview

One embodiment of the present invention is described with reference to Figure 1, which illustrates a simplified diagram of a block request streaming system embodying the present invention.

In FIG. 1, a block streaming system 100 is illustrated. The block streaming system 100 includes a block provisioning infrastructure ("BSI") 101, which in turn includes an ingestion system 103 for ingesting content 102. The content is prepared and packaged for provision by the HTTP streaming server 104 via the content store 110, which is accessible to both the ingest system 103 and the HTTP streaming server 104. As shown, system 100 can also include HTTP cache memory 106. In operation, client 108, such as an HTTP streaming client, sends a request 112 to HTTP streaming server 104 and receives response 114 from HTTP streaming server 104 or HTTP cache 106. In each case, in Figure 1 The illustrated elements can be implemented, at least in part, in software, including code executed on a processor or other electronic device.

Content may include movies, audio, 2D flat video, 3D video, other types of video, images, timed text, timed relay data, or the like. Some content may relate to material to be presented or consumed in a timed manner, such as for presenting auxiliary information (stage identification, advertising, stock price, ^FlashTM sequence, etc.) along with other media being broadcast. It is also possible to use other media combinations and/or other hybrid presentations beyond audio and video alone.

As shown in FIG. 2, the media block may be stored within the block provisioning infrastructure 101(1), which may be, for example, an HTTP server, a content delivery network device, an HTTP proxy, FTP proxy or server, or some other media server or system. The block provisioning infrastructure 101(1) is connected to a network 122, which may be, for example, an Internet Protocol ("IP") network such as the Internet. The block request stream system client is shown with six functional elements, namely: block selector 123, which is provided with the relay material described above and executes a plurality of blocks indicated by the relay material The function of the block or partial block to be requested is selected among the available blocks; the block requester 124 receives the request command from the block selector 123 and performs the supply to the block on the network 122. The infrastructure 101(1) transmits a request for a specified block, a portion of the block, or a plurality of blocks, and receives an operation including the data of the block as a reply; and a block buffer 125; a buffer monitor 126 Media decoder 127; and one or more media transducers 128 that facilitate media consumption.

The block data received by the block requester 124 is transferred to the block buffer 125 storing the media material for temporary storage. Alternatively, received The block data can be directly stored into the block buffer 125 as shown in FIG. The media decoder 127 provides the media data from the block buffer 125 and performs the transformation necessary to provide the appropriate input to the media transducer 128, and the media transducer 128 renders in a form suitable for the user or other consumer. The media. Examples of media transducers include visual display devices such as those found in mobile phones, computer systems, or televisions, and may also include audio rendering devices such as speakers or headphones.

An example of a media decoder may be to convert data in a format described in the H.264 video coding standard to an analog or digital representation of a video frame (such as a YUV format map with associated presentation timestamps for each frame or sample). Meta map) function.

The buffer monitor 126 receives information about the contents of the block buffer 125 and provides input to the block selector 123 based on the information and possibly other information that is used to determine the selection of the block to be requested, As described herein.

In the terminology used herein, each block has a "broadcast time" or "duration", which means that the receiver plays the media included in the block at normal speed. The amount of time. In some cases, the playout of media within a tile may depend on having received material from a previous tile. In rare cases, the broadcast of some media in a block may depend on subsequent blocks. In this case, the broadcast time of the block can be broadcasted without reference to subsequent blocks in the block. The media defines, and the broadcast time of the subsequent block increases the broadcast time of the media that can only be broadcasted after the subsequent block has been received in the block. As included in the block depends on the follow-up The media of the block is a rare case, so in the remainder of the case, it is assumed that the media in one block does not depend on subsequent blocks, but it should be noted that those skilled in the art will recognize that such variations can be easily added. To the embodiments described below.

The receiver may have controls such as "pause", "fast forward", "reverse", etc., which may cause the block to be consumed by playing at different rates, but if the receiver can acquire and decode each consecutive block If the lumped time of the sequence is equal to or less than the lumped play time in the case of excluding the last block in the sequence, the receiver can present the media to the user without stagnation. In some descriptions herein, a particular location in a media stream is referred to as a particular "time" in the media, the particular "time" corresponding to the beginning and end of the media broadcast at that particular location in the video stream. The time that will elapse between time. The time or location in the media stream is a well-known concept. For example, where the video stream includes 24 frames per second, the first frame may be referred to as having a position or time of t=0.0 seconds, and the 241th frame may be referred to as having a position of t=10.0 seconds or time. Naturally, in a frame-based video stream, the position or time need not be continuous, because the first bit from the 241th frame to the first bit before the first bit of the 242th frame in the stream The elements can all have the same time value.

Using the above term, a Block Request Streaming System (BRSS) includes one or more clients making requests to one or more content servers (eg, HTTP servers, FTP servers, etc.). The ingestion system includes one or more ingest processors, wherein the ingest processor receives the content (on the fly or not), processes the content for use by the BRSS and possibly also the relay data generated by the ingest processor Stored together in a storage accessible by the content server.

The BRSS can also include content cache memory that is coordinated with the content server. The content server and the content cache memory may be a conventional HTTP server and HTTP cache memory, and the HTTP server and the HTTP cache memory receive a request for an archive or a segment in the form of an HTTP request including a URL, and the The request may also include a byte range to request all of the files or segments that are not indicated by the URL. The client may include a conventional HTTP client that makes requests to the HTTP server and handles responses to their requests, where the HTTP client is requested by the request, the request is passed to the HTTP client, and the HTTP client is obtained. The endpoints respond and process their responses (or store, transform, etc.) to provide the response to the novel client system driver that presents the player for broadcast by the client device. Typically, the client system does not know in advance which media will be needed (because such needs may depend on user input, user input changes, etc.), so the client system is referred to as a "streaming" system because of the media It is "consumed" once it is received or shortly thereafter. As a result, the response delay and bandwidth constraints may result in a delay in rendering, such as a pause in rendering when the stream catches up where the user is consuming the presentation.

In order to provide a presentation that is perceived as having good quality, multiple details can be implemented in the BRSS on the client or at the ingestion end or both. In some cases, the details achieved are considered and addressed in response to the client-server interface on the network. In some embodiments, both the client system and the ingestion system are aware of the enhancement, while in other embodiments, only one side is aware of the enhancement. In such cases, the entire system would benefit from the enhancement even if one side did not know the enhancement, while in other cases, the benefit was only known on both sides. This happens only in strong situations, but when it is not known on one side, the benefit is still operating without failure.

As illustrated in FIG. 3, in accordance with various embodiments, the ingestion system can be implemented as a combination of a hardware component and a software component. The ingestion system can include a set of instructions that can be executed to cause the system to perform any one or more of the methods discussed herein. The system can be implemented as a specialized machine in the form of a computer. The system can be a server computer, a personal computer (PC), or any system capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the system. Moreover, although only a single system is illustrated, the term "system" shall also be taken to include any collection of systems that individually or jointly execute a set of instructions (or sets of instructions) to perform any of the discussed herein. One or more methods.

The ingestion system can include an ingest processor 302 (e.g., a central processing unit (CPU)), a memory 304 that can store code during execution, and a disk storage 306, all of which communicate with each other via the bus bar 300. The system can further include a video display unit 308 (eg, a liquid crystal display (LCD) or a cathode ray tube (CRT)). The system can also include a digital sign input device 310 (e.g., a keyboard), and a network interface device 312 for receiving content sources and delivering content storage.

Disk storage unit 306 can include machine readable media on which one or more sets of instructions (eg, software) that implement any one or more of the methods or functions described herein can be stored. The instructions may also reside wholly or at least partially within the memory 304 and/or the ingest processor 302 during execution of the instructions by the system, wherein the memory 304 and the ingest processor 302 also constitute machine readable media.

As illustrated in Figure 4, a client system can be implemented as a combination of hardware and software components, in accordance with various embodiments. The client system can include a set of instructions that can be executed to cause the system to perform any one or more of the methods discussed herein. The system can be implemented as a specialized machine in the form of a computer. The system can be a server computer, a personal computer (PC), or any system capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the system. Moreover, although only a single system is illustrated, the term "system" shall also be taken to include any collection of systems that individually or jointly execute a set of instructions (or sets of instructions) to perform any of the discussed herein. One or more methods.

The client system can include a client processor 402 (e.g., a central processing unit (CPU)), a memory 404 that can store code during execution, and a disk storage 406, all of which communicate with each other via the bus bar 400. The system can further include a video display unit 408 (eg, a liquid crystal display (LCD) or a cathode ray tube (CRT)). The system can also include a digital sign input device 410 (e.g., a keyboard) and a network interface device 412 for transmitting requests and receiving responses.

Disk storage unit 406 can include machine readable media on which one or more sets of instructions (eg, software) that implement any one or more of the methods or functions described herein can be stored. The instructions may also reside wholly or at least partially within the memory 404 and/or the client processor 402 during execution of the instructions by the system, wherein the memory 404 and the client processor 402 also constitute a machine readable media.

3 GPP file format usage

3GPP file format or ISO-based media file format (such as Any other file of the MP4 file format or the 3GPP2 file format) can be used as a container format for HTTP streaming with the following features. A segment index may be included in each segment to signal the notification time offset and the byte range to enable the client to download the appropriate profile or media segment as needed. The global presentation time base presented by the entire media and the local time base within each 3GP file or media segment can be accurately aligned. Individual tracks within a 3GP file or media segment can be accurately aligned. Individual trajectories across the various representations may also be aligned by assigning each of the individual trajectories to a global isochronal line such that the switching across the representations may be innocent and the joint presentation of the media components in the different representations may be synchronized .

The file format may contain a profile for self-adjusting streaming having the following properties. All movie material can be included in the movie clip - the "moov" package can contain no sampling information. The audio and video sampling data can be interleaved as required to meet the requirements of the progressive download profile as specified in TS 26.244. The "moov" package can be placed at the beginning of the file, followed by the fragment offset data, also referred to as the segment index, which contains each segment of the containment segment or at least the time offset of the subset of fragments. The amount of shift information and the range of bytes.

It is also possible for the media to present a description that references the file that follows the existing progressive download profile. In such a case, the client can simply select a suitable alternate version from among the plurality of available versions using the media presentation description. The client may also use the HTTP partial fetch request for files conforming to the progressive download profile to request a subset of each alternate version and thereby achieve a less efficient form of self-adjusting streaming. In such a case, different representations of the media included in the progressive download profile may still adhere to a common global isochronal line to enable infinity switching across representations.

Advanced method overview

In the following subsections, a method for an improved block request streaming system is described. It should be understood that some of these improvements may be used with or without other improvements in such improvements, depending on the needs of the application. In normal operation, the receiver makes a request to a server or other transmitter for a particular data block or data block portion. A file, also referred to as a segment, may contain multiple tiles and be associated with one representation of the media presentation.

Preferably, index information, also referred to as "segment index" or "segment map", is generated, and the index information provides a mapping from the broadcast or decode time to the byte offset of the corresponding block or segment within the segment. The segment index can be included in the segment, typically at the beginning of the segment (at least some segments are mapped at the beginning) and tend to be small. The segment index can also be set in a separate index segment or file. Especially in the case where the segment index is included in the segment, the receiver can quickly download some or all of the segment map and then use some or all of the segment map to determine the time offset and the same file. A mapping between the corresponding byte locations of the segments associated with the time offset.

The receiver can use the byte offset to request data from segments associated with a particular time offset without having to download all of the data associated with other segments that are not associated with the time offset of interest. In this way, the segment map or segment index can greatly improve the receiver's ability to directly access the portion of the segment that is related to the current time offset of interest, including the improved content swap time, on the network. When conditions change, the ability to quickly switch from one representation to another, and the reduction of wasted network resources to download media that is not broadcast at the receiver.

In the case of considering switching from one representation (herein referred to as "switching from" to another representation (referred to herein as "switching to" representation), the segment index may also be used to identify "switched to" Indicates the start time of the random access point in the representation, thereby identifying the amount of data to be requested in the "switched from" representation to ensure innocent switching. The meaning of this innocent handover means that the media in the "switched from" representation is The rendering time until the broadcast of the representation of "switched to" can be started innocently from the random access point.

These blocks represent segments of video media or other media required by the requesting receiver to produce output to the user of the receiver. The receiver of the media can be a client device, such as when the receiver receives content from a server that delivers the content. Examples include set-top boxes, computers, game consoles, specially equipped televisions, handheld devices, specially equipped mobile phones, or other client receivers.

Many advanced buffer management methods are described in this article. For example, the buffer management method enables a client to request a block with the highest media quality that can be received in time for continuous broadcast. The variable block size feature improves compression efficiency. The ability to have multiple connections for transmitting blocks to the requesting device while limiting the frequency of requests provides improved transmission performance. Some of the received data blocks can be used to continue media presentation. Connections can be reused for multiple blocks without having to commit to a particular set of blocks by the connection at the outset. The consistency of selecting servers from among multiple possible servers by multiple clients is improved, which reduces the frequency of content repetition in nearby servers and improves the probability that the server will contain the entire file. The client may request the media regions based on relay material (such as available media encoding) embedded in the URL for the file containing the media block. Piece. The system can provide a calculation and minimization of the amount of buffering time required before the content can be broadcast without incurring a subsequent pause in the media playout. The available bandwidth can be shared between multiple media blocks, adjusted as the play time of each block approaches, so that a larger share of the available bandwidth can be preferentially assigned to the necessary The block with the closest broadcast time.

HTTP streaming can use relay data. Presentation level relay data includes, for example, streaming duration, available encoding (bit element rate, transcoder, spatial resolution, frame rate, language, media type), indicators pointing to each encoded stream relay material, And content protection (Digital Rights Management (DRM) information). The streaming relay data can be a URL about the segment file.

Segment relay data may include byte range versus time information for intra-segment request and identification of random access points (RAPs) or other view points, some or all of which may be part of a segment index or segment map .

Streaming can include multiple encodings of the same content. Each code can then be broken down into segments, with each segment corresponding to a storage unit or file. In the case of HTTP, a segment is typically a resource that can be referenced by a URL, and a request for such a URL results in returning the segment as the entity body of the request response message. A segment may include multiple picture groups (GoPs). Each GoP may further include a plurality of segments, wherein the segment index provides time/byte set offset information for each segment, that is, the unit of the index is a segment.

The fragment or fragment portion can be requested via a parallel TCP connection to increase throughput. This can alleviate the problems that arise when the connection on the shared bottleneck link or when the connection is lost due to congestion, thereby increasing the overall speed and reliability of the delivery, which can significantly improve the speed and reliability of the content change time. can To wait for time with excessive bandwidth by over-requesting, but care should be taken to avoid making requests for too distant futures, which increases the risk of scarcity.

Multiple requests for segments on the same server can be pipelined (the next request is made before the current request is completed) to avoid repeated TCP startup delays. Requests for consecutive fragments can be aggregated into one request.

Some CDNs prefer big files and can trigger the background retrieval of the entire file from the original server when the range request is first seen. However, the vast majority of CDNs will request a range of services from the cache memory - if the data is available. Therefore, it may be advantageous to have a portion of the client request for the entire segment. Such requests may be cancelled later if necessary.

The effective switching point may be a view point in the target stream, specifically, for example, a RAP. Different implementations are possible, such as fixed GoP structures or RAP alignment across streams (based on the beginning of the media or based on GoP).

In one embodiment, the segments and GoPs can be aligned across streams of different rates. In this embodiment, the GoP can be of variable size and can include multiple segments, but the segments are not aligned between the streams of different rates.

In some embodiments, gear redundancy may be advantageously employed. In these embodiments, an erasure code is applied to each segment to produce a redundant version of the material. Preferably, source formatting is not altered by the use of FEC, and additional steps in the ingestion system can be used to generate additional repair segments containing subordinate representations of the FEC repair material, such as a source representation, and to make additional repair segments available. A client that can reconstruct a fragment using only the source material of the fragment can request only the source material of the fragment within the segment from the server. If the server is unavailable or the connection to the server is slow - this may be determined before or after requesting the source material, then Requesting additional repair information about the segment from the repair segment, which reduces the time for reliably delivering enough data to recover the segment, making it possible to use FEC decoding to recover using a combination of received source and repair data. Source of the piece. In addition, if the clip becomes urgent, that is, the play time of the clip becomes imminent, additional repair data can be requested to allow the clip to be restored, which increases the share of the data for the clip on the link, but Other connections on the link are more efficient at releasing bandwidth. This can also alleviate the risk of scarcity due to the use of parallel connections.

The fragment format may be a stored Real-Time Transport Protocol (RTP) packet stream with audio/video synchronization via instant messaging control protocol RTCP.

The segment format may also be a stored MPEG-2 TS packet stream with audio/video synchronization via an MPEG-2 TS internal time base.

Use signal passing and/or block building to make streaming more efficient

Several features may or may not be used in the block request streaming system to provide improved performance. Performance may be related to the ability to broadcast presentations without stagnation, to obtain media material within bandwidth constraints, and/or to do so within limited processor resources at the client, server, and/or ingestion system. Some of these features will now be described.

Index within the segment

In order to compose a partial acquisition request for a movie segment, the client may be informed of the byte offset and start time of all media components contained in the segment or segment within the decoding or presentation time, and also what segment Starting with or containing a random access point (and therefore suitable for use as a switching point between replacement representations), This information is often referred to as a segment index or a segment map. The start time of the decoding or presentation time can be directly expressed or can be expressed as Δ with respect to the reference time.

This time and byte offset index information may require at least 8 bytes of material per movie segment. As an example, for a 2-hour movie contained within a single file with a 500 ms movie clip, this would be a total of approximately 112 kilobytes of material. Downloading all of this material at the start of the presentation may result in significant additional startup delays. However, the time and byte offset data can be hierarchically encoded so that the client can quickly find small time blocks and offset data associated with the point in the presentation that is desired to begin. The information may also be distributed within the segments such that some refinement of the segment index may be interleaved with the media material.

Note that if the representation is segmented into multiple segments by time, then using this hierarchical encoding may not be necessary, as the complete time and offset data for each segment may already be very small. For example, if the segment is one minute instead of 2 hours in the above example, the time-byte offset index information is about 1 kilobyte data, and the 1 kilobyte data can usually be loaded into a single TCP/IP. Inside the package.

There are different options that may be used to add clip time and byte offset data to the 3GPP file: First, a Movie Fragment Random Access Packet ("MFRA") may be used for this purpose. The MFRA provides a table that assists the reader in using movie clips to find random access points in the file. To support this feature, the MFRA includes, in tandem, the byte offset of the MFRA packet with random access points. The MFRA can be placed at or near the end of the file, but this is not required. By scanning the movie fragment random access offset packet from the end of the file and using the size information in the movie fragment random access offset packet, it is possible to locate the beginning of the movie fragment random access packet. however, Placing MFRA at the end for HTTP streaming typically requires at least 3 to 4 HTTP requests to access the desired data: at least one for requesting MFRA from the end of the file, one for obtaining MFRA and the last one for obtaining the file. The desired piece. Therefore, it may be desirable to put it at the beginning because the mfra can be downloaded along with the first media material in a single request. In addition, using MFRA for HTTP streaming may be inefficient because any information other than time and moof_offset in "MFRA" is not required, and specifying the offset instead of the length may require more Multi-bit.

Second, the Item Location Pack (ILOC) can be used. The "ILOC" provides a directory of relay data resources in the file or other files by locating the metadata of the relay data resource, the offset of the relay data resource in the file, and the length of the relay data resource. For example, the system can consolidate all externally referenced relay data resources into one file and re-adjust the file offset and file references accordingly. However, "ILOC" is intended to provide a location for relaying data, so it may be difficult to coexist "ILOC" with real relay data.

Finally and perhaps most suitable is the provision of a new package called Time Index Packet ("TIDX"), which is dedicated to providing exact fragment time or duration and byte offsets in an efficient manner. This point is described in more detail in the next section. A replacement package with the same functionality may be a segment index package ("SIDX"). In this context, the two may be interchangeable unless otherwise indicated, as both packages provide the ability to provide exact fragment time or duration and byte offsets in an efficient manner. The differences between TIDX and SIDX are provided below. How to interchange TIDX packages and SIDX packages should be obvious, because both packages implement segment indexing.

Segment index

The segment has the identified start time and the number of identified bytes. Multiple segments can be concatenated into a single segment, and the client can issue a request identifying a particular range of bytes within the segment that corresponds to the requested segment or subset of segments. For example, when using HTTP as a request protocol, the HTTP range header can be used for this purpose. This method requires the client to access the "segment index" of the segment, which specifies the location of the different segments within the segment. The "segment index" can be provided as part of the relay material. This approach has the result that there is much less need to build and manage than the way each block is kept in a separate file. The management of creating, delivering, and storing very large numbers of files (such as for 1 hour presentation, which can be extended to thousands of files) can be complex and error prone, so reducing the number of files represents an advantage.

If the client only knows the desired start time for the smaller portion of the segment, the client can request the entire file and then read the file from beginning to end to determine the appropriate play start location. In order to improve the bandwidth utilization, the segment may include an index file as a relay material, wherein the index file maps the byte range of the individual block to the time range corresponding to the blocks, and is referred to as a segment index or a segment map. The relay material can be formatted as XML material or the relay material can be binary, such as an atomic and packet structure that follows the 3GPP file format. The index can be simple, where the time and byte range of each block is absolute relative to the beginning of the file, or the time and byte range of each block can be hierarchical, with some blocks being Compilation of parent blocks (and their parent blocks are grouped into grandparents, etc.) and the time and byte range of a given block is relative to the time and/or bit of the parent block of the block The tuple range is expressed.

Sample index mapping structure

In one embodiment, the original source material of a representation of the media stream may be included in one or more media files referred to herein as "media segments", wherein each media segment includes for replaying the media Media data for consecutive periods, such as 5 minute media replays.

Figure 6 illustrates an example overall structure of a media segment. Within each segment, at the beginning of the source segment or throughout the source segment, there may also be index information including a time/byte tuple offset segment map. The time/byte tuple offset segment map in one embodiment may be a list of time/byte tuple offset pairs (T(0), B(0)), (T(1), B(1). ), ..., (T(i), B(i)), ..., (T(n), B(n)), where T(i-1) represents the segment relative to the medium The initial start time in all media segments is used to replay the start time of the i- th media segment, T(i) represents the end time of the i- th segment (and therefore the start time of the next segment), and the byte offset The quantity B(i-1) is the corresponding byte index of the beginning of the material in the source segment relative to the beginning of the i- th media segment at the beginning of the source segment, and B(i) is the corresponding end of the i- th segment. The byte index (and therefore the index of the first byte of the next segment). If the segment contains a plurality of media components, T(i) and B(i) may be provided for each of the segments in an absolute manner, or T(i) and B(i) may be used relative to the reference media component. Another media component is expressed.

In this embodiment, the number of segments in the source segment is n, where n may vary from segment to segment.

In another embodiment, the time offset for each segment in the segment index can be determined by the absolute start time of the first segment and the duration of each segment. In this case, the segment index can record the start time of the first segment and The duration of all the clips included in this paragraph. The segment index can also only record a subset of fragments. In this case, the segment index records the duration of the sub-segment defined as one or more consecutive segments that end at the end of the containment segment or at the beginning of the next sub-segment.

For each segment, there may also be a value indicating whether the segment starts or contains a view point, that is, at a certain point, the media after the point does not depend on any media before the point, and thus from the piece The forward media can be broadcast independently of the previous clips. The point of view is generally the point in the media where it can be broadcast independently of all previous media. Figure 6 also illustrates a simple example of a possible segment index for a source segment. In its example, the time offset value is in milliseconds, and thus the first segment of the source segment begins 20 seconds from the beginning of the media, and the first segment has a broadcast time of 485 milliseconds. The byte offset of the beginning of the first segment is 0, and the byte offset of the beginning of the first segment/the second segment is 50245, and thus the size of the first segment is 50245 bytes. If the segment or sub-segment does not start at a random access point, but the segment or sub-segment contains a random access point, the decoding time or presentation time difference between the start time and the actual RAP time may be provided. This allows the client to accurately know when the "switched from" representation needs to be presented all the time in the case of switching to the media segment.

Additionally or alternatively, a simple or hierarchical index, a daisy chain index, and/or a hybrid index may be used.

Since the sampling durations of different tracks may not be the same (for example, video sampling may play for 33ms, and audio sampling may last for 80ms), different tracks in a movie clip may not start and end at exactly the same time, that is, audio. May start slightly earlier or later than video, but in front of it The break may be exactly the opposite case as compensation. To avoid ambiguity, the timestamps specified in the time and byte offset data can be specified relative to a particular trajectory, and this trajectory can be the same trajectory for each representation. Usually this will be the video track. This allows the client to accurately identify the next video frame when switching the representation.

Despite the above problems, attention can be paid to maintaining a strict relationship between the trajectory time stamp and the presentation time during presentation to ensure smooth broadcast and maintain audio/video synchronization.

FIG. 7 illustrates some examples, such as a simple index 700 and a hierarchical index 702.

Two specific examples of a packet containing a segment map are provided below, one called a time index packet ('TIDX') and one called ('SIDX'). This definition follows the package structure according to the ISO base media file format. Other designs for such packages to define similar syntax and have the same semantics and functionality should be apparent to the reader.

Time index package

definition

Package type: ‘tidx’

Container: file

Mandatory: No

Quantity: any number 0 or 1

The time index package may provide a set of time and byte offset indexes that associate certain regions of the file with certain time intervals presented. The time index package may include a target type field, and the target type field indication The type of information used. For example, a time index packet with the target type "moof" provides an index of the media segments contained in the file in the sense of both time and byte offset. A time index packet with the target type "Time Index Box" can be used to construct a hierarchical time index, allowing the user of the file to quickly navigate to the desired portion of the index.

The segment index can, for example, contain the following syntax:

Semantics

Targettype: The type of package material referenced by the index package for this time. This target type can be a movie clip header ("moof") or a time index packet ("tidx").

Time-reference_track_id: indicates the trajectory to be referenced when specifying the time offset in this index.

Number_of_elements: The number of elements indexed by this time indexed packet.

First_element_offset (the first element offset): The byte offset of the first indexed element since the beginning of the file.

First_element_time: The start time of the first indexed element, using the time stamp specified in the media header package of the track identified by the "time reference track id".

Random_access_flag (random access flag): 1 if the start time of the element is a random access point. Otherwise 0.

Length: The length of the indexed element in bytes.

deltaT (ΔT): The difference between the start time of the element and the start time of the next element, which is the form of the time stamp specified in the media header packet of the track identified by the "time reference track id".

Segment index package

The segment index package ('sidx') provides a compact index of movie fragments and other segment index packets in the segment. There are two loop structures in the segment index package. The first loop records the first sample of the sub-segment, that is, the sample in the first movie segment referenced by the second loop. The second loop provides an index of the sub-segment. The container of the ‘sidx’ package is either the file or the segment directly.

syntax

Semantics:

The reference_track_ID (reference track ID) provides the track ID of the reference track.

Track_count: The number of tracks indexed in the next loop (1 or greater); reference_count (reference count): the number of elements indexed by the second loop (1 or greater); track_ID (track ID) ): The track segment is included in the ID of the track in the first movie segment identified by the index; there is exactly one "track ID" equal to "reference track ID" in the loop.

Decoding_time: the decoding time of the first sample in the track identified by the "track ID" in the movie segment referenced by the first item in the second loop, expressed as the time stamp of the track (eg, in the track) Referenced in the time stamp field of the media header package; reference_type (reference type): when set to 0, indicates that the reference is a reference to a movie fragment ('moof') package; when set to 1, indicates the reference Is a reference to the segment index ('sidx') package; reference_offset (reference offset): from the first byte after the inclusion segment index packet to the first byte of the referenced packet in bytes Distance; subsegment_duration: When the reference is a reference to the segment index package, the field carries the sum of the "sub-segment duration" fields in the second loop of the packet; when the reference is for the movie fragment When quoted, this field carries the reference movie, the indicated movie clip, and the first movie clip recorded in the next entry in the loop or the subsequent power of the end of the sub-segment (whichever is earlier) The sum of the sample durations of the samples in the film break; the duration is expressed in the time stamp of the track (as recorded in the time stamp field of the media header package of the track); contains_RAP (including RAP): when the reference is When a movie clip is referenced, if the track segment of the track in which the "track ID" is equal to the "reference track ID" in the movie segment contains at least one random access point, the bit may be 1, otherwise the bit is set. 0; when a reference is a reference to a segment index, the bit is set to 1 only if the referenced bit in any of the segments is set to 1, otherwise set to 0; RAP_delta_time(RAP_△ Time): If "Include RAP" is 1, the presentation (synthesis) time of the random access point (RAP) is provided; if "Include RAP" is 0, the value 0 is reserved. This time is expressed as the difference between the decoding time of the first sample of the sub-segment described by the entry and the presentation (synthesis) time of the random access point in the trajectory of the "track ID" equal to the "reference track ID".

Difference between TIDX and SIDX

TIDX and SIDX provide the same functionality for indexing. The first cycle of SIDX supplements also provides the global time base of the first movie segment, but the global time base can also be included in the movie segment itself, which is either absolute or relative to the reference trajectory.

The second cycle of SIDX implements the functionality of TIDX. Specifically, SIDX grants a mix of targets with references to each index referred to by "reference type", while TIDX simply refers to either only TIDX or only MOOF. The "number of elements" in TIDX corresponds to the "reference count" in SIDX, the "time reference track ID" in TIDX corresponds to the "reference track ID" in SIDX, and the "first element offset" in TIDX corresponds to In the first entry of the second loop "reference offset", "first element time" in TIDX corresponds to "decoding time" of "reference track" in the first loop, and "random access flag" in TIDX corresponds to "include RAP" in SIDX The latter also has extra freedom - in SIDX, RAP may not necessarily be placed at the beginning of the segment, and therefore requires "RAP_△ time", and "length" in TIDX corresponds to "reference offset" in SIDX. And finally ΔT in TIDX corresponds to the "sub-segment duration" in SIDX. Therefore, the functionality of the two packages is equivalent.

Variable block size control and sub-GoP blocks

For video media, the relationship between the video encoding structure and the block structure for the request may be important. For example, if each block starts from a view point such as a random access point ("RAP") and each block represents an equal video time period, the positioning of at least some of the view points in the video medium is fixed. And the view points will appear at regular intervals within the video code. As is well known to those skilled in the art of video coding, if the view points are placed according to the relationship between the video frames, and in particular if the view points are placed at a frame that has little in common with the previous frame, Then the compression efficiency can be improved. Therefore, it is required that each block represents the same amount of time to constrain the video coding, so that the compression may be optimal.

It is desirable to allow the position of the view point within the video presentation to be selected by the video coding system, rather than requiring the view point to be in a fixed position. Allowing the video encoding system to select the view point for improved video compression, and thus providing a higher video media quality using a given available bandwidth, resulting in an improved user experience. The current block request streaming system may require all blocks to have the same duration (in video time), and each block must start at the view point and This is therefore a disadvantage of existing systems.

A novel block request streaming system that provides advantages over the above described system is now described. In one embodiment, the video encoding process of the first version of the video component can be configured to select the location of the viewing point in an effort to optimize compression efficiency, but requires a maximum of the duration between the viewing points. The latter requirement does limit the selection of the viewing point by the encoding process and thus reduces the compression efficiency. However, if the maximum duration between the inspection points is not too small (eg, greater than about 1 second), the reduction in compression efficiency is small compared to the reduction in compression efficiency incurred if the inspection point requires a regular fixed position. . In addition, if the maximum duration between the inspection points is a few seconds, the reduction in compression efficiency is generally negligible compared to the fully free positioning of the inspection points.

In many embodiments including this embodiment, there may be some RAPs that are not view points, that is, there may be frames that are RAPs that are not selected as view points between two consecutive view points, for example due to the RAP at time The view is too close to the surrounding view point, or the amount of media data between the view point before or after the RAP and the RAP is too small.

The location of the view point within all other versions of the media presentation can be constrained to be the same as the view point in the first (eg, highest media data rate) version. This does reduce the compression efficiency of these other versions compared to allowing the encoder to freely select the view points.

The use of a view point typically requires that the frame be independently decodable, which generally results in a low compression efficiency of the frame. Frames that are not required to be independently decodable can be encoded with reference to data in other frames. This generally increases the compression efficiency of the frame by the amount of commonality between the frame to be encoded and the reference frame. the amount. For efficient selection of the view point location, it is preferred to select a frame having a low commonality with the previous frame as the view point frame, and thereby minimizing the compression efficiency penalty incurred by encoding the frame in an independently decodable manner.

However, the degree of commonality between the frame and the potential reference frame is highly correlated across different representations of the content because the original content is the same. Therefore, the constraint that the view points in the other variants have the same position as the view points in the first variant does not make a big difference in compression efficiency.

The view point structure is preferably used to determine the block structure. Preferably, each view point determines the beginning of the block and there may be one or more blocks covering the material between the two consecutive view points. Since the duration between the views is not fixed for the purpose of encoding with good compression, it is not required that all blocks have the same play duration. In some embodiments, the tiles are aligned between versions of the content - that is, if there is a block spanning a particular frame group in one version of the content, then in another version of the content There are blocks that span the same frame group. Each block of a given version of content does not overlap, and each frame of content is contained within exactly one block of each version.

A feature that enables efficient use of the variable duration between the view points and thus the variable duration GoP is a segment index or segment map that can be included in the segment or provided to the client via other means, ie, this can be A relay profile associated with the segment in the representation that includes the start time and duration of each block of the presentation is provided. When the user has requested that the presentation begin at a particular point within the segment, the client may use the segment index material when deciding which block to start the rendering in. If no such relay material is provided, the rendering can only be at the beginning of the content. Or starting at a random or approximate point close to the point (eg, selecting the starting block by dividing the requested starting point (in terms of time) by the average block duration to provide an index of the starting block).

In one embodiment, each block may be provided as a separate file. In another embodiment, a plurality of consecutive blocks may be aggregated into a single file to form a segment. In this second embodiment, each version of the relay material may be provided, the relay data including the start time and duration of each block and the byte offset at the beginning of the block within the file. The relay material may be provided in response to an initial agreement request, that is, may be obtained separately from the segment or file, or may be included in the same file or segment as the block itself, such as at the beginning of the file. As will be apparent to those skilled in the art, the relay material can be encoded in a compressed form such as gzip or delta encoding or in binary form to reduce the network resources required to transmit the relay material to the client.

6 illustrates an example of a segment index in which a block has a variable size, and wherein the category of the block is a partial GoP, that is, a partial amount of media material between one RAP and the next RAP. In this example, the view point is indicated by a RAP indicator, where a RAP indicator value of 1 indicates that the block begins or contains a RAP or a view point, and wherein the RAP indicator 0 indicates that the block does not contain a RAP or a view point. In this example, the first three blocks, that is, the bytes 0 to 157033 constitute the first GoP, and the first GoP has a presentation duration of 1.623 seconds, and the presentation time of the first GoP goes from 20 seconds into the content. 21.623 seconds. In this example, the first of the first three blocks includes a rendering time of 0.485 seconds and includes the first 50245 bytes of media material in the segment. In this example, blocks 4, 5 and block 6 constitute a second GoP, block 7 and block 8 constitute a third GoP, and blocks 9, 10 and Block 11 constitutes a fourth GoP. Note that there may be other RAPs in the media material that are not designated as view points, and therefore are not passed as RAPs in the segment map.

Referring again to Figure 6, if the client or receiver wants to access content starting at a time offset of about 22 seconds into the media presentation, the client may first use an MPD such as described in more detail below. Other information to first determine that the relevant media information is indeed in the paragraph. The client can download the first portion of the segment, for example using an HTTP byte range request, to obtain a segment index, which in this example has only a few bytes. Using this segment index, the client can decide that the first block that it should download is the first block with a time offset of at most 22 seconds and starting at RAP, which is the actual view point. In this example, although block 5 has a time offset of less than 22 seconds, ie, the time offset of block 5 is 21.965 seconds, the segment index indicates that block 5 does not begin with RAP and is thus based on Segment index, the client instead selects download block 4, because block 4 starts at most 22 seconds, that is, block 4 has a time offset of 21.623 seconds, and block 4 starts with RAP. Therefore, based on the segment index, the client will make an HTTP range request starting at byte offset 157034.

If the segment index is not available, the client may have to download all of the previous 157034 bytes of data before downloading the material, resulting in much longer startup time or channel swap time, and wasted downloading of useless data. Alternatively, if the segment index is not available, the client can approximate where the desired data is at the beginning of the segment, but the approximation may be bad and the approximation may miss the appropriate time and then need to retreat, which again increases Startup delay.

In general, each block covers a portion of the media material, the department The points, along with the previous blocks, can be broadcast by the media player. Therefore, such a block structure and a mechanism for transmitting a notification segment index to a client signal into a block structure (whether included in a segment or provided to a client via other means) can significantly improve the client to provide fast channel switching, and The ability to broadcast innocently in the face of network changes and disruptions. Support for a variable duration block and a block that only covers portions of GoP as implemented by the segment index can significantly improve the streaming experience. For example, referring again to FIG. 6 and the example described above in which the client wants to start playing at about 22 seconds into the presentation, the client may request the material in block 4 via one or more requests and then once the The block can be used to feed the data in the block to the media player to begin the replay. Thus, in this example, once the 42011 byte of block 4 is received at the client, the playout begins, thereby enabling fast channel change times. If instead the client is required to request the entire GoP before the playback is about to start, the channel change time will be longer because there is 144211 bytes of data.

In other embodiments, a RAP or a view point may also be present in the block, and the segment index may have information indicating where the RAP or view point is within the block or segment. In other embodiments, the time offset may be the decoding time of the first frame within the block, rather than the presentation time of the first frame within the block.

Figures 8(a) and 8(b) illustrate examples of variable block size control for a view point structure across multiple versions or representations; Figure 8(a) illustrates the media stream Variable block size control with aligned view points on multiple versions, and Figure 8(b) illustrates variable block size control for non-aligned view points on multiple versions of media streams .

The time is shown in seconds across the top, and the two segments of the two representations The block and the view point are illustrated from left to right in the form of the time bases of the blocks and the view points with respect to the isochronal line, and thus the length of each block shown and the play time of each block Proportional rather than proportional to the number of bytes in the block. In this example, the segment indices of the two segments of the two representations will have the same time offset for the view points, but with potentially different numbers of tiles or segments between the view points, and due to The amount of media data is different, so the blocks have different byte offsets. In this example, if the client wants to switch from representation 1 to representation 2 at a presentation time of about 23 seconds, the client may request block 1.2 in the segment representing 1 and start requesting from block 2.2. A segment representing 2, and thus the switching will occur at the presentation coincident with the view point 1.2 in representation 1, the view point 1.2 being at the same time as the view point 2.2 in representation 2.

As should be apparent from the foregoing, the described block request streaming system does not constrain the video encoding to place the view point at a particular location within the content, and this alleviates one of the problems of prior systems.

In the embodiments described above, the view points that are organized such that the various representations of the same content are presented are aligned. However, in many cases, the alignment requirements are preferably relaxed. For example, it is sometimes the case that an encoding tool has been used to generate a representation that does not have the ability to produce a representation of the point of view alignment. As another example, content presentations can be independently encoded into different representations without view point alignment between different representations. As another example, the representation may include more view points when the representation has a lower rate and is more generally required to switch or when representing a trick mode that includes a view point to support such as fast forward or rewind or fast view. Accordingly, it is desirable to provide a method that enables a block request stream system to efficiently and flawlessly address various view points that represent misalignment across content presentations.

In this embodiment, the locations of the viewing points across the representations may not be aligned. The tiles are constructed such that the new block begins at each view point, and thus there may be no alignment between the different versions of the tiles presented. An example of a non-aligned view point structure between such different representations is illustrated in Figure 8(b). The time is illustrated in seconds across the top, and the blocks and viewpoints of the two segments of the two representations are illustrated from left to right in the form of the time bases of the blocks and the view points relative to the isochronal line, and Thus the length of each block shown is proportional to the playout time of each block rather than to the number of bytes in the block. In this example, the segment indices of the two segments of the two representations will have potentially different time offsets for the view points, and there will also be a potentially different number of blocks or segments between the view points, and due to each block The amount of media data in the media is different, so the blocks have different byte offsets. In this example, if the client wants to switch from representation 1 to representation 2 at a presentation time of about 25 seconds, the client may request block 1.3 in the segment representing 1 and start requesting from block 2.3. Represents a segment of 2, and thus the handover will occur at a presentation that coincides with the view point 2.3 in representation 2, which is located in the playout of block 1.3 in representation 1, and thus some of the media in block 1.2 will not Broadcast (although the media material of the frame that was not broadcasted in block 1.3 may have been loaded into the receiver buffer for decoding of other broadcast frames of block 1.3).

In this embodiment, the operation of the block selector 123 can be modified such that whenever the block selector 123 is required to select a block from a different representation than the previously selected version, the first frame of the block is not too late. The latest block of the frame after the last frame of the last selected block is selected.

The recently described embodiment eliminates constraints other than the first version The requirement for view point locations within other versions, and thus the compression efficiency of such versions, results in a higher quality presentation for a given available bandwidth and this is an improved user experience. A further consideration is that video encoding tools that perform a plurality of encoded (versioned) view point alignment functions across content may not be universally available, and thus the advantage of this recently described embodiment is that currently available video encoding tools can be used. Another advantage is that the encoding of different versions of the content can be done in parallel without the need for coordination between different versions of the encoding process. Another advantage is that an extra version of the content can be encoded at a later time and added to the presentation without having to provide the encoding tool with a list of specific view point locations.

In general, where the picture is encoded as a picture group (GoP), the first picture in the sequence can be a view point, but this is not always necessary.

Optimal block division

One concern in the block request streaming system is the interaction between the structure of the encoded media, such as video media, and the block structure for the tile request. As will be appreciated by those skilled in the art of video coding, it is often the case that the number of bits required for the encoded representation of each video frame sometimes varies from frame to frame. Therefore, the relationship between the amount of data received and the duration of the media encoded by the material may not be straightforward. In addition, the division of media data into blocks within a block request streaming system adds one-dimensional complexity. In particular, in some systems, the media material of the block may not be broadcast until the entire block is received, for example, the media data layout within the block using the erasure code or the media sample within the block. The interdependence may lead to this nature. Due to the complex interaction between block size and block duration and the possibility to receive the entire block before starting the broadcast, the client system usually takes A conservative approach in which media information is buffered before the broadcast begins. Such buffering results in long channel switching times and therefore poor user experience.

Pakzad describes the "block partitioning method", which is based on the underlying structure of the data stream to determine how to divide the data stream into contiguous blocks. The Pakzad further describes the methods in the string. Several advantages in the context of a streaming system. A further embodiment of the present invention for applying the block partitioning method of Pakzad to a block request stream system will now be described. The method can include arranging the media material to be presented in a substantially presentation time order such that the broadcast time of any given element of the media material (eg, video frame or audio sample) and the broadcast time of any adjacent media material elements The difference is less than the set threshold. Such sorted media material can be considered a stream of data in the case of Pakzad, and any Pakzad method applied to the stream of data identifies the block boundary of the stream of data. The data between any pair of adjacent block boundaries is considered a "block" in the context of the present case, and the method of the present invention is applied to provide presentation of the media material within the block request streaming system. As will be apparent to those skilled in the art upon reading this disclosure, several advantages of the method disclosed in Pakkad can thus be implemented in the context of a block request streaming system.

As described in Pakzad, the decision on the block structure of the segment (including blocks that cover part of the GoP or parts that cover more than one GoP) can affect the client's ability to implement fast channel change times. In Pakzad, a method is provided that will provide a block structure and a target download rate in the case of a target boot time, which will ensure that if the client starts downloading the representation at any view point and is at that target After the start time has elapsed and the broadcast starts, the amount of data that the client has downloaded at least at each point in time is at least the target. The download rate is multiplied by the elapsed time since the download began, and the broadcast will continue innocently. It is advantageous to enable the client to access the target boot time and the target download rate, since this provides the client with a means of deciding when to start broadcasting the representation at the earliest point in time, and allows the client to continue as long as the download satisfies the above conditions. Broadcast the representation. Therefore, the method described later provides means for including the target startup time and the target download rate within the media presentation description, so that the method can be used for the above purpose.

Media presentation data model

Figure 5 illustrates a possible structure for content storage shown in Figure 1, including segment and media presentation description ("MPD") files, and segmentation, time base, and other structures within the MPD file. Details of possible implementations of the MPD structure or file will now be described. In many instances, the MPD is described as a file, but a non-file structure can also be used.

As illustrated therein, the content store 110 houses a plurality of source segments 510, MPDs 500, and repair segments 512. The MPD may include a time period record 501, which in turn may include a representation record 502 indicating that the record 502 includes segment information 503 such as a reference to the initialization segment 504 and the media segment 505.

Figure 9(a) illustrates an example relay data table 900, while Figure 9(b) illustrates how the HTTP streaming client 902 obtains the relay data table 900 and media over the connection with the HTTP streaming server 906. An instance of block 904.

In the methods described herein, a "media presentation description" is provided that includes information regarding representations of media presentations available to the client. The representation can be alternative. The meaning of substitution is that the client chooses one of the different alternatives, or the representation can be complementary. The meaning of complementarity means that the client chooses its A number of representations, each representation may also come from a set of alternatives, and the representations are presented jointly. The representation may advantageously be assigned to a group, where the client is programmed or configured to understand that for a representation in a group, the representations are each a substitute for each other, and the representations from the different groups enable one or more representations to be jointly presented. . In other words, if there is more than one representation in the group, the client picks a representation from the group, picks a representation from the next group, etc. to form the presentation.

The information represented by the description may advantageously include details of the applied media transcoder, including the profile and level of the transcoders required to decode the representation, the video frame rate, the video resolution, and the data rate. The client receiving the media presentation description can use this information to determine in advance whether the representation is suitable for decoding or rendering. This represents an advantage, because if the difference information will only be included in the binary data of the representation, then it will be necessary to request binary data from all representations and parse and extract relevant information in order to find information about the applicability of the difference information. The multiple requests and the parsing of the data and the extraction may take some time, which results in a long startup time and thus a poor user experience.

In addition, the media presentation description may include information limiting the client request based on the hour. For example, for live services, the client can be limited to the portion of the request presentation that is close to the "current broadcast time." This represents an advantage because for live broadcasts, it may be desirable to empt the content of the content broadcasted above the threshold set by the current broadcast time from the service infrastructure. This is desirable for reuse of storage resources within the service infrastructure. This may also be desirable depending on the type of service provided, for example, in some cases, due to receiving a certain subscription model of the client device, the presentation may be made available only live, Other media can be rendered live and on-demand, and other presentations can be made available only for the first type of client device, only on-demand for the second type of client device, and for the third type of client device. A live or on-demand combination is available. The method described in the "Media Presentation Data Model" section (below) allows the client to be notified of such policies so that the client can avoid making requests and adjusting the supply to the user for materials that may not be available in the service infrastructure. Alternatively, for example, the client may present to the user a notification that the material is not available.

In a further embodiment of the invention, the media segment may conform to the ISO-based media file format described in ISO/IEC 14496-12 or the derivative specification (such as the 3GP file format described in 3GPP Technical Specification 26.244). (above) "Use of the 3GPP Archive Format" This subsection describes a novel enhancement to the ISO Base Media Archive format to permit efficient use of the data structure of the file format within the Block Request Streaming System. As described in this reference, information can be provided within the file to permit a fast and efficient mapping between the time period of media presentation and the range of bytes within the file. The media material itself can be structured according to the movie fragment construction defined in ISO/IEC 14496-12. This information providing time and byte offsets can be hierarchically structured or structured into a single block. This information can be provided at the beginning of the file. The use of efficient coding as described in this section of the "Use of 3GPP Archive Format" to provide this information causes the client to retrieve the information quickly, for example in the case of a block download protocol used by the block request stream system in the case of HTTP. The request is retrieved to quickly retrieve the information, which results in a very short start, view or streaming switching time and thus results in an improved user experience.

The representation in the media presentation is synchronized on the global isochronal to ensure innocent switching across representations (typically replacements) and to ensure simultaneous presentation of two or more representations. Thus, the sampling time base of the media contained in each representation within the self-adjusting HTTP streaming media presentation can be related to successive global isochrones across multiple segments.

Encoded media blocks containing multiple types of media (eg, audio and video) may have different presentation end times for different types of media. In a block request streaming system, such media blocks may be coherently broadcast in such a way that each media type is continuously played, and thus one type of media sample from one block may be in the previous block. Another type of media is sampled before it is broadcast, which is referred to herein as "continuous block splicing." Alternatively, such media blocks can be played in such a way that any type of earliest sample of a block plays after the last sample of any type of the previous block, which is referred to herein as a "discontinuous block". splice". Continuous block stitching may be appropriate when the two blocks contain sequentially encoded media from the same content item and the same representation or in other situations. Typically, within one representation, contiguous block splicing can be applied when splicing two blocks. This is advantageous because existing encodings can be applied and can be segmented without having to align the media tracks at the block boundaries. This is illustrated in Figure 10, where video stream 1000 includes block 1202 and other blocks with RAPs such as RAP 1204.

Media presentation description

Media rendering can be viewed as a structured set of files on an HTTP streaming server. The HTTP streaming client can download sufficient information to present to the user Streaming service. The replacement representation may include one or more 3GP files or 3GP files, wherein the 3GP files follow the 3GPP file format or at least follow a defined set of data structures that can be easily converted or converted to 3GP files.

The media presentation can be described by a media presentation description. The Media Presentation Description (MPD) may contain relay data that the client can use to construct a suitable file request, such as an HTTP access request, to access the data at the appropriate time and provide streaming services to the user. . The media presentation description provides sufficient information for the HTTP streaming client to select the appropriate 3GPP files and files. The unit that the signal delivery informs the client to access is called a segment.

The media presentation description may include, inter alia, the following elements and attributes, and the like.

Media presentation description element (MediaPresentationDescription)

An element that encapsulates the relay material used by the HTTP streaming client to provide streaming services to the end user. The Media Rendering Description element can contain one or more of the following attributes and elements.

Version: The version number of the agreement to ensure scalability.

Presentation Identifier: Information that enables the presentation to be uniquely identified among other presentations. It can also contain a private field or name.

UpdateFrequency: The frequency of update of the media presentation description, ie how often the client can reload the actual media presentation description. If the element does not appear, the media presentation can be static. Updating the media presentation can mean that the media presentation cannot be cached.

Media Presentation Description URI (MediaPresentationDescriptionURL): used to agree on the media presentation description URI.

Stream: Describes the type of streaming or media presentation: video, audio, or text. The video stream type can contain audio and can contain text.

Service: Describes the type of service with additional attributes. The type of service can be live or on-demand. This can be used to notify the client that viewing and access beyond a certain current time is not permitted.

Maximum client prebuffer time (MaximumClientPreBufferTime): The maximum amount of time the client can pre-buffer media streams. The time base distinguishes the stream from the progressive download that the client is limited to download beyond the maximum pre-buffer time. This value may not appear, indicating that no restrictions on pre-buffering are applicable.

SafetyGuard IntervalLiveService: Information about the maximum turnaround time of the live service on the server. Provides the client with instructions on what information is available at the current time. This information may be necessary if the client and server are expected to operate in UTC time and do not provide tight time synchronization.

Time Shift BufferDepth: Information about how far the client is moving back in the live service relative to the current time. With this depth extension, time-shifted viewing and tracking services can also be permitted without making specific changes in the service offering.

LocalCachingPermitted: This flag indicates whether the HTTP client can cache the data locally after the downloaded data has been played.

LivePreview Inverval: Contains the time interval available for presentation between the start time (StartTimes) and the end time (EndTimes). The "start time" indicates the start time of the service and the "end time" indicates the end time of the service. If no end time is specified, the end time is unknown at the current time, and the "update frequency" ensures that the client can access the end time before the actual end time of the service.

OnDemand Availability Interval: This presentation interval indicates the availability of the service on the network. Multiple presentation intervals can be provided. The HTTP client may not be able to access the service outside of any specified time window. Additional time shift viewing can be specified by providing an "on-demand interval". For live services, this property can also appear. In the event that this attribute occurs for live services, the server can ensure that the client can access the service in the form of an on-demand service during all available availability intervals. Therefore, the "live presentation interval" cannot be overlapped with any "on-demand availability interval".

MPD File Info Dynamic (MPDFileInfoDynamic): Describes the preset dynamic construction of a file in a media presentation. More details are provided below. Preset designation on the MPD level can save unnecessary duplication if the same rules are used for several or all of the replacement representations.

MPD Transcoder Description (MPDCodecDescription): Describes the main preset transcoder in the media presentation. More details are provided below. Preset designation on the MPD level can save unnecessary duplication if the same transcoder is used for some or all of the representations.

MPD mobile packet header size unchanged (MPDMoveBoxHeaderSizeDoexNotChange): indicates the mobile packet header The size of the logo in the entire media presentation between the various body files. This flag can be used to optimize the download and can only occur in the case of a particular segment format (especially where the segment format contains segments of the moov header).

FileURIPattern: A mode used by the client to generate a request message for a file within a media presentation. Different attributes permit a unique URI for each file within the media presentation. The base URI can be an HTTP URI.

Replacement representation: Description of the list of representations.

"AlternativeRepresentation" element: An XML element that encapsulates all of the relayed data represented. The Replace Representation element can contain the following attributes and elements.

Representation ID: The unique ID of the particular alternate representation within the media presentation.

Static Info (FilesInfoStatic): Provides an explicit list of the start time and URI of all files that are rendered. This static provisioning of the manifest can provide the advantage of having an exact time base description for the media presentation, but may not be compact enough, especially if the replacement representation contains many files. In addition, the file names can have any name.

FilesInfoDynamic: Provides an implicit way of constructing a list that replaces the start time and URI of the rendering. The dynamic provision of this list of files provides the advantage of having a more compact representation. If only the starting time series is provided, then the time base advantage is also true, but the file name will be dynamically constructed based on the file mode URI (FilePatternURI). If only the duration of each segment is provided, the representation is compact and can be adapted for use in live services, but the generation of files can be governed by a global time base.

The AP mobile packet header size is unchanged (APMoveBoxHeaderSizeDoesNotChange): a flag indicating whether the size of the mobile packet header is changed between the individual body names in the replacement description. This flag can be used to optimize the download and can only occur in the case of a particular segment format (especially where the segment format contains segments of the moov header).

AP Transcoder Description (APCodecDescription): A primary transcoder that describes the replacement of the file in the presentation.

Media description element

MediaDescription: An element that encapsulates all relay material for the media contained in the representation. In particular, the element may contain information about the trajectory in the alternate presentation and the recommended trajectory grouping (if applicable). The Media Description attribute contains the following attributes: TrackDescription: Encapsulates the XML attributes of all relayed material for the media contained in the representation. The Track Description attribute contains the following attributes: Track ID: Replaces the unique ID of the track within the representation. This can be used in situations where the trajectory is part of a group description.

Bitrate: The bitwise rate of the track.

TrackCodecDescription: Contains XML attributes for all attributes of the transcoder used in the track. The Track Transcoder Description property contains the following properties: MediaName: The property that defines the media type. Media types include "Audio", "Video", "Text", "Application" and "Message".

Codec: Transcoder type including profile and level.

LanguageTag: The language tag (if applicable).

Maximum width (MaxWidth), maximum height (MaxHeight): For video, it refers to the height and width of the included video in the primitive.

Sampling Rate: The sampling rate for audio.

GroupDescription: Provides the client with recommended attributes for the appropriate group based on different parameters.

GroupType: Based on this type, the client can decide how to group the tracks.

The information in the media presentation description is advantageously used by the HTTP streaming client to perform a request for a file/segment or file/segment portion at an appropriate time to select from, for example, access bandwidth, display capabilities, transcoder capabilities. And segments that match the competency representation of information capabilities in the sense of user preferences, such as language. In addition, since the "media presentation description" describes the time alignment and is mapped to the representation of the global isochronal line, the client can also use the information in the MPD to initiate appropriate actions during the ongoing media presentation to cross the representation. Switching, jointly presenting each representation, or viewing within the media presentation.

Signal delivery notification segment start time

Indicates that it can be split into multiple segments by time. There is an inter-track timebase problem between the last segment of a segment and the next segment of the next segment. Furthermore, in the case of using segments with a constant duration, there is another time base problem.

Use the same duration for each segment to have an MPD that is both compact and static The advantages. However, each segment may still start at a random access point. Thus, either the video encoding can be constrained to provide random access points at the particular points, or the actual segment duration may not be as general as specified in the MPD. It may be desirable for the streaming system to impose no unnecessary restrictions on the video encoding process, and thus the second option may be preferred.

Specifically, if the time history is specified as d seconds in the MPD, the nth file may start at a random access point at time (n-1)d or immediately after time (n-1)d.

In this approach, each file may include information about the exact start time of the segment in the sense of a global presentation time. Three possible ways to signal this point are:

(1) First, the start time of each segment is limited to the exact time base as specified in the MPD. However, the media encoder may not have any flexibility in the placement of the IDR frame and the stream may require special encoding.

(2) Second, add the exact start time to the MPD for each segment. For the on-demand case, the compactness of the MPD may be reduced. For live situations, this may require periodic updates to the MPD, which reduces scalability.

(3) Third, adding a global time to the segment in the MPD or an exact start time relative to the declared start time of the representation or the declared start time of the segment, the meaning added to the segment means that the segment contains the information. This information can be added to a new package dedicated to self-tuning streaming. The package may also include information as provided by the "TIDX" or "SIDX" package. The result of this third approach is that when viewing a particular location near the beginning of one of the segments, the client can select a subsequent segment of the segment containing the requested viewpoint based on the MPD. A simple response in this case may be to move the view point forward to the beginning of the retrieved segment (ie, after moving to the view point) Next random access point). In general, random access points are provided at least every few seconds (and making random access points less frequent often results in almost no coding gain) and therefore in the worst case, the view points can be moved to a later than specified second. Alternatively, the client may determine that the requested view point is actually in the previous segment and instead requests the segment when retrieving the header information of the segment. This may result in an increase in the time required to perform the viewing operation from time to time.

List of accessible segments

The media presentation includes a set of representations, each representation providing an encoding of a different version of the original media content. The representations themselves advantageously contain information about the distinguishing parameters of the representation compared to other parameters. The representations also explicitly or implicitly contain a list of accessible segments.

A segment can be distinguished as a time-independent segment containing only relay data and a media segment containing primarily media material. The Media Presentation Description ("MPD") advantageously identifies each segment and assigns each segment a different attribute, either implicitly or explicitly. The attributes that are advantageously assigned to each segment include the time period during which the period is accessible, the resources and agreements through which the segment can be accessed. In addition, the media segment is advantageously assigned attributes such as the start time of the segment in the media presentation, and the duration of the segment in the media presentation.

Where the media presentation is generally of the "on-demand" type as indicated by attributes in the media presentation description, such as on-demand availability intervals, the media presentation description typically describes the entire segment and also provides when the segments are accessible and An indication of when the segments are not accessible. The start time of the segment is advantageously expressed relative to the beginning of the media presentation, so that two clients starting to replay the same media presentation at different times can use the same media presentation description and the same Media segment. This advantageously increases the ability to cache the segments of the memory.

Where the media presentation is generally of the "live" type as indicated by an attribute (such as a " service " attribute) in the media presentation description, then the segment of the portion of the media presentation that exceeds the actual time is generally not generated or at least not Access, although the segments are fully described in the MPD. However, with the media presentation service being a "live" type of indication, the client can generate access to the client's internal time "now" based on the wall clock based on the information contained in the MPD and the download time of the MPD. A list of simultaneous base attributes. The server advantageously operates in the sense that the resource is accessible so that the reference client operating with the instance of the MPD at the wall clock time can access the resource.

Specifically, the reference client generates a list of segment-based base attributes accessible to the client internal time "now" based on the wall clock time based on the information contained in the MPD and the download time of the MPD. As time progresses, the client will use the same MPD and will create a new list of accessible segments that can be used to continuously broadcast the media presentation. Therefore, the server can announce the segments in the MPD before the segments can actually be accessed. This is advantageous because it reduces frequent updates and downloads of the MPD.

It is assumed that a list of segments each having a start time tS is implicitly described via a playlist in an element such as "Static File Information" or by using an element such as "dynamic file information". The following describes an advantageous way to use the "dynamic file information" to generate a list of segments. Based on this construction rule, the client can access a list of each URI representing r (referred to herein as FileURI( r , i )) and a start time tS ( r , i ) of each segment indexed i .

The use of information in the MPD to establish a segment's accessible time window can be performed using the following rules.

For a service of the general type "on-demand" as indicated by an attribute such as " service ", if the current wall clock time at the client "now" falls, such as advantageously by an "on-demand availability interval" All of the described segments of the on-demand presentation are accessible within any usability range of the MPD element representation of the class. If the current wall clock time at the client "now" falls outside of any availability range, the described segments of the on-demand presentation are not accessible.

For services that are generally "live" as indicated by attributes such as " services ", the start time tS ( r , i ) advantageously expresses the availability time in wall clock time. The availability start time can be derived as a combination of the live service time of the event and some turnaround time at the server for capture, encoding, and publishing. The time of the process can be specified, for example, in the MPD, for example, using a security protection interval tG designated as "security zone live service" in the MPD. This will provide the smallest difference between UTC time and data availability on the HTTP streaming server. In another embodiment, the MPD explicitly specifies the availability time of the segments in the MPD without providing a turnaround time as the difference between the event live time and the turnaround time. In the following description, it is assumed that any global time is specified as the availability time. A person of ordinary skill in the field of live media broadcasting may derive this information from the appropriate information in the media presentation description after reading this description.

If the current wall clock time at the client "now" falls outside any range of the live presentation interval that is advantageously expressed by an MPD element such as a "live presentation interval", then the described segments of the live presentation are Can not be saved In the event that the current wall clock time at the client "now" falls within the live presentation interval, then at least some of the segments of the live presentation may be accessible.

The limit on the accessible segment is governed by the following values: wall clock time "now" (as available to the client).

For example, the allowed time shifting buffer depth tTSB specified as "time shifting buffer depth" in the media presentation description.

The client may only be allowed to allow the request start time tS ( r , i ) to fall in the (now - tTSB) to "now" interval or the end time of the segment that makes the duration d to be included in the relative event time t ₁ (There is a segment in the interval of the interval (now -tTSB-d) to "now").

Update MPD

In some embodiments, the server does not know the segment or segment locator and start time of the segment beforehand because, for example, the server location will change, or the media presentation includes some advertisements from different servers, or the duration of the media presentation is unknown. , or the server wants to confuse the locator of the successor segment.

In such an embodiment, the server may only describe segments that are already accessible or accessible shortly after the instance of the MPD is published. Moreover, in some embodiments, the client advantageously consumes media that is close to the media described in the MPD such that the user experiences the media program contained as close as possible to the production of the media content. Once the client expects to arrive at the end of the media segment described in the MPD, the client advantageously requests a new instance of the MPD to continue the continuous playout as expected by the server to publish a new MPD describing the new media segment. The server advantageously generates a new instance of the MPD and updates the MPD to make the client The terminal can rely on these programs for continuous updates. The server can make its own MPD update program along with the segment generation and release of the reference client program that adapts to the behavior of the client, such as the behavior of the normal client.

If the new instance of the MPD describes only a short amount of time advancement, the client needs to frequently request a new instance of the MPD. This can lead to scalability issues and unnecessary uplink and downlink traffic due to unnecessary frequent requests.

Therefore, it is relevant to describe on the one hand the segment that enters the future as far as possible without having to make the segments accessible, and on the other hand to make the unforeseen updates in the MPD express the new server location, permit insertion such as New content such as advertisements or changes to transcoder parameters.

Moreover, in some embodiments, the duration of the media segment may be small, such as in the range of a few seconds. The duration of the media segment is advantageously flexible in order to adjust to a suitable segment size that can be optimized for delivery or cache memory properties, to compensate for end-to-end delays in live service, or to cope with other aspects of segment storage or delivery. Or for other reasons. Especially in situations where the segment is small compared to the media presentation duration, then a significant amount of media segment resources and start time need to be described in the media presentation description. As a result, the size of the media presentation description may be large, which can adversely affect the download time of the media presentation description and thus affect the startup delay of the media presentation and also the bandwidth usage on the access link. Therefore, it is advantageous to not only permit the use of a playlist to describe a list of media segments, but also to permit descriptions by using a template or URL construction rules. Template and URL rule rules are used synonymously in this description.

In addition, the template can advantageously be used to describe beyond in a live situation. The segment locator of the current time. In such cases, the update to the MPD itself is not necessary because the locator and the list of segments are described by the template. However, unforeseen events may still occur, which require changes to the description of the representation or segment. When content from multiple different sources is stitched together, such as when an advertisement is inserted, it may be necessary to self-adjust the changes in the HTTP streaming media presentation description. Content from different sources may vary in various ways. Another reason during live presentation is that it may be necessary to change the URL for the content file to provide for fault tolerance transfer from one live origin server to another.

In some embodiments, it may be advantageous if the MPD is updated so that the updated MPD is compatible with the previous MPD, a compatible meaning means any for up to the effective time of the previous MPD The time, the reference client and thus any implemented client's list of accessible segments generated from the updated MPD is equivalent to the list of accessible segments that the client will generate from this previous instance of the MPD. This requirement ensures that (a) the client can immediately start using the new MPD without having to synchronize with the old MPD because the new MPD is compatible with the old MPD before the update time; and (b) the update time does not need to occur with the actual change to the MPD. Time synchronization. In other words, updates to the MPD can be advertised in advance, and once new information is available, the server can replace the old instance of the MPD without having to maintain a different version of the MPD.

There are two possibilities for media time bases that span MPD updates for a set of representations or all representations. (a) the existing global isochronal line continues over the MPD update (referred to herein as "continuous MPD update"), or (b) the current isochron is over and the new isochron is from the segment following the change (This is called "non-continuous MPD update" in this article).

The difference between these options may be significant in view of the fact that the trajectory of the media segment and thus the trajectory of the segment are generally not starting and ending at the same time due to differences in sampling granularity across the trajectories. During normal rendering, the sampling of one track of a segment may be rendered prior to some sampling of another track of the previous segment, ie there may be some overlap between the segments, although there may be no overlap within a single track.

The difference between (a) and (b) is whether it is permissible to implement such overlap across MPD updates. Such overlap is generally difficult to achieve when the MPD update is due to the splicing of completely separate content, as the new content requires a new encoding to be stitched with the previous content. It is therefore advantageous to provide the ability to non-continuously update the media presentation by restarting the isochronous line for certain segments, and possibly also to define a new set of representations after the update. Moreover, if the content has been independently encoded and segmented, it is also avoided that the timestamp is adjusted to fit within the global isochronal line of the previous piece of content.

Overlap and continuous updates may be allowed when the update is for secondary reasons, such as merely adding a new media segment to the list of described media segments, or if the location of the URL is changed.

In the case of a non-continuous MPD update, the last indicated isochronal end of the last segment ends at the latest presentation end time of any samples in the segment. The isochronal line of the next representation (or more accurately, the first presentation time of the first media segment of the new portion of the media presentation, also referred to as the new time period) typically and advantageously begins with the presentation of the previous time period The end of the same moment to ensure innocent and continuous broadcast.

These two scenarios are illustrated in FIG.

It is preferred and advantageous to limit the MPD update to segment boundaries. Put this class The basic principle of changing or updating to the segment boundaries is as follows. First, a change to each of the represented binary relay data (typically the movie header) can occur at least at the segment boundaries. Second, the media presentation description can include metrics (URLs) that point to segments. In a sense, an MPD is an "umbrella" data structure that groups all the segments associated with the media presentation. To maintain this containment relationship, each segment can be referenced by a single MPD, and when the MPD is updated, it is advantageous to update the MPD only at the segment boundaries.

Segment boundary alignment is generally not required, however, for the case of content spliced from different sources and generally for non-continuous MPD updates, it makes sense to align the segment boundaries (specifically, the last segment of each representation can be the same The video frame ends and cannot contain audio samples whose presentation start time is later than the presentation time of the frame. Non-continuous updates can then begin a new set of representations at a common time (called a time period). The start time of the validity of the new set of representations is provided, for example, by the start time of the time period. The relative start time of each representation is reset to zero and the start time of the time period places the set of representations in the new time period in the global media presentation isochron.

For continuous MPD updates, segment boundary alignment is not required. Each segment of each alternate representation can be hosted by a single media presentation description, and thus an update request for the new instance of the description is presented to the media (these update requests are generally not expected to be described in this working MPD due to the absence of additional media segments) Triggered) Depending on the group representation consumed, it can occur at different times, where the group representation includes the set of representations that are expected to be consumed.

In order to support the update of MPD elements and attributes in a more generalized case, not only a representation or a set of representations, but any element can be associated with the validity time. Association. Thus, if certain elements of the MDP need to be updated, such as where the number of representations has changed or the URL construction rules have changed, then the non-intersecting validity times are provided for multiple copies of the elements, each of which can be specified The time is updated separately.

The validity is advantageously associated with the global media time such that the described elements associated with the validity time are valid for the time period of the global isochronal of the media presentation.

As discussed above, in one embodiment, the validity time is only added to the full group representation. Each full group constitutes a time period. The validity time then constitutes the start time of the time period. In other words, in the specific case of using the validity element, the full group representation can be valid for a period of time indicated by a set of global validity times. The validity time of a set of representations is called the time period. At the beginning of the new time period, the validity of the previous set of representations expires and the new set represents valid. Note again that the validity period is preferably disjoint.

As described above, the change to the media presentation description occurs at the segment boundaries, and thus for each representation, the element change actually occurs at the next segment boundary. The client can then form a valid MPD, which includes a list of segments at each moment in the presentation time of the media.

In situations where the block contains media material from different representations or from different content (eg, from content segments and advertisements) or in other situations, non-contiguous tile stitching may be appropriate. In a block request streaming system it may be required that changes to the presentation relay material only occur at the block boundaries. This may be advantageous for implementation reasons, as updating media decoder parameters within a block may be more complicated than updating the parameters only between blocks. In this case Advantageously, it may be specified that the validity interval as described above may be interpreted as approximate such that the element is considered to be from a first block boundary that is not earlier than the beginning of the specified validity interval to no earlier than specified The first block boundary at the end of the validity interval is valid.

A novel enhanced example embodiment of the block request stream system described above is described in the section entitled "Changes to Media Presentation" provided later.

Segment duration signal transmission

A non-continuous update effectively divides the presentation into a series of disjoint intervals called time periods. Each time period has its own isochronal time for the media sampling time base. The media time base of the representations within the time period may advantageously be indicated via a separate compact list specifying the duration of the segments represented by each of the time periods or time periods.

An attribute associated with an element within the MPD, such as a time period start time, may specify the validity time of certain elements during the media presentation time. This attribute can be added to any element of the MPD (the element can be replaced with an attribute that can be assigned validity).

For non-contiguous MPD updates, all represented segments can end at non-contiguous points. This generally means at least that the last segment before the discontinuous point has a different duration than the previous segment. Signaling the notification segment duration may involve indicating that all segments have the same duration or indicating a separate duration for each segment. It may be desirable to have a compact representation of the segment duration list, which is efficient in situations where there are many segments with the same duration.

The duration of each representation or group of representations can advantageously be implemented with a single string that specifies the beginning of the discontinuous update (ie, the beginning of the period) up to the last media segment described in the MPD. All segments of a single interval up to the duration. In one embodiment, the format of the element is a literal string that conforms to the production of the list of segment duration entries, wherein each entry includes a duration attribute dur and an optional multiplier mult of the attribute, the multiplier indicating the Represents a segment of < mult > containing the first entry, a < dur > that lasts for the first entry, followed by a < mult > of the second entry, a segment of < dur > that lasts for the second entry, and so on.

Each diachronic entry specifies the duration of one or more segments. If the < dur > value is followed by a " ^* " character and a digit, the digit specifies the number of consecutive segments with that duration (in seconds). If the multiplier symbol " ^* " does not exist, the number of segments is 1. If there is a " ^* " without a successor digit, then all subsequent segments have the specified duration and there may be no further entries in the list. For example, the string "30 ^* " means that all segments have a duration of 30 seconds. The string "30 ^* 12 10.5" indicates that there are 12 segments lasting 30 seconds, followed by a segment with a duration of 10.5 seconds.

If the segment duration is specified separately for each replacement representation, the sum of the segment durations within each interval may be the same for each representation. In the case of a video track, the interval may end in the same frame in each of the alternate representations.

One of ordinary skill in the art, upon reading this disclosure, will find similar and equivalent ways of expressing segment duration in a compact manner.

In another embodiment, the signal delivery notification is signaled by the signal "duration attribute <duration>". The duration of the segment is constant for all segments in the representation except for the last segment. The last paragraph before the non-continuous update can be shorter, as long as the start point of the next discontinuous update or the start of the new period is provided That's it, and this means the last delay and the beginning of the next period.

Change and update of the relay data

The binary-represented representation of the relay data, such as a change in the movie header "moov", can be done in different ways: (a) in a separate file referenced in the MPD, there can be one moov packet for all representations. (b) In each individual file referenced in each replacement representation, there may be one moov package for each replacement representation, (c) each segment may contain a moov package and is therefore self-contained, (d) in conjunction with the MPD There can be a moov package for all representations in a 3GP file.

Note that in the case of (a) and (b), it is advantageous to combine a single "moov" with the concept of validity from above, which means that more "moov" packets can be referenced in the MPD. As long as the validity of these "moov" packages does not intersect. For example, with the definition of a time period boundary, the validity of the 'moov' in the old time period may expire as the new time period begins.

In the case of option (a), a reference to a single moov package can be assigned a validity element. Multiple presentation headers can be allowed, but only one presentation header can be active at a time. In another embodiment, the full set of representations in the time period as defined above or the validity time of the entire time period may be used as the validity time of the representation relay material, typically provided as a moov header.

In the case of option (b), a reference to each represented moov package can be assigned a validity element. Multiple representation headers may be allowed, but only one representation per time is valid for the header. In another embodiment, the validity time of the entire representation or the entire time period as defined above may be used as the validity time of the representation relay material, typically provided as a moov header.

In the case of option (c), you can not add signal passing in the MPD. However, additional signal passing can be added to the media stream to indicate whether the moov packet will change for any upcoming segments. This is further explained in the context of this section below, "Updates in Signal Delivery Notification Segment Relay Data".

Signal delivery notification segment update in the relay data

In order to avoid frequent updates to the media presentation description to gain knowledge about potential updates, it is advantageous to signal any such updates along with the media segment. Additional one or more elements may be provided within the media segment itself, which may indicate that updated relay material (such as a media presentation description) is available and must be accessed within a certain amount of time to successfully continue to establish A list of segments can be accessed. In addition, for updated relay profiles, such elements may provide a file identifier (such as a URL) or information that may be used to construct a file identifier. The updated relay profile may include relay data equal to the relay data provided in the original relay profile of the presentation to the indication validity interval, along with additional relay material also accompanied by the validity interval. Such an indication may be provided in a media segment of all available representations of the media presentation. The client of the access block request stream system may use the file download protocol or other means to retrieve the updated relay data file upon detecting such indication within the media block. This provides the client with information about the changes in the media presentation description and the time at which the changes will occur or have occurred. Advantageously, each client requests an updated media presentation description once only when such a change occurs, rather than "polling" and receiving the file many times to obtain a possible update or change.

Examples of changes include additions or removals of representations, changes to one or more representations, such as bit element rate, resolution, aspect ratio, included trajectory or transcoder parameters, and rules for URL construction. Change, for example The original server is sent to the different advertisements. Some changes may only affect the initialization segment associated with the representation, such as the movie header ("moov") atom, while other changes may affect the media presentation description (MPD).

In the case of on-demand content, the time bases of such changes and such changes may be known in advance and may be signaled in the media presentation description.

For live content, the change may be unknown until the point in time when the change occurred. One solution is to allow the media presentation descriptions available at a particular URL to be dynamically updated and require the client to periodically request the MPD to detect the change. This solution has drawbacks in terms of scalability (original server load and cache memory efficiency). In scenarios with a large audience, the cache may receive many requests for MPDs after the previous version of the MPD has expired from the cache and before the new version is received, and all such requests may be forwarded Give the original server. The original server may need to constantly process requests from the cache for each updated version of the MPD. Moreover, such updates may not be easily aligned in time with changes in media presentation.

Since one of the advantages of HTTP streaming is the ability to leverage standard web infrastructure and services for scalability, a preferred solution may involve only "static" (ie, cacheable) files. Depends on the client "polling" file to see if the files have changed.

A solution for addressing updates of relay data including media presentation descriptions and binary representation relay data (such as self-adjusting "moov" atoms in HTTP streaming media presentations) is discussed and proposed.

For the case of live content, you may not know when constructing MPD The point in time when MPD or "moov" may change. Since frequent "polling" of the MPD should be avoided to check for updates for reasons of bandwidth and scalability, the update to the MPD can be indicated "in-band" in the segment file itself, ie each media segment can have Indicates the option to update. Depending on the segment format (a) to segment format (c) above, the signal can be communicated to notify different updates.

In general, it may be advantageous to provide an indication in the signal within the segment that the MPD may be updated before the start time of the next segment or segment within the request is requested to be greater than any next segment of the start time of the current segment. The update can be announced in advance to indicate that the update only needs to occur in any segment that is later than the next segment. In the case where the locator of the media segment is changed, the MPD update can also be used to update the binary representation of the relay material, such as the movie header. Another signal may indicate that when the segment is completed, no more segments that advance the time should be requested.

In the case where the segment is formatted according to the segment format (c), that is, in the case where each media segment can contain self-initializing relay data such as a movie header, another signal can be added to indicate that the subsequent segment contains Updated movie header (moov). This advantageously allows the movie header to be included in the segment, but the movie header only needs to be served by the client if the previous segment indicates a movie header update or in the case of a view or random access when switching representations Request. In other cases, the client may issue a byte range request for the segment that requests the download of the movie header to be excluded, thus advantageously saving bandwidth.

In another embodiment, if the signal delivery notifies the MPD update indication, the signal may also include a locator such as a URL for the updated media presentation description. In the case of non-continuous updates, the updated MPD may use validity attributes such as new time periods and old time periods to describe before and after the update. Presented. This may advantageously be used to permit time-shifted viewing as further described below, but also advantageously allows the MPD update to be signaled at any time prior to the change included in the MPD update taking effect. The client can immediately download the new MPD and apply the new MPD to the ongoing presentation.

In a specific implementation, the signal delivery notification for any changes to the media presentation description, moov header, or presentation end may be included in the streaming packet formatted in accordance with the rules of the segment format of the packet structure using the ISO base media file format. in. This package provides a special signal for any different update.

Streaming packet

definition

Package type: ‘sinf’

Container: none

Mandatory: No

Quantity: 0 or 1.

The streaming packet contains information about what part of the stream is presented.

syntax

Semantics

Streaming_information_flags (streaming information flag) contains logical OR of 0 or more of the following: 0x00000001 followed by movie header update

0x00000002 rendering description update

0x00000004 rendering ends

The mpd_location (mpd position) appears if and only if the Presentation Description update flag is set, and the mpd_location (mpd location) provides a uniform resource locator for the new media presentation description.

Example use case for MPD updates for live services

It is assumed that the service provider wants to provide live football events using the enhanced block request stream described herein. Perhaps millions of users may want to access the presentation of the event. The live event is interrupted by a break when the request is suspended or other breaks in the action, during which an advertisement can be inserted. Typically, there is no or almost no prior notice of the exact time base for the break.

Service providers may need to provide redundant infrastructure (eg, encoders and servers) to enable seamless switching in the event of any component failure during a live event.

Assume that the user Anna accesses the service on her bus with her mobile device and the service is immediately available. Next to her is another user, Paul, who watches the event on his laptop. The ball was scored and two people celebrated the event at the same time. Paul told Anna that the first ball in the game was even more exciting and that Anna used the service so she could look back at the event 30 minutes ago. After watching the goal, she returned to the live game.

In order to resolve this use case, the service provider should be able to update the MPD, communicate to the client signal that the updated MPD is available, and permit the client to access the streaming service to enable the MPD to present the data in near-immediate manner.

It is possible to update the MPD in a non-synchronous manner with segment delivery, as explained elsewhere herein. The server can provide the receiver with a guarantee that the MPD will not be updated for a certain period of time. The server can depend on the current MPD. However, no explicit signaling is required when the MPD is updated before a certain minimum update period.

Fully synchronized playouts are almost impossible to achieve because the client may be operating on different MPD update instances so the client may drift. With MPD updates, the server can communicate changes and the client can be reminded of changes, even during presentation. In-band signal delivery on a piecemeal basis can be used to indicate an update of the MPD, so updates may be limited to segment boundaries, but this should be acceptable in most applications.

An MPD element can be added that provides the release time of the MPD in wall clock time and an optional MPD update package added at the beginning of the segment to signal the MPD update with signal delivery notification. This update can be performed hierarchically like an MPD.

MPD "Publish Time" provides the unique identifier of the MPD and when the MPD is sent. The MPD "release time" also provides an anchor for updating the program.

The MPD update package may exist after the "styp" packet in the MPD and is defined by the packet type = "mupe", does not require a container, is not mandatory, and has a number of 0 or 1. The MPD update package contains a section on the media presentation of the segment. Information about points.

The example sentence is as follows:

The semantics of the various objects of the MPDUpdateBox class can be as follows: mpd_information_flags: Logic OR of zero or more of the following:

i. 0x00 current media presentation description update

Ii. 0x01 Future Media Presentation Description Update

Iii. 0x02 end of presentation

Iv. 0x03-0x07 Reserved

New_location flag: If set to 1, a new media presentation description is available at the new location specified in mpd_location (mpd location).

Latest_mpd_update time: Specifies the time (in ms) at which the MPD update must be performed at the latest relative to the latest MPD's MPD issue time. The client can choose between any time between the client and the present Update the MPD.

Mpd_location (mpd location): Appears if and only if the "new location flag" is set, and if so, the "mpd location" provides a uniform resource locator for the new media presentation description.

If the bandwidth used by the update is a problem, the server can supply MPDs for certain device capabilities such that only those portions are updated.

Time-shifted viewing and network PVR

When time-shifted viewing is supported, it may happen that two or more MPDs or movie heads are valid during the lifetime of the communication period. In this case, by updating the MPD when necessary, but adding a validity mechanism or a time period concept, an effective MPD can exist for the entire time window. This means that the server can ensure that any MPD and movie heads are declared at any time period that falls within the valid time window for time-shifted viewing. It is effective for the client to ensure that the available MPD and relay data for the client's current rendering time is valid. It is also possible to support the migration of live communication periods to network PVR communication periods using only a small number of MPD updates.

Special media segment

A problem when using the file format of ISO/IEC 14496-12 within a block request streaming system is that, as described above, storing a single version of the media material presented may be stored in multiple files arranged in consecutive time periods. It is beneficial. Furthermore, it may be advantageous to arrange each file to start at a random access point. Furthermore, it may be advantageous to select the location of the view point during the video encoding process and segment the presentation into a plurality of files each starting at the view point based on the selection of the view points made during the encoding process, each of which is randomly stored Take points can be placed The beginning of the file may also not be placed at the beginning of the file, but each of the files starts at the random access point. In an embodiment having the above-described properties, the presentation relay material or media presentation description may include the exact duration of each file, wherein the duration of the video media, for example, that is considered to represent the file, and the start time of the video media of the next file, Difference. Based on the information in the presentation relay material, the client can construct a mapping between the global isochronal line of the media presentation and the local isochronal line of the media within each file.

In another embodiment, the size of the presentation relay material may advantageously be reduced by specifying that each file or segment has the same duration. However, in such a situation and where the media file is constructed according to the above method, the duration of each file may not be strictly equal to the duration specified in the media presentation description, since the designation has just passed since the beginning of the file. There may be no random access points at the point in time.

Yet another embodiment of the present invention is now described for enabling proper operation of the block request stream system even in the case of the above-mentioned contradictions. In this method, an element may be provided within each file that specifies a local isochron of the media within the file (the local isochron of the media within the file refers to the slave according to ISO/IEC 14496-12) The isochronous line that begins with timestamp 0 and which is used to specify the decoding of the media samples in the file and the isochronous time of the composite timestamp) presents the mapping of the isochrones to the global domain. The mapping information may include a single timestamp corresponding to a zero timestamp in the local file isochronal time in the global rendering time. The mapping information may alternatively include an offset value that specifies that the global presentation time corresponding to the zero time stamp in the local time isochronal line corresponds to the start of the same file based on the information provided in the presentation relay data. The difference between the global presentation times.

Examples of such packets may be, for example, a track segment decoding time ('tfdt') packet or a track segment adjustment packet ('tfad') along with a track segment media adjustment ('tfma') packet.

Instance client generated by segment list

The example client will now be described. The client can be used as a reference client for the server to ensure proper generation and update of the MPD.

The HTTP streaming client is guided by the information provided in the MPD. It is assumed that the client can access the MPD received by the client at time T (i.e., when the client can successfully receive the MPD). Determining successful reception may include the client obtaining an updated MPD or the client verifying that the MPD has not been updated since the previous successful reception.

The example client behavior is described below. In order to provide the user with a continuous streaming service, the client first parses the MPD and establishes the current system time for each representation, taking into account the possible use of playlists or segment list generation procedures using URL construction rules as detailed below. A list of segments that the client can access at the local time. The client then selects one or more representations based on the information in the representation attribute and other information such as available bandwidth and client capabilities. Depending on the grouping, the representation can be rendered ad hoc or in conjunction with other representations.

For each representation, the client obtains binary relay data (if any), such as the "moov" header of the representation, and the media segment of the selected representation. The client accesses the media content via a byte array that may use a segment list or the like to request a segment or segment. The client may initially buffer the media before starting the presentation, and once the presentation has begun, the client continues to consume the media content by continuously requesting a segment or segment portion while accounting for the MPD update program. .

The client can switch the representation with account of updated MPD information and/or updated information from the client environment (eg, changes in available bandwidth). The client can switch to a different representation with any request for a media segment containing random access points. In the forward movement, that is, the current system time (referred to as "current time", used to indicate the time relative to the presentation), the client consumes the accessible segments. With each advance in "current time", it is possible for the client to extend the list of accessible segments for each representation according to the procedure specified herein.

If the media presentation has not yet arrived and if the current replay time falls within the threshold of the media in the media described in the MPD of the representation that is being consumed or will be consumed by the client, the client may request an update of the MPD, This update has a new retrieval time "Receive Time T". Once received, the client then considers the likely updated MPD and new time T to generate a list of accessible segments. Figure 29 illustrates a procedure for live service at different times at the client.

Accessible segment list generation

It is assumed that the HTTP streaming client can access the MPD and may want to generate a list of segments that are accessible for the wall clock time "now". The client synchronizes to the global time base with some precision, but advantageously does not require direct synchronization to the HTTP streaming server.

The list of accessible segments for each representation is preferably defined as a list pair of segment start times and segment locators, without loss of generality, the segment start time may be defined as relative to the beginning of the representation. The beginning of the representation can be compared to the beginning of the time period Standard (if the concept is applied). Otherwise, the indication begins at the beginning of the media presentation.

The client constructs rules and time bases using, for example, URLs as further defined herein. Once the list of described segments is obtained, the list is further limited to accessible segments, which may be a subset of the segments of the full media presentation. This construct is governed by the current value of the clock at the client's "now" time. In general, segments are only available for any "now" time within a set of availability times. For the "now" time that falls outside the window, no segment is available. In addition, for live services, it is assumed that a certain time "checktime" provides information about how far this media is described to enter the future. The "check time" is defined on the media timeline recorded by the MPD; when the client's replay time reaches the check time, the client advantageously requests a new MPD. When the client's replay time reaches the check time, it advantageously requests a new MPD.

Subsequently, the segment list is further limited by the check time along with the MPD attribute TimeShiftBufferDepth (time shift buffer depth) so that the available media segment only has the sum of the start time and the start time of the media segment falling into the "now" minus the time shift buffer. Depth subtracts the segments of the interval between the last described segment and the inspection time or the smaller of the "now" segments.

Scalable block

Sometimes, the available bandwidth drops so low that one or more blocks currently being received at the receiver become less likely to be fully received in time for broadcast without pause. The receiver may detect such a situation in advance. For example, the receiver can decide that it is receiving a block of 5 units of media coded every 6 units of time, and has a buffer of 4 units of media, so the receiver can It can be expected that the presentation will have to be stalled or paused to a time of approximately 24 units. With sufficient attention to this, the receiver can react to such situations via, for example, abandoning the current block stream and begin requesting different representations from the content (such as less time per unit playout time). One or more blocks of representation of the bandwidth. For example, if the receiver switches to a representation in which at least 20% of the video time encoded by the block is for a block of the same size, the receiver may be able to eliminate the need for stagnation until the bandwidth situation is improved.

However, it may be wasteful for the receiver to completely discard the data that has been received from the abandoned representation. In an embodiment of the block stream system described herein, the data within each block can be encoded and arranged in such a way that certain first codes of data within the block can be used to have not received the The presentation continues with the rest of the block. For example, well-known techniques for scalable video coding can be used. Examples of such video coding methods include time scalability of H.264 Scalable Video Coding (SVC) or H.264 Advanced Video Coding (AVC). Advantageously, the method allows the presentation to proceed based on the received portion of the block, even if the reception of one or more blocks may be discarded, for example due to a change in the available bandwidth. Another advantage is that a single profile can be used as a source for multiple different representations of the content. This is possible, for example, via an HTTP partial acquisition request that utilizes a subset of the selected blocks corresponding to the required representation.

One improvement detailed in this article is the enhanced segment: the scalable segment map. The scalable segment map contains the locations of the different layers in the segment so that the client can access portions of the segments and extract the layers accordingly. In another embodiment, the media material in the segments is ordered such that the quality of the segments is also increasing as the data is gradually downloaded from the beginning of the segment. In another embodiment, the quality is gradually improved It is applied to each block or fragment contained in the segment to enable fragment requests to resolve the scalable approach.

FIG. 12 is a diagram illustrating an aspect of a scalable block. In the figure, the transmitter 1200 outputs the relay data 1202, the scalable layer 1 (1204), the scalable layer 2 (1206), and the scalable layer 3 (1208), wherein the latter is delayed. Receiver 1210 can then render media presentation 1212 using relay material 1202, scalable layer 1 (1204), and scalable layer 2 (1206).

Independent scalability layer

As explained above, it is undesirable for the block request streaming system to have to stall when the receiver is unable to receive the requested portion of the particular representation of the media material in time for the receiver to broadcast, as this often results in a bad user experience. . By limiting the data rate of the selected representation to be much smaller than the available bandwidth to make the presentation less likely that any given portion will not be received in time, stagnation can be avoided, reduced or mitigated, but the strategy is Having media quality is inevitably much lower than the quality of the media that can be supported in principle. A presentation that is lower than the quality that may be achieved may also be interpreted as a poor user experience. Therefore, designers of block request streaming systems face the following choices when designing client programs, client programming, or hardware settings: either request a content version with a much lower data rate than the available bandwidth, here In this case, the user may suffer from poor media quality; or request a version of the content having a data rate close to the available bandwidth, in which case the user will suffer a pause with a high probability of changing with the available bandwidth during presentation.

To handle such situations, the block streaming system described herein can be configured to independently handle multiple scalability layers to enable the receiver to make The layered request and the emitter can respond to the layered request.

In such an embodiment, the encoded media material for each tile may be divided into a plurality of disjoint slices (referred to herein as "layers") such that the layers combine to form the entire media material of the block and A client that has received certain subsets of the layers may perform decoding and rendering of the representation of the content. In this approach, the ordering of the data in the stream causes the contiguous range to be incremental in quality and the relay data to reflect that point.

An example of a technique that can be used to generate a layer having the above properties is, for example, the scalable video coding technique described in the ITU-T standard H.264/SVC. Another example of a technique that can be used to generate a layer having the above properties is the temporal scalability layer technique as provided in the ITU-T standard H.264/AVC.

In such embodiments, the relay material may be provided in the MPD or in the segment itself, thereby enabling construction of individual layers and/or layer combinations for any given block and/or a given layer of multiple blocks. And/or a request for a layer combination of multiple blocks. For example, the layers that make up the block can be stored in a single file and can provide relay data specifying the range of bytes within the file that correspond to the individual layer.

A file download protocol (eg, HTTP 1.1) capable of specifying a bit range can be used to request an individual layer or multiple layers. Moreover, as will be apparent to those skilled in the art upon review of this disclosure, the above-described techniques relating to the construction, request and download of variable size blocks and variable block combinations are also applicable to this context.

combination

Several embodiments are now described, which may advantageously be employed by a tile request streaming client to use media divided into layers as described above. Physical data to achieve improvements in user experience compared to prior art and/or reduction in service infrastructure capacity requirements.

In a first embodiment, a known technique of a block request stream system is applied, the modification of which is that different versions of the content are replaced in some cases by different combinations of layers. In other words, where an existing system can provide two distinct representations of content, the enhanced system described herein can provide two layers, where one representation of the content in the existing system is at bit rate, quality, and possibly other. The metric aspect is similar to the first layer in an enhanced system, while the second representation of content in an existing system is similar to the combination of the two layers in an enhanced system in terms of bit rate, quality, and possibly other metrics. Therefore, the required storage capacity within the enhanced system is reduced compared to the required storage capacity in existing systems. In addition, a client of an existing system can issue a request for a block of one representation or another representation, and a client of the enhanced system can issue a request for a first or two layer of the block. Therefore, the user experience in the two systems is similar. In addition, improved cache memory is provided because even for different qualities, a shared segment is used, whereby the segment is more highly cached.

In a second embodiment, a client in an enhanced block request stream system employing the layer method now described may maintain separate data buffers for each of several layers of media encoding. As will be apparent to those skilled in the art of data management within a client device, the "separate" buffers may be assigned a physically or logically separate memory region for the separate buffers or Implemented by other techniques in which the buffered material is stored in a single or multiple memory regions and the separation of data from different layers is It is logically achieved by using a data structure containing references to storage locations of data from separate layers, and thus the term "separate buffer" should be understood to include that the data of the different layers can be separated. Any method of identification. The client issues a request for an individual layer of each tile based on the occupancy of each buffer, for example, the layers may be ordered in order of priority such that requests for material from one layer are lower in order of priority. If the occupancy of any buffer of the layer is lower than the threshold of the lower layer, it will not be issued. In the method, the data received from the lower priority order is given priority so that if the available bandwidth is reduced to a lower bandwidth than the layer that receives the higher priority order, then only the request is requested. The lower layers. Moreover, the thresholds associated with different layers may be different such that, for example, the lower layers have higher thresholds. In the case where the available bandwidth is changed such that higher layer data cannot be received before the tile's playout time, the lower layer data will necessarily have been received and thus the presentation can be used with only the lower layers. Come on. The threshold for buffer occupancy may be defined in terms of the data byte, the play duration of the data contained in the buffer, the number of blocks, or any other suitable measurement.

In a third embodiment, the methods of the first embodiment and the second embodiment can be combined such that a plurality of media representations are provided, each media representation comprising a subset of layers (as in the first embodiment) and enabling The second embodiment is applied to a subset of the layers within the representation.

In a fourth embodiment, the methods of the first embodiment, the second embodiment and/or the third embodiment may be combined with embodiments in which a plurality of independent representations of content are provided, such that, for example, in the independent representations At least one independent representation includes a plurality of techniques applying the first embodiment, the second embodiment, and/or the third embodiment Layers.

Advanced buffer manager

In combination with buffer monitor 126 (see Figure 2), an advanced buffer manager can be used to optimize the client side buffers. The block request streaming system wants to ensure that media play can begin quickly and smoothly, while providing maximum media quality to the user or destination device. This may require the client to request a block that has the highest media quality, but can also start quickly and be received in time to be broadcasted without forcing a pause in the presentation.

In an embodiment using an advanced buffer manager, the manager decides which blocks of media material to request and when to make their requests. The relay data set of the content to be presented may be provided, for example, to an advanced buffer manager, the relay material including a list of representations available for the content and relay information for each representation. The represented relay data may include information about the indicated data rate and other parameters, such as video, audio or other transcoder and codec parameters, video resolution, decoding complexity, audio language, and impact on the client. Any other parameters of the selection.

The represented relay data may also include an identifier of the block indicating that the segment has been segmented, and the identifiers provide information required by the client to request the block. For example, where the request protocol is HTTP, the identifier may be an HTTP URL, possibly along with additional information identifying the byte range or time span within the file identified by the URL, the byte range or The time span identifies a particular block within the file identified by the URL.

In a specific implementation, the advanced buffer manager determines when the receiver is made A request for a new block is made and the advanced buffer manager itself may handle the sending of the request. In a novel aspect, the Advanced Buffer Manager makes a request for a new block based on the value of the balance ratio between the use of excess bandwidth and the exhaustion of media during streaming.

The information received by the buffer monitor 126 from the block buffer 125 may include each of when the media material was received, how much has been received, when the broadcast of the media material has started or stopped, and the speed of the media broadcast, etc. An indication of the event. Based on this information, buffer monitor 126 can calculate the variable B _current representing the current buffer size. In these examples, B _currently represents the amount of media contained in the client or other one or more device buffers and can be measured in time units such that B _currently represents that no more blocks or partial blocks are received. The amount of time it will take to broadcast all of the media represented by the block or partial block stored in the one or more buffers. Therefore, B _currently represents the "broadcast duration" of the media data available at the client but not yet played at the normal broadcast speed.

As time passes, the _current value of B will decrease as the media is broadcast and will increase each time new material for the block is received. Note that for the purposes of this explanation, it is assumed that a block is received when all of the data for that block is available at the block requester 124, but other measures may be used instead to take into account, for example, the reception of a partial block. . In practice, the receipt of a block can occur over a period of time.

Figure 13 illustrates the change in the _current value of B over time as the media is broadcast and the block is received. As shown in FIG. 13, for a time of less than t _0, B _current is 0, indicating information has not been received. At t ₀ , the first block is received and the _current value of B is increased to be equal to the playback duration of the received block. At this point, the broadcast has not yet started and therefore the _current value of B remains constant until time t ₁ , at which point the second block arrives and B _currently increases the size of the second block. At this time, _{the current} broadcast starts and B value starts to decrease linearly until time t _2, the third block to arrive at this time.

B The _current progressive continues in this "sawtooth" manner, stepwise increasing each time a block is received (at times t ₂ , t ₃ , t ₄ , t _{5 ,} and t ₆ ) and is broadcast along with the data It is reduced smoothly. Note that in this example, the broadcast is done at the normal playout rate of the content, and therefore the slope of the curve between block receptions is exactly -1, meaning that there is one second for each second real time elapsed. The media material is played. When the frame-based media is broadcast at a given number of frames per second (eg, 24 frames per second), the slope-1 will be approximated by a small step function indicating the broadcast of each individual data frame, such as each When the frame is broadcast, the step size is -1/24 seconds.

Figure 14 illustrates another example of B _current evolution over time. In this example, the first block arrives at t ₀ and the broadcast begins immediately. Blocks and broadcast continues until arrival time t _{3, the current} value B at this time reaches zero. When this happens, no more media material is available for broadcast, forcing the media to pause. At time t ₄ , the fourth block is received and playback is resumeable. This example thus illustrates a situation in which reception of a fourth block is later than desired, resulting in a pause in playout and thus a poor user experience. Therefore, the goal of the Advanced Buffer Manager and other features is to reduce the probability of this event while maintaining high media quality.

Buffer monitor 126 may also be another metric calculating _{the ratio} B (t), the ratio of the length _{ratio of} B (t) is received in a given period of time and the period of time of media. More specifically, B _ratio (t) _{is received} equal to T / (T _now - t), where T is _received in the media from the received amount t until the _current period of the current time T (in media playout Time to measure), t is some time earlier than the current time.

The B _ratio (t) can be used to measure the _current rate of change of B. _Ratio B (t) = 0 is a case wherein starting from the time t has not received data; assumed media being broadcast, the _current from the B reduces the time (T _now - t). _Ratio B (t) = 1 in which for the time (T _now - t) the same amount in terms of the amount of media being received and broadcast case; B _current at time T will _now have the same value at time t . The B _ratio (t) > 1 is a case where more data is received than time for the time ( T _now - t ); B _{is now} increasing from time t to time T.

Buffer monitor 126 for further calculations "State (status)" value "State (status)" The value can be a discrete number of values. Buffer monitor 126 is further provided with a function of the NewState _(current B, B _ratio), the new "Status" value for the case where the function t <T to the _current value _{of the current ratio of} the current value B and B set as the output. Whenever the B _current and B _ratio causes the function to return a value different from the current value of the "state", the new value is assigned to the "state" and the block selector 123 is indicated to the new state value.

The function NewState can be evaluated with reference to the space of all possible values of ( B _current , B _ratio ( T _now - T _x )), where T _x can be a fixed (configuration) value, or can be, for example, from B _{the current} values are mapped to the T _x values derived from the configuration table B _current output, or may depend on the "status" of the previous value. One or more partitions of the space are supplied to the buffer monitor 126, wherein each partition includes a set of disjoint regions, each region being labeled with a "state" value. On the evaluation of the function "NewState" thus determined is divided and includes an identification (B _current, B _ratio (T _current - T _x)) fall within the operating region of. The return value is thus the label associated with the area. In the simple case, only one division is provided. In a more complex case, the division (B _current, B _ratio (T _current - T _x)) at the time of evaluation before NewState function may depend or depends on other factors.

In a particular embodiment, the division may be based on the above described _current B comprise several thresholds and several configuration tables B _ratio threshold values. Specifically, let _{the current} threshold B is _{a threshold} B (0) = 0, B _threshold (. 1), ..., B _threshold (n _{1), the threshold} B (n ₁ +1) = ∞, where n ₁ is B The _current number of non-zero thresholds. Let B _ratio threshold _{ratio threshold} B (0) = 0, B _{ratio threshold} (1), ..., B _{ratio threshold} (n _2), B _{ratio threshold} (n ₂ +1) = ∞, where n ₂ is The number of thresholds for the B _ratio . The thresholds define a division of a cell grid comprising ( n ₁ +1) × ( n ₂ +1), wherein the i- th unit of the j-th row corresponds to where the B- _threshold ( i -1) <= B is _currently < A region where B _threshold ( i ) and B _{ratio threshold} ( j -1) <= B _ratio < B _{ratio threshold} ( j ). Each cell of the grid described above is labeled with a state value, such as via association with a particular value stored in the memory, and the function NewState then returns a _{ratio of current} and B values by value B ( T _now - T _x ) The status value associated with the indicated cell.

In another embodiment, a hysteresis value can be associated with each threshold. In this enhanced method, the evaluation of the function NewState may be based on a temporary partition constructed as follows using a set of temporarily modified thresholds. For each B _current threshold less than the B _current range corresponding to the cell selected at the last evaluation of NewState , the threshold is decreased by subtracting the hysteresis value associated with the threshold. For each B _current threshold greater than the B _current range corresponding to the cell selected at the last evaluation of NewState , the threshold is increased by adding a hysteresis value associated with the threshold. For each threshold is less than the _ratio B at the last evaluation of NewState the selected cells corresponding to the range B _ratio, hysteresis value by subtracting the threshold value associated with the threshold value is reduced. For each B- _ratio threshold greater than the B- _rate range corresponding to the cell selected at the last evaluation of NewState , the threshold is increased by adding a hysteresis value associated with the threshold. The modified threshold is used to evaluate the value of NewState and then the threshold returns the original value of the threshold.

Other ways of defining the division of space will become apparent to those skilled in the art upon reading this. For example, B can be divided based on _{the ratio} through the use of _current and inequalities B linear combination, e.g. α0, α1, and α2 is _{the ratio of} real-valued α1‧ B ≦ α0 + α2‧ B _this form of linear inequality defined threshold to define the entire The half space within the space and the definition of each disjoint set as the intersection of several such half spaces.

The above description is for illustrating the basic process. For example, in the real-time programming field, the skilled person will be clear when reading the case, and efficient implementation is possible. For example, each time new information is provided to the buffer monitor 126, it is possible to calculate the future time that NewState will shift to the new value if, for example, no further data for the block is received. The timer is then set for this time and, in the absence of further input, the expiration of the timer will cause a new status value to be sent to the block selector 123. Therefore, it is only necessary to perform calculations when new information is supplied to the buffer monitor 126 or when the timer expires, without performing continuously.

The appropriate values for the status can be Low, Stable, and Full. An example of a suitable threshold set and resulting cell grid is illustrated in FIG.

In Fig. 15, the _current threshold of B is shown on the horizontal axis in milliseconds, and the hysteresis value is shown in the form of "+/- value " below the _current threshold of B. The B _ratio threshold is plotted on the vertical axis in parts per thousand (i.e., multiplied by 1000), and the hysteresis value is shown in the form of "+/- value " below the B _ratio threshold. The "low", "stable", and "full" status values are labeled "L", "S", and "F" in the grid cells, respectively.

The block selector 123 receives a notification from the block requester 124 whenever there is an opportunity to request a new block. As described above, the block selector 123 is provided with information about the available plurality of blocks and relay information for the blocks, including, for example, information about the media data rate for each of the blocks.

The information about the media data rate of the block may include the actual media data rate of the specific block (that is, the block size in bytes divided by the broadcast time in seconds), the representation of the block. The average media data rate, or a combination of the above available bandwidths that are required to maintain the coverage of the block, or a combination of the above.

The block selector 123 selects a block based on the state value newly indicated by the buffer monitor 126. When the status value is "stable", the block selector 123 selects the block from the same representation as the previously selected block. The selected block is the first block (in the order of play) of the media material containing the time period of the media material in the presentation that has not previously requested the presentation.

When the status value is "low", the block selector 123 selects a block from a representation having a media data rate lower than the media data rate of the previously selected block. Several factors influence the exact choice of representation in this situation. For example, the tile selector 123 can be provided with an indication of the aggregation rate of the incoming material and can select a representation having a media data rate that is less than the value.

When the status value is "full", the block selector 123 selects a block from a representation having a media data rate higher than the media data rate of the previously selected block. Several factors influence the exact choice of representation in this situation. For example, the tile selector 123 can be provided with an indication of the aggregation rate of incoming data and can be selected A representation of the media data rate that is no higher than this value.

Several additional factors may further affect the operation of the block selector 123. In particular, the frequency of increasing the media data rate of the selected block may be limited, even if the buffer monitor 126 continues to indicate the "full" state. In addition, the block selector 123 is likely to receive a "full" status indication but no higher media data rate is available (eg, since the last selected block already corresponds to the highest available media data rate). In this case, block selector 123 may delay the selection of the next block by a time selected such that the total amount of media material buffered in block buffer 125 is bounded.

Additional factors may affect the set of blocks considered during the selection process. For example, the available blocks may be limited to blocks from which the coded resolution of the blocks falls within a particular range provided to the block selector 123.

The block selector 123 may also receive input from other elements of other aspects of the monitoring system, such as the availability of computing resources for media decoding. If such resources become scarce, block selector 123 may indicate blocks of lower computational complexity of decoding thereof within the relay material (eg, representations with lower resolution or frame rate are generally lower) Decoding complexity).

The substantial advantage brought about by the embodiment described above is that the use of the value B _{ratio in the} evaluation of the NewState function within the buffer monitor 126 allows the quality to rise faster at the beginning of the presentation than the method that only considers B _current . Without considering the B _ratio , it is possible that the system can select blocks with higher media data rates and therefore higher quality after accumulating a large amount of buffered data. However, when the B _ratio value is large, the indication available bandwidth is much higher than the media data rate of the previously received block and even if the buffered data is relatively small (ie, the _current value of B is very low), the request has Higher media data rates and therefore higher quality blocks are still safe. Similarly, if the B _ratio value is low (eg, <1), the indicated available bandwidth has fallen below the media data rate of the previously requested block and therefore the system will switch to lower even if B is _currently high. The media data rate and hence the lower quality, for example, avoids reaching the point where B is _currently = 0 and the broadcast of the media is stagnant. Such improved behavior may be particularly important in environments where network conditions and thus delivery speeds may change rapidly and dynamically (eg, users are streaming to mobile devices).

The use of configuration data to specify the partitioning of the value space ( B _current , B _ratio ) brings another advantage. Such configuration data may be provided to the buffer monitor 126 as part of the presentation relay material or via other dynamic means. In a practical deployment, since the behavior of the user's network connection may be highly variable over time between users and for individual users, it may be difficult to predict a good job for all users. . The possibility of dynamically providing such configuration information to the user allows for the development of good configuration settings based on accumulated experience over time.

Variable request size control

If each request is for a single block and if each block encodes a short media segment, then a high frequency request may be required. If the media block is short, the video broadcast moves quickly between the blocks, which provides the receiver with more frequent opportunities to adjust or change the selected data rate of the receiver via the change representation, thereby improving the broadcast performance. The possibility of continuing without stagnation. However, a disadvantage of high frequency requests is that such requests may not be sustainable on certain networks where the client's available bandwidth in the server network is constrained, such as in 3G and 4G, for example. In a wireless WAN network such as a wireless WAN, the capacity of the data link from the client to the network is limited or may become limited in a short or long period of time due to changes in radio conditions.

High frequency requests also mean a high load on the service infrastructure, which brings associated costs in terms of capacity requirements. Therefore, it would be desirable to have some of the benefits of high frequency requests without all of these disadvantages.

In some embodiments of the block stream system, the flexibility of the high request frequency is combined with the low frequency request. In this embodiment, the tiles may be constructed as described above and also aggregated into segments comprising a plurality of tiles as described above. At the beginning of the presentation, the process described above in which each request references a single block or multiple concurrent requests to request portions of the block is applied to ensure a fast channel change time at the beginning of the presentation and is therefore good User experience. Subsequently, when a certain condition to be described below is met, the client can issue a request to cover multiple blocks in a single request. This is possible because the blocks have been aggregated into larger files or segments and can be requested using a byte or time range. Coherent bytes or time ranges can be aggregated into a single larger byte or time range, resulting in a single request for multiple blocks, and even a non-contiguous block can be requested in one request.

A basic configuration that can be driven by whether a single block (or partial block) is requested or multiple consecutive blocks is requested is based on whether the decision is likely to be played out. For example, if it is likely that a change to another representation will be required in the near future, then the client preferably makes a request for a single block, ie a small amount of media material. One reason for this is that if a request for multiple blocks is made when it is likely to switch to another representation, the switch may be at the most The last few blocks were made before being broadcast. Therefore, the download of the last few blocks may delay the delivery of the media material of the indicated switch, which may result in media playout.

However, requests for a single block do result in higher frequency requests. On the other hand, if it is unlikely that a change to another representation will be needed in the near future, it may be better to make a request for multiple blocks, since all of these blocks are likely to be broadcast, and this results in lower frequencies. The request can thus greatly reduce the burden of request management, especially if there are no upcoming changes in the representation.

In conventional block aggregation systems, the amount requested in each request is not dynamically adjusted, ie typically each request is for the entire file, or each request is for approximately the same amount as the indicated file (sometimes Measured in time, sometimes measured in bytes). Therefore, if all requests are small, the request management burden is high, and if all requests are large, this increases the chance of media stagnation events and/or selects a lower quality representation to avoid having to follow the network. The opportunity to provide lower quality media broadcasts is increased with changes in road conditions and rapid changes.

An instance of a condition that can cause subsequent requests to reference multiple blocks when satisfied is the _current threshold for buffer size B. If B is _currently below the threshold, each request issued references a single block. If B is _currently greater than or equal to the threshold, each request issued references multiple blocks. If a request to reference multiple blocks is issued, the number of blocks requested in each individual request can be determined in one of several possible ways. For example, the number can be a constant, such as two. Alternatively, the number of blocks requested in a single request may depend on the buffer status and in particular on B _current . For example, a number of thresholds may be set in which the number of blocks requested in a single request is derived from the highest of a plurality of thresholds less than B _current .

Another example of a condition that may result in a request to reference multiple blocks when satisfied is the "state" value variable described above. For example, when the status is "stable" or "full", a request for multiple blocks can be issued, but when the status is "low", all requests can be directed to one block.

Another embodiment is illustrated in FIG. In this embodiment, when the next request is to be issued (determined in step 1300), the current state value and B _{current are} used to determine the size of the next request. If the current state value is "low", or the current state value is "full" and the current representation is not the highest available representation (determined in step 1310, the answer is yes), then the next request is selected as a short request, for example Only for the next block (the block is determined and requested in step 1320). The basic principle behind this move is that these conditions are the conditions in which it is likely that there will be a change soon. If the current status value is "stable", or the current status value is "full" and the current representation is the highest available representation (determined in step 1310, the answer is no), then the consecutive blocks requested in the next request The duration of a certain fixed α < 1 is selected to be proportional to the _current alpha score of B (the block is determined in step 1330, the request is made in step 1340), for example, for α = 0.4, if B is _currently = 5 seconds The next request may be for a block of about 2 seconds, and if B is _currently = 10 seconds, the next request may be for a block of about 4 seconds. One of the basic principles behind this is that under these conditions, it is unlikely that a switch to the new representation will be made in the amount of time that is _currently proportional to B.

Flexible pipeline

The block stream system can use a specific bottom with, for example, TCP/IP A file transfer agreement for a layer transport agreement. At the beginning of a TCP/IP or other transport protocol connection, it may take a considerable amount of time to achieve utilization of all available bandwidth. This may result in a "connection initiation penalty" each time a new connection is initiated. For example, in the case of TCP/IP, the connection initiation penalty occurs due to both the time taken to establish a connection by the initial TCP handshake and the time taken by the congestion control protocol to achieve full utilization of the available bandwidth.

In this case, it may be desirable to use a single connection to issue multiple requests to reduce the frequency of initiating connection initiation penalties. However, some file transfer protocols, such as HTTP, do not provide a mechanism to cancel the request without completely closing the transport layer connection, and thus incur a connection initiation penalty when establishing a new connection instead of the old one. If it is decided that the available bandwidth has changed and instead requires a different media data rate, ie a decision to switch to a different representation, the request issued may need to be cancelled. Another reason for canceling the issued request may be that the user has requested to end the media presentation and start a new presentation (perhaps the same content item at a different point of the presentation, or perhaps a new content item).

As is known, the connection initiation penalty can be avoided by keeping the connection open and reusing the same connection for subsequent requests, and as is also known, if multiple requests are issued simultaneously on the same connection (called in the context of HTTP) For the "pipelined" technology, the connection can be fully utilized. However, the disadvantage of simultaneously or more generally issuing multiple requests in such a way that multiple requests on the connection are issued before the previous request is completed may be that the connection is then responsible for carrying the response to the request and therefore should be changed if desired If a request is made, the connection may be closed if you need to cancel a request that has been sent but is no longer wanted.

The probability that the issued request needs to be cancelled may depend in part on the duration of the time interval between the issuance of the request and the broadcast time of the requested block, in part depending on the meaning that when the time interval is large, The probability that a request to be sent needs to be cancelled is also high (because the available bandwidth is likely to change during this interval).

As is known, some file download protocols have the property that a single underlying transport layer connection can advantageously be used for multiple download requests. For example, HTTP has this property because reusing a single connection for multiple requests avoids the "connection initiation penalty" described above for requests other than the first one. However, the disadvantage of this approach is that the connection is responsible for transmitting the data requested in each issued request, and therefore if one or more requests need to be cancelled, either the connection may be closed, thereby inviting a connection when establishing a replacement connection. The penalty is initiated, or the client may wait to receive data that is no longer needed, thereby incurring a delay in receiving subsequent data.

Embodiments that preserve the advantages of connection reuse without incurring this disadvantage and additionally increasing the frequency at which connections can be reused are now described.

Embodiments of the block streaming system described herein are configured to reuse connections for multiple requests without having to commit to the connection being responsible for a particular set of requests from the beginning. Essentially, a new request is made on the connection when the issued request on the existing connection has not completed but is near completion. One reason for not waiting until the existing request is completed is that the connection speed may be degraded if the previous request is completed, that is, the underlying TCP communication period may enter an idle state or the TCP cwnd variable may be significantly reduced, thereby significantly reducing the The initial download speed of the new request made on the connection. One reason to wait until the completion of an additional request is close to completion is because if the new request was issued long before the previous request was completed, the newly issued request may not even be activated for a long period of time, and it is possible During this period of time before the newly issued request is initiated, the decision to make a new request is no longer valid, for example due to a decision to switch the representation. Thus, embodiments of the client implementing the technology will issue new requests on the connection as late as possible without slowing down the download capabilities of the connection.

The method includes monitoring and applying a test to the number of bytes received on the connection in response to the latest request sent on the connection. This can be done by configuring the receiver (or transmitter, if applicable) for monitoring and testing.

If the test passes, another request can be made on the connection. An example of a suitable test is whether the number of bytes received is greater than a fixed fraction of the size of the requested material. For example, the score can be 80%. Another example of a suitable test is based on the following calculations, as illustrated in FIG. In this calculation, let R be an estimate of the data rate of the connection, T is an estimate of the round trip time ("RTT"), and X is a value which can be, for example, a constant set to a value between 0.5 and 2. A factor in which the estimates for R and T are updated on a regular basis (updated in step 1410). Let S be the size of the data requested in the latest request, and B be the number of bytes received in the requested data (calculated in step 1420).

A suitable test would be to have the receiver (or transmitter, if applicable) perform the routine of evaluating the inequality ( S - B ) < X ‧ R ‧ T (tested in step 1430) and take action if "yes". For example, a test can be made to see if another request is ready to be issued on the connection (tested in step 1440), and if "yes" then the request is made to the connection (step 1450) and if "no" then The process returns to step 1410 to continue the update and testing. If the result of the test in step 1430 is "NO", then the process returns to step 1410 to continue the update and testing.

The inequality test in step 1430 (eg, performed by a suitably programmed component) causes each subsequent to be issued when the amount of remaining data to be received is equal to X multiplied by the amount of data that can be received at the current estimated reception rate within one RTT. request. Several methods of estimating the data rate R in step 1410 are known in the art. For example, the data rate can be estimated as Dt / t , where Dt is the number of bits received in the previous t seconds and where t can be, for example, 1 second or 0.5 seconds or some other interval. Another method is exponentially weighted average or first order infinite impulse response (IIR) filtering of incoming data rates. Several methods for estimating RTT "T" in step 1410 are known in the art.

The test in step 1430 can be applied to the aggregation of all active connections on the interface, as explained in more detail below.

The method further includes constructing a candidate request list, associating each candidate request with a set of suitable servers that can make the request to the server, and ordering the candidate request list in order of priority. Some of the entries in the candidate request list may have the same priority order. The servers in the list of suitable servers associated with each candidate request are identified by the host name. Each host name corresponds to a set of internet protocol addresses that are available from the network function variable name system, as is well known. Thus, each possible request on the candidate request list is associated with a set of internet protocol addresses, specifically with the candidate request The associated server name of the associated server is associated with the union of the sets of Internet Protocol addresses associated with each other. Whenever the connection satisfies the test described in step 1430 and a new request has not been issued on the connection, the highest priority request associated with the internet protocol address of the destination of the connection is selected on the candidate request list, and on the connection Issue the request. The request is also removed from the list of candidate requests.

The candidate request may be removed (cancelled) from the candidate request list, a new request having a higher priority than the existing request on the candidate list may be added to the candidate list, and the prioritization of existing requests on the candidate list may be changed. The dynamic nature of the request on the candidate request list and the dynamic nature of the priority of the requests on the candidate list may depend on when the test of the type described in step 1430 is satisfied and the next request can be made. .

For example, it is possible that if the answer to the test described in step 1430 is "yes" at some time t , then the next request issued will be request A, and if the answer to the test described in step 1430 is not "yes" Until a certain time t '> t , the next request issued will be changed to request B because request A is removed from the candidate request list between times t and t ', or because at time t and t ' The request B with a higher priority order than the request A is added to the candidate request list, or since the request B is already on the candidate list at time t but the priority order is lower than the request A, and between the times t and t ', The priority order of request B is changed to be higher than the priority order of request A.

Figure 18 illustrates an example of a request list on a candidate request list. In this example, there are three connections, and there are six requests on the candidate list, labeled A, B, C, D, E, and F. Each request on the candidate list can be issued on the connected subset as indicated, for example request A can be sent on connection 1, and request F can be Connection 2 or connection 3 is issued. The priority order for each request is also indicated in Figure 18, and the lower priority value indicates that the request has a higher priority. Thus, requests A and B with priority 0 are the highest priority requests, while requests F with priority value 3 are the lowest priority among the requests on the candidate list.

If at this point in time t , connection 1 passes the test described in step 1430, then request A or request B is issued on connection 1. If instead the connection 3 passes the test described in step 1430 at this time t , then request D is issued on connection 3 because request D is the highest priority request that can be issued on connection 3.

Suppose that for all connections, the answer to the test described in step 1430 is "No" from time t to some later time t ', and between time t and t ', the priority of request A changes from 0. At 5, the request B is removed from the candidate list, and a new request G having the priority order 0 is added to the candidate list. Subsequently, at time t ', the new candidate list can be as shown in FIG.

If connection 1 passes the test described in step 1430 at time t ', then request C with priority order 4 is issued on connection 1, since request C is the highest on the candidate list that can be sent on connection 1 at that point in time. Priority order request.

It is assumed that in the same case, the request A is originally issued on connection 1 at time t (request A is one of the two highest priority choices for connection 1 at time t as shown in FIG. 18). Since the answer to the test described in step 1430 is "No" for all connections from time t to some later time t ', then connection 1 still delivers the request for the request issued before time t until at least time t ', and therefore request A will not start until at least time t '. Requesting C at time t ' is a better decision than issuing request A at time t , because request C is started after t ' with the same time that request A would have started, and because the time C is requested, Request A has a higher priority.

As a further alternative, if the type of test described in step 1430 is applied to the aggregation of active connections, the Internet Protocol address of the connected destination may be selected with the first request on the candidate request list or the same A request is associated with another request with the same priority order.

There are several ways to construct a list of candidate requests. For example, the candidate list may include representatives of n represents the n next Request information portion chronological order of the currently presented, wherein a request having the highest priority of the first data portion of the request has the latest data portion The lowest priority. In some cases, n can be one. The value of n may depend on the buffer size B _current , or other measure of the state variable or the state of the client buffer occupancy. For example, a number of thresholds may be _currently set for B , and a value is associated with each threshold, and then the value of n is taken to be a value associated with a _current highest threshold less than B.

The embodiment described above ensures that the request is flexibly assigned to the connection, thereby ensuring that the existing connection is reused, even if the highest priority request is not suitable for the connection (because the destination IP address of the connection is not assigned to be associated with the request) The same IP address of any host name) is also true. B n of _current or client buffer occupancy state of dependency or other measures to ensure that the next request for information associated with the portion of the issue and the urgent need to complete the chronological order to be broadcast in the client's time, such " A request to leave the priority order is not issued.

These methods can be advantageously combined with coordinated HTTP and FEC.

Consistent server selection

As is well known, files that are to be downloaded using a file download protocol are typically identified by an identifier including a host name and a file name. This is the case, for example, for an HTTP protocol, in which case the identifier is a Uniform Resource Identifier (URI). The host name may correspond to multiple hosts identified by each internet protocol address. For example, this is a common way to spread the load of requests from multiple clients across multiple physical machines. Specifically, this approach is often adopted by the Content Delivery Network (CDN). In this scenario, a request to any physical host on the connection is expected to succeed. A variety of methods are known for clients to use to select from various internet protocol addresses associated with host names. For example, the addresses are typically provided to the client via the domain name system and are provided in order of priority. The client can then select the highest priority (first) internet protocol address. However, in general, there is no coordination between the clients on how to make the selection, so different clients may request the same file from different servers. This may result in the same file being stored in the cache memory of multiple servers in the vicinity, which reduces the efficiency of the cache memory infrastructure.

This can be handled by a system that advantageously increases the probability that two clients requesting the same block will request the block from the same server. The novel methods described herein include in a manner determined by the identifier of the file to be requested and in such a way that different clients selected by the same or similar Internet Protocol address and file identifier will make the same selection. You can choose from the Internet Protocol address.

A first embodiment of the method is described with reference to FIG. The client first obtains a set of internet protocol addresses IP ₁ , IP ₂ , ..., IP _n as shown in step 1710. If, as determined in step 1720, there is a file to be requested for the file, the client decides which Internet Protocol address to use to issue the request for the file, as determined in steps 1730 through 1770. Given a set of internet protocol addresses and identifiers for the files to be requested, the method includes sorting the internet protocol addresses in a manner determined by the file identifier. For example, for each internet protocol address, a byte string comprising the concatenation of the internet protocol address and the file identifier is constructed, as shown in step 1730. Applying a hash function to the byte string, as shown in step 1740, and arranging the resulting hash values according to a fixed order, such as a numerical power, as shown in step 1750, thereby causing an internet protocol address. Sort. The same hash function can be used by all clients, thus ensuring that the hash function produces the same result for a given input for all clients. The hash function can be statically configured into all clients in the client set, or all clients in the client set can obtain portions of the hash function when the client obtains the list of internet protocol addresses or A full description, or all clients in the client collection, may obtain a partial or complete description of the hash function when the client obtains the file identifier, or the hash function may be determined by other means. The first internet protocol address in the ranking is selected and the address is then used to establish a connection and issue a request for all or part of the file, as shown in steps 1760 and 1770.

The above method can be applied when a new connection is established to request a file. The method can also be applied when multiple built connections are available, and one of the connections can be selected to make a new request.

Furthermore, when a built-in connection is available and a request can be selected from a set of candidate requests having equal priority, the ordering of the candidate requests is caused, for example, via the same hash value method described above, and the ranking is first The candidate candidates that appear are selected. The methods can be combined to sort the hash values from a set of joins and requests with equal priority, respectively, by computing a hash of each combination of joins and requests, sorting the hash values according to a fixed order, and selecting the pair of requests and The first occurrence of the combination in the sorting resulting from the set of connected combinations selects both the connection and the candidate request.

This approach has advantages for the following reasons: a typical approach taken by the block provisioning infrastructure such as shown in Figure 1 (BSI 101) or Figure 2 (BSI 101), in particular the CDN typically takes the approach of providing multiple receiving clients The cached proxy server that is requested by the client. The cached memory proxy server may not have the file requested in the given request and in such a case, such a server typically forwards the request to another server, receiving a response from the server ( The requested file is typically included and forwarded to the client. The cache memory proxy server can also store (cache memory) the requested file, so that the cache memory proxy server can immediately respond to subsequent requests for the file. The common approach described above has the property that the set of files stored on a given cached proxy server is largely determined by the set of requests that the cached proxy server has received.

The method described above has the following advantages. If all clients in the client set are provided with the same list of Internet Protocol addresses, then those clients will use the same Internet Protocol address for all requests made for the same file. If there are two different lists of internet protocol addresses and each client is provided with one of the two lists, then the client will use up to two different internet protocol addresses for all requests for the same file. In general, if the list of Internet Protocol addresses provided to the client is similar, then the clients A small collection of Internet Protocol addresses provided will be used for all requests made for the same file. Since short-range clients tend to be provided with a similar list of Internet Protocol addresses, it is likely that the proximity client will issue a request for a file to only a small portion of the cached proxy server available to those clients. Therefore, only a small fraction of the cache memory proxy server caches the file, which advantageously minimizes the amount of cache memory used for the cache memory.

Preferably, the hash function has the property that different inputs of a small fraction are mapped to the same output, and different inputs are mapped to an essentially random output to ensure that for a given set of internet protocol addresses, The proportion of a given address in the Internet Protocol address to the first address in the sequenced list generated by step 1750 is approximately the same for all Internet Protocol addresses in the list. On the other hand, it is important that the hash function is decisive, and the decisive meaning is that for a given input, the output of the hash function is the same for all clients.

Another advantage of the method described above is as follows. Assume that all clients in the client collection are provided with the same list of Internet Protocol addresses. Due to the nature of the hash function just described, it is likely that requests from different clients for different files will be evenly distributed across the set of internet protocol addresses, which in turn means that the requests will span the cache memory. The proxy server is evenly distributed. Therefore, the cache memory resources for storing the files are evenly distributed across the cache memory proxy server, and requests for files are evenly distributed across the cache memory proxy servers. Thus, the method provides both storage balance and load balancing across the cache memory infrastructure.

Many variations of the approaches described above are known to those skilled in the art, and in many cases, such deformations retain the set of files stored on a given agent that is at least partially comprised by the cached proxy server. The set of requests received determines this property. In the common case where a given host name resolves to multiple entity cache memory proxy servers, it would be common for all such servers to eventually store any copies of a given file that are frequently requested. Such repetition may be undesirable because the storage resources on the cached memory proxy server are limited, and thus the file may sometimes be removed (emptied) from the cache memory. The novel method described herein ensures that requests for a given file are directed to the cache memory proxy server in a manner that reduces such duplication, thereby reducing the need to remove files from the cache memory and thereby increasing The probability that any given file exists in the proxy cache (ie, has not been emptied from it).

When the file exists in the proxy cache, the response sent to the client is faster, which has the advantage of reducing the probability that the requested file is late, and the requested file is late, which causes the media to be paused and thus causes Bad user experience. In addition, when the file does not exist in the proxy cache, the request can be sent to another server, causing additional load on the service infrastructure and on the network connection between the servers. In many cases, the server to which the request is sent may be located at a remote location and returning the file from the server to the cached memory proxy server may incur transmission costs. Thus, the novel methods described herein enable such transmission costs to be reduced.

Probabilistic full file request

A particular concern in the context of using HTTP protocols with scope requests is the cache memory that is typically used to provide scalability in the service infrastructure. The behavior of the server. Although it may be common for HTTP cache memory servers to support HTTP range headers, the exact behavior of different HTTP cache servers varies by implementation. Most cache memory servers implement service range requests from the cache in situations where the file is available in cache memory. A common implementation of the HTTP cache memory server always forwards the downstream HTTP request containing the range header to the upstream node unless the cache server has a copy of the file (cache server or raw server) . In some implementations, the upstream response to the range request is the entire file, and the entire file is cached and the response to the downstream range request is extracted from the file and the response is sent. However, in at least one implementation, the upstream response to the range request is only the data byte in the scope request itself, and the data bytes are not sent by the cache but only as a response to the downstream scope request. . Therefore, the possible consequence of the client using the scope header is that the file itself is never put into the cache and the desirable scalability properties of the network will be lost.

In the above, the operation of the cache memory proxy server is described, and a method of requesting a tile from an aggregated file as a plurality of blocks is also described. For example, this can be achieved by using an HTTP range request header. Such requests are referred to below as "partial requests." Another embodiment will now be described which has advantages in the case where the patch provisioning infrastructure 101 does not provide full support for the HTTP range header. Typically, a server within the block provisioning infrastructure, such as a content delivery network support partial request, may not store a response to a partial request in the local storage area (cache memory). Such a server may fulfill a partial request by forwarding the request to another server, unless the full file is stored in the local storage area, in which case the request may not be transferred. Send a response if it is sent to another server.

The novel enhanced block request streaming system for block aggregation described above may be ineffective in the event that the block provisioning infrastructure exhibits this behavior, as all requests that are actually partially requested will be forwarded to another server. And no request will be served by the cache memory proxy server, which first makes the purpose of providing the cache memory proxy server fall. During the block request streaming process as described above, the client may request a block at the beginning of the file at some point.

According to the novel method described herein, such a request can be converted from a request for the first block in the file to a request for the entire file each time a certain condition is met. When the cached memory proxy server receives a request for the entire file, the proxy server typically stores the response. Therefore, the use of the requests causes the file to be placed in the cache memory of the local cache memory proxy server, so that subsequent requests can be directly from the cache memory whether for full file or partial request. Proxy server to serve. The condition may be such that in a set of requests associated with a given file (eg, a set of requests generated by a group of clients viewing the proposed content item), the condition is at least for a specified score in the requests The words will be satisfied.

An example of a suitable condition is that the randomly selected number is above a prescribed threshold. The threshold may be set such that such conversions that convert a single block request into a full file request occur on average in the scores specified in the requests, for example once within 10 requests (in this case, from the interval) [0, 1] The random number is selected and the threshold may be 0.9). Another example of a suitable condition is a hash of some information associated with a block and some information associated with the client. Counts one of the specified set of values. The method has the following advantages: for files that are frequently requested, the file will be placed in the cache memory of the local cache memory proxy server, but the block requests the operation of the streaming system with each of the requests There is no significant change in the standard operation of a single block. In many cases, where a request to convert a single block request to a full file request occurs, the client program will then request other blocks within the file. If this is the case, such a request can be suppressed because the block in question will be received as a result of the full file request anyway.

URL construction and segment list generation and viewing

The segment list generates a question of how the client's local time at the specific client "now" generates a segment list from the MPD for a specific representation starting at a start time "starttime", where the start time "starttime" is for the on-demand situation. In terms of the beginning of the media presentation, or in terms of wall clock time. The segment list may include locators, such as a URL pointing to an optional initial representation relay material, and a media segment list. Each media segment may have been assigned a start time, duration, and locator. The start time typically represents an approximation of the media time of the media contained in the segment, but not necessarily the exact time of sampling. The start time is used by the HTTP streaming client to issue a download request at the appropriate time. The generation of the segment list (including the start time of each segment) can be done in different ways. The URL can be provided as a playlist, or the URL construction rules can be advantageously used for a compact representation of the segment list.

A URL-based segment list construction may be performed, for example, if the MPD signals the notification via a particular attribute or element such as FileDynamicInfo or an equivalent signal. The following is a small "URL construction overview" A general way of constructing a list of segments from a URL construct is provided in the section. The playlist-based construct can signal notifications, for example, by different signals. The function of viewing in the segment list and reaching the exact media time is also advantageously implemented in this context.

URL constructor overview

As previously described, in one embodiment of the invention, a relay profile containing URL construction rules can be provided that allow the client device to construct a file identifier for the rendered tile. Further novel enhancements to the block request stream system are now described that provide for changes to the relay profile, including changes to URL construction rules, changes in the number of available codes, and relay data associated with available codes, such as bits. Changes in element rate, aspect ratio, resolution, audio or video transcoder or codec parameters or other parameters.

In this novel enhancement, additional material associated with each element of the relay profile can be provided indicating a time interval throughout the presentation. During this time interval, the element can be considered valid, and the element can be ignored except for the time interval. In addition, the syntax of the relay material can be enhanced such that elements that were previously allowed to appear only once or at most may appear multiple times. Additional restrictions may be applied in such cases, which specify that for such elements, the specified time intervals must not intersect. At any given moment, considering only the time interval of the element containing the element at this given time will result in a relay data file that is consistent with the original relay material syntax. Such time intervals are referred to as validity intervals. The method thus provides a means of signaling the notification of the above categories within a single relay profile. Advantageously, such methods can be used to provide a media presentation that supports a change in the described category at a specified point within the presentation.

URL constructor

As described herein, a common feature of a block request streaming system is the need to provide "relay data" to the client, the relay data identifying the available media encodings and providing the client with the required blocks from the encoded blocks. News. For example, in the case of HTTP, the information may include a URL containing a file of the media block. A playlist file can be provided that lists the URL of the given coded block. A plurality of playlist files are provided, each file being for one type of code, together with a "main playlist for playlists" listing playlists corresponding to different codes. The disadvantage of this system is that the relay data can become quite large, and therefore it takes some time to request the relay data when the client starts streaming. Another disadvantage of the system is evident in the case of live content, where the file corresponding to the media data area block is from a stream of media that is being captured (lively) (eg, live sports or news programs). Produced in "in execution". In this case, the playlist file can be updated each time a new block is available (eg, every few seconds). The client device can repeatedly retrieve the playlist file to determine if a new block is available and obtain the URL of the new block. This can place a significant load on the service infrastructure, and in particular means that the relay profile cannot be cached longer than the update interval, which is equal to the block size, typically on the order of a few seconds.

An important aspect of the block request stream system is a method for notifying a client of a file identifier (e.g., a URL) that should be used with a file download agreement to request a block. For example, there is a method of providing a playlist file for each representation presented, the playlist file listing URLs of files containing media material blocks. The disadvantage of this method is that at least some of the playlist files themselves need to be downloaded and then broadcast to start, which increases the channel change time and thus leads to Bad user experience. For long media presentations with several or many representations, the list of file URLs may be large, and thus the playlist file may be large, which further increases the channel change time.

Another disadvantage of this method occurs in the case of live content. In this case, the complete URL list is not available in advance, and the playlist file is periodically updated as new blocks become available, and the client periodically requests the playlist file to receive the updated version. Since the file is frequently updated, the file cannot be stored in the cache memory proxy server for a long time. This means that many requests for this file will be forwarded to other servers and eventually forwarded to the server that generated the file. In the case of popular media presentations, this may result in high load on the server and the network, which may result in slow response times and thus long channel switching times and poor user experience. In the worst case, the server becomes overloaded and this causes some users to be unable to view the presentation.

It is desirable in the design of the block request stream system to avoid restrictions on the form of the available file identifiers. This is due to a variety of considerations that may motivate the use of specific forms of identifiers. For example, in the case where the block provisioning infrastructure is a content delivery network, there may be file naming or storage practices associated with desires or other requirements for distributing storage or service load across the network, which results in unpredictable system design. The specific form of the file identifier.

Another embodiment will now be described which alleviates the above disadvantages while retaining the flexibility to select a proper profile recognition convention. In the method, relay data including a file identifier construction rule can be provided for each representation of the media presentation. The file identifier construction rule may include, for example, a text string. In order to decide what to present The file identifier of the fixed block may provide a method for interpreting the file identifier construction rule, the method comprising determining the input parameter and evaluating the file identification construction rule together with the input parameters. The input parameters may, for example, include an index of the file to be identified, where the first gear has index 0, the second gear has index 1, the third gear has index 2, and so on. For example, in the case where each file spans the same time duration (or substantially the same time duration), then the index of the file associated with any given time within the presentation can be easily determined. Alternatively, the time spanned by each file within the presentation may be provided within the presentation or version relay material.

In one embodiment, the file identifier construction rule can include a text string that can include certain special identifiers corresponding to the input parameters. The evaluation method of the file identifier construction rule includes determining the position of the special identifier in the character string, and replacing each such special identifier with a string representation of the value of the corresponding input parameter.

In another embodiment, the file identifier construction rule can include a text string that follows the expression language. The expression language includes the definition of the syntax that the expression of the language can follow and the set of rules for evaluating the string following the syntax.

Specific examples will now be described with reference to FIG. 21 and the following. An example of a syntactic definition of a suitable arithmetic language defined by the augmented Backus Naur Form is illustrated in FIG. An example for evaluating the rule of the string following the <expression> production in FIG. 21 includes recursively transforming the string (<expression>) following the <expression> production to follow the <word> generation String of the formula: follows the <expression> of the <literal> production.

The <expression> following the <variable> production is generated by the <variable> The value of the variable identified by the <character> string is replaced by the value of the variable.

The <expression> following the <function> production formula evaluates each of its arguments according to the rules as described below and applies a transformation to the arguments depending on the <function> element of the <function> production formula. To evaluate.

The <expression> following the last option of the <expression> production is evaluated by describing the two <expression> elements as follows and applying the last option depending on the <expression> production formula to the arguments The operation of the <opero> element is evaluated.

In the method described above, it is assumed that the evaluation occurs in a context in which a plurality of variables can be defined. A variable is a (name, value) pair, where "name" is a string that follows the <character> production, and "value" is a string that follows the <literal> production. Some variables can be defined outside the evaluation process before the evaluation begins. Other variables can be defined internally within the evaluation process itself. All variables are "global", and the meaning of "global" means that there is only one variable for each possible "name".

An example of a function is the "printf" function. This function accepts one or more arguments. The first argument can follow the <string> production formula (hereinafter referred to as "string"). The printf function evaluates to get the transformed version of the first argument of the printf function. The applied transform is the same as the "printf" function of the C standard library, where the extra arguments included in the <function> production supply the extra arguments expected by the C standard library printf function.

Another example of a function is the "hash" function. The function accepts two arguments, where the first argument can be a string and the second argument can follow the <digit> production (hereafter referred to as "digit"). The "hash" function applies a hash algorithm to the first argument and returns a non-negative integer less than the second argument result. An example of a suitable hash function is provided in the C function shown in Figure 22, the arguments of which are the input string (excluding the enclosed quotes) and the value input value. Other examples of hash functions are well known to those skilled in the art.

Another example of a function is a "Subst" function that takes one, two, or three string arguments. In the case of supplying an argument, the result of the "Subst" function is the first argument. In the case of supplying two arguments, then the result of the "Subst" function is erased by the second argument (excluding the enclosed quotes) in the first argument and returns the first argument thus modified To calculate. In the case of supplying three arguments, the result of the "Subst" function replaces the second argument (excluding the enclosed quotes) by using the third argument (excluding the enclosed quotes) within the first argument. Any occurrence and return of the first argument thus modified is calculated.

Some examples of operands are addition, subtraction, division, multiplication, and modulo operation elements, which are identified by the <operator> production formulas '+', '-', '/', ' ^* ', '%', respectively. These operands require that the <expression> production expression on either side of the <operating element> production formula obtains a digit. The evaluation of the operands involves applying the appropriate arithmetic operations (addition, subtraction, division, multiplication, and modulo, respectively) to the two digits in a general manner and returning the results in a form that follows the <digit> production.

Another example of an operand is to set an operand, which is identified by the <operator> production formula '='. The operand requires the argument on the left to be evaluated to obtain a string whose contents follow the <character> production. The contents of the string are defined as strings within the enclosed quotes. A variable equal to the operand causes the variable's name to be equal to the content of the left argument to be assigned a value equal to the result of evaluating the right argument. This value is also the result of evaluating the operand expression.

Another example of an operand is a sequential operand, generated by an <operating element> Formula ‘;’ identification. The result of evaluating the operand is the argument on the right. Note that, like all operands, both arguments are evaluated and the arguments on the left are evaluated first.

In one embodiment of the invention, the identifier of the file may be obtained by evaluating the file identifier construction rule with a particular set of input variables identifying the requested file according to the above rules. An instance of the input variable is a variable whose name is "index" and whose value is equal to the index of the value of the file within the rendering. Another example of an input variable is a variable named "bitrate" and having a value equal to the average bit rate of the requested version of the presentation.

Figure 23 illustrates some examples of file identifier construction rules where the input variable is the "id" of the identifier providing the representation of the desired presentation and the "seq" of the serial number of the provided file.

Many variations of the above methods are possible as will be apparent to those skilled in the art upon reading this. For example, not all of the functions and operands described above may be provided, or additional functions or operands may be provided.

URL construction rules and time base

This section provides basic URI construction rules to assign file or segment URIs and the start time of each segment within the presentation and media presentation.

For this article, assume that a media presentation description is available at the client.

Assume that the HTTP streaming client client is playing out the media downloaded within the media presentation. The actual rendering time of the HTTP client can be defined by where the rendering time is relative to where the rendering begins. At initialization, the presentation time t=0 can be assumed.

At any point t, the HTTP client can download the playback time tP (also the phase Any material that is at least "MaximumClientPreBufferTime" before the actual presentation time t, and any material that is required due to user interaction (eg, view, fast forward, etc.). In some embodiments, the "maximum client pre-buffer time" may not even be specified. The meaning of not being specified means that the client can download data in advance (tP) ahead of the current play time without restriction.

The HTTP client can avoid downloading unnecessary material, for example, any segment from a representation that is not expected to be broadcast may typically not be downloaded.

The basic process of providing a streaming service may be via downloading a suitable request to download an entire file/segment or file/segment subset, such as via an HTTP acquisition request or an HTTP partial acquisition request. This description addresses how to access data for a particular airtime tP, but in general, the client can download data for a larger time range of play time to avoid inefficient requests. The HTTP client can minimize the number/frequency of HTTP requests when providing streaming services.

In order to access the media material in the specific representation at the play time tP or at least close to the play time tP, the client decides to point to the URL of the file containing the play time and additionally determines the byte range in the file for accessing the play time. .

The media presentation description may assign a representation id "r" to each representation, for example via the use of a RepresentationID attribute. In other words, the content of the MPD will be interpreted as a presence assignment when written by the ingest system or when read by the client. In order to download data for a particular play time tP with a particular representation of id r, the client can construct a suitable URI for the file.

The media presentation description can be assigned to each file or segment of each representation r Lower attribute: (a) represents the sequence number i of the file within r, where i = 1, 2, ..., Nr, (b) represents the relative position of the file with the id r and the index index i relative to the presentation time. The start time, defined as ts(r, i), (c) represents the file URI of the file/segment with the id r and the file index i, denoted as FileURI(r, i).

In one embodiment, the start time and file URI of the file may be explicitly provided for the representation. In another embodiment, a list of file URIs may be explicitly provided, where each file URI is inherently assigned an index i according to the location in the list, and the start time of the segment is as a segment from 1 to i-1 The sum of all the segments is derived from the sum of time. The duration of each segment can be provided in accordance with any of the rules discussed above. For example, anyone who understands basic mathematics can use other methods to derive a method for easily deriving the start time from a single element or attribute and the location/index of the file URI in the representation.

If a dynamic URI construction rule is provided in the MPD, the start time of each file and each file URI may be via the use of construction rules, an index of the requested file, and possibly some additional parameters provided in the media presentation description. Dynamically constructed. This information can be provided, for example, in MPD attributes and elements such as file URI mode (FileURIPattern) and dynamic file information (FileInfoDynamic). The file URI mode provides information on how to construct a URI based on the file index number i and the representation ID r. The file URI format (FileURIFormat) is constructed as: file URI format = sprintf ("%s%s%s%s%s.%s", base URI, base file name, indicating ID format, separator symbol format, file sequence ID Format, file extension name); and FileURI(r,i) is constructed as: FileURI(r,i)=sprintf(file URI format, r,i); relative start time ts(r,i) of each file/segment ) may be derived from an attribute included in the MPD that describes the duration of the segment in the representation, such as the "dynamic file information" attribute. The MPD may also contain a "dynamic file" attribute sequence for all representations in the media presentation or in the same manner as specified above, at least for all representations in the time period. If the media material representing the specific play time tP in r is requested, the corresponding index i(r, tP) can be derived as i(r, t _p ) such that the play time of the index falls at the start time ts ( r, i(r, tP)) and ts(r, i(r, tP) + 1). Segment access can be further limited by the above, for example, the segment is inaccessible.

Once the index and URI of the corresponding segment are obtained, the exact playback time tP is accessed depending on the actual segment format. In this example, without loss of generality, it is assumed that the media segment has a local isochronal line starting at zero. In order to access and present the data of the play time tP, the client can download the material corresponding to the local time from the file/segment accessible via the URI FileURI(r, i) where i=i(r, t _p ).

In general, the client can download the entire file and then access the play time tP. However, not all information of the 3GP file needs to be downloaded, because the 3GP file provides a structure that maps the local time base to the byte range. Therefore, as long as sufficient random access information is available, accessing only a specific bit range of the playback time tP is sufficient to play the media. Likewise, sufficient information about the structure and mapping of the byte ranges and local time bases of the media segments can be provided, for example, using segment indices, etc., in the initial portion of the segment. The client can have sufficient information to directly access the playback time tP via the initial, eg, 1200, byte that can access the segment. The range of bytes.

In another example, it is assumed that a segment index (which may be designated as a "tidx" packet hereinafter) may be used to identify the byte offset of the required one or more segments. A partial acquisition request may be formed for the required one or more segments. There are other alternatives, for example, the client can issue a standard request for a file and cancel the request when the first "tidx" package has been received.

View

The client can attempt to view the specific presentation time tp in the representation. Based on the MPD, the client can access the media segment start time and the media segment URL for each segment of the representation. The client may obtain the maximum segment index i whose start time tS(r, i) is less than or equal to the presentation time tp as the segment index segment_index of the segment most likely to contain the media sample of the presentation time tp, ie segment index=max{i|tS (r,i)<=tp}. Get the segment URL as FileURI(r,i).

Note that the time base information in the MPD may be approximate due to problems associated with placement of random access points, alignment of media tracks, and media timebase drift. Therefore, the segment identified by the above procedure may start at a later time than tp, and the media material presenting the time tp may be located in the previous media segment. In the case of viewing, the viewing time may be updated to be equal to the first sampling time of the retrieved file, or alternatively the previous file may be retrieved. It should be noted, however, that during continuous play, including in the case of switching between replacement representations/versions, media material for the time between tp and the beginning of the retrieved segment is still available.

In order to accurately view the presentation time tp, the HTTP streaming client needs to access a random access point (RAP). In order to determine a random access point in a media segment in the case of 3GPP self-adjusting HTTP streaming, the client may use, for example, ‘ Information in the tidx' or ‘sidx' package (if present) to locate random access points in the media presentation and corresponding rendering times. In the case where the segment is a 3GPP movie clip, it is also possible for the client to use the information in the 'moof' and 'mdat' packets to, for example, locate the RAP and obtain the necessary information from the information in the movie segment and the segment start time derived from the MPD. Presentation time. If no RAP is available for the presentation time before the requested presentation time tp, the client may access the previous segment or may only use the first random access point as the viewing result. When the media segment starts at RAP, the programs are simple.

It should also be noted that all information in the media segment does not necessarily need to be downloaded in order to access the presentation time tp. The client may, for example, first request a 'tidx' or 'sidx' packet from the beginning of the media segment using a byte range request. By using a 'tidx' or 'sidx' packet, the segment time base can be mapped to the byte range of the segment. By continuously using partial HTTP requests, only the relevant parts of the media segment need to be accessed, resulting in an improved user experience and low startup delay.

Segment list generation

As described herein, how to implement a simple direct HTTP streaming client that uses the information provided by the MPD to create a list of segments for the representation of the approximate segment duration dur with signal delivery notifications should be apparent. In some embodiments, the client may assign a coherent index i=1, 2, 3, . . . to the media segment within the representation, ie, the first media segment is assigned an index i=1, the second media segment The index is assigned i=2, and so on. Then, the list of media segments having the segment index i is assigned startTime[i] (start time [i]), and the URL [i] is generated, for example, as follows. First, set the index i to 1. The start time of the first media segment is 0, that is, startTime[1]=0. The URL [i] of the media segment i is obtained as FileURI(r, i). The process continues for all described media segments with index i, and obtains startTime[i] for media segment i as (i-1) ^* dur and obtains URL[i] as FileURI(r,i).

Concurrent HTTP/TCP request

One concern in the block request streaming system is that it is desirable to always request the highest quality block that can be completely received in time for broadcast. However, the data arrival rate may not be known in advance, and thus it may happen that the requested block did not arrive in time for broadcast. This results in the need to pause media play, resulting in a poor user experience. This problem can be mitigated by a conservative client-side algorithm that selects the block to be requested. This conservative approach request is more likely to be received in a timely manner even if the data arrival rate drops during the block reception period. A block of quality (and therefore smaller size). However, this conservative approach has the potential to deliver lower quality broadcasts to users or destination devices, which is also a poor user experience. This problem may be magnified when multiple HTTP connections are used simultaneously to download different blocks, as described below, because available network resources are shared across connections and are therefore used simultaneously for blocks with different playout times.

It may be advantageous for the client to issue requests for multiple blocks concurrently, wherein in this context, "concurrently" means that the response to the request occurs in overlapping time intervals, and the requests are not necessarily exact or Even made at roughly the same time. In the case of the HTTP protocol, this approach improves the utilization of available bandwidth due to the behavior of the TCP protocol, which is well known. This may be especially important for improving the content change time, because the corresponding HTTP/TCP requesting the data of the blocks is requested when the new content is first requested. The connection may start very slowly, and thus using several HTTP/TCP connections at this time can greatly speed up the data delivery time for the first batch of blocks. However, requesting different blocks or fragments on different HTTP/TCP connections may also result in performance degradation because the request for the block to be broadcast first is contending with the request for the subsequent block, contending for each HTTP/TCP Downloads are very different in terms of the delivery speed of such downloads, and therefore the completion time of the request may be highly variable, and it is generally impossible to control which HTTP/TCP connections will complete quickly and which will be slower, so it is likely that at least some The HTTP/TCP download of the first few blocks will be finalized, resulting in a large and variable channel changeover time.

Suppose that each block or segment of the segment is downloaded on a separate HTTP/TCP connection, and the number of concurrent connections is n and the broadcast duration of each block is t seconds, and the content associated with the segment The stream rate is S. When the client initially starts streaming the content, it can issue a request for the first n blocks, which represents the media data for n ^* t seconds.

As is known to those skilled in the art, the data rate of TCP connections varies widely. However, to simplify this discussion, assume that all connections are ideally performed in parallel so that the first block will be fully received at approximately the same time as the other n-1 requested blocks. To further simplify the discussion, it is assumed that the aggregate bandwidth utilized by the n download connections is fixed to a value B during the entire duration of the download, and the stream rate S is constant throughout the representation. Further assuming that the media data structure enables the broadcast of the block to be performed when the entire block is available at the client, that is, the broadcast of the block can only start after receiving the entire block, for example due to the structure of the underlying video coding or due to Encryption is used to encrypt each segment or block individually, and thus the entire segment or block needs to be received before it can be decrypted. Therefore, in order to simplify the following discussion, it is assumed that the entire block needs to be received before any part of the block can be broadcast. Then, the time required before the first block arrives and can be broadcast is about n ^* t ^* S/B.

Minimizing n ^* t ^* S/B is desirable because it is desirable to minimize the content swap time. The value of t can be determined by factors such as the underlying video coding structure and how to utilize the ingestion method, and thus t can be modestly small, but very small t values result in overly complex segment mapping and possibly with efficient video encoding and decryption (if used) )incompatible. The value of n may also affect the value of B, ie B may be large for a larger number of connections n, and thus reducing the number of connections n has the potential to potentially reduce the negative effect of the available bandwidth amount B, thus achieving a reduction in content The goal of changing the time may not be valid. The value of S depends on which of the downloads and broadcasts are selected, and ideally S should be as close as possible to B to maximize the broadcast quality of the media for a given network condition. Therefore, to simplify the discussion, S is assumed to be approximately equal to B. Then, the channel change time is proportional to n ^* t. Therefore, if the aggregate bandwidth used by each connection is sub-linearly proportional to the number of connections (which is usually the case), using multiple connections to download different segments can degrade the channel change time.

As an example, assume t = 1 second, and the value of B = 500 Kbps when n = 1, the value of B = 700 Kbps when n = 2, and the value of B = 800 Kbps when n = 3. Assume that the representation of S=700Kbps is selected. Then, when n=1, the download time of the first block is 1 ^* 700/500=1.4 seconds, and when n=2, the download time of the first block is 2 ^* 700/700=2 seconds, at n When =3, the download time of the first block is 3 ^* 700/800 = 2.625 seconds. In addition, as the number of connections increases, the variability of individual download speeds of each connection is likely to increase (although even in the case of one connection) It is also likely to have some obvious variability). Therefore, in this example, the channel change time and the channel change time variability increase as the number of connections increases. Intuitively, the blocks being delivered have different priorities, that is, the first block has the earliest delivery deadline, the second block has the next earliest deadline, etc., and the downloads are being used to deliver the blocks. The connection is contending for network resources while it is being delivered, and therefore the block with the earliest deadline is more delayed as more and more contention blocks are requested. On the other hand, even in this case, the final use of more than one download connection allows for a higher stream rate that can be maintained, for example in the case of three connections, a stream rate of up to 800 Kbps can be supported in this example. And in the case of a connection, only 500Kbps of stream can be supported.

In practice, as described above, the data rate of the connection may be highly variable over time and between different connections within the same connection, and therefore, the n requested blocks are generally not completed at the same time, and in fact usually It may be that one block may have been completed in half of the other block. This effect leads to unpredictable behavior, because in some cases the first block may be completed much faster than the other blocks, while in other cases, the first block may be completed later than the other blocks. Many, and therefore the beginning of the broadcast may occur relatively quickly in some situations and may occur very slowly in other situations. Such unpredictable behavior can be frustrating for the user and can therefore be considered a bad user experience.

Therefore, there is a need for a method that can utilize multiple TCP connections to improve channel change time and channel change time variability while supporting possible good quality streaming rates. It also needs to be allowed to be assigned to the approximation of the broadcast time of the block. The share of the available bandwidth of each block, and thus a larger share of the available bandwidth, if necessary, can be preferentially assigned to the block with the most imminent play time.

Coordinated HTTP/TCP request

A method of using concurrent HTTP/TCP requests in a coordinated manner is now described. The receiver may employ multiple concurrent coordinated HTTP/TCP requests, for example using a plurality of HTTP byte range requests, each of which is directed to a portion of a fragment in the source segment, or a fragment of the source segment, or Repair part of the segment or repair piece, or all of the repair piece for the repair segment.

The advantages of coordinated HTTP/TCP requests, along with the use of FEC to repair data, may be especially important to provide consistently fast channel change times. For example, at channel switching time, the TCP connection is likely to have just started or has been idle for a period of time, in which case the congestion window cwnd is at its minimum for such connections, and therefore the delivery speed of the TCP connections will be spent The round trip time (RTT) ramps up and the delivery speed on different TCP connections during this ramp up time will be highly variable.

An overview of the FEC-free method is now described. The FEC-free method is a coordinated HTTP/TCP request method in which multiple concurrent HTTP/TCP connections are used to request only the media material of the source block, ie no FEC repair material is requested. Via the FEC-free method, portions of the same segment are requested on different connections, for example using HTTP byte-range requests for portions of the segment, and thus, for example, each HTTP byte range request for segment mapping for the segment Part of the range of bytes indicated in . This may be the case: the individual HTTP/TCP delivery speed ramps up over several RTTs (round trip time) to fully utilize the available bandwidth And therefore the delivery speed is less than the available bandwidth over a relatively long period of time, so if a single HTTP/TCP connection is used to download, for example, the first segment of the content to be broadcast, the channel change time may be large. Using the FEC-free method, downloading different parts of the same clip on different HTTP/TCP connections can significantly reduce channel swap time.

An overview of the FEC method is now described. The FEC method is a coordinated HTTP/TCP request method in which multiple concurrent HTTP/TCP connections are used to request media material for the source segment and FEC repair data generated from the media material. Via the FEC method, the HTTP byte range request for each part of the fragment is used to request portions of the same fragment and FEC repair material generated from the fragment on different connections, and thus for example each HTTP byte range request is directed to A portion of the range of bytes indicated in the segment map of the segment. It may be the case that the delivery speed of an individual HTTP/TCP request ramps up over several RTTs (round trip time) to fully utilize the available bandwidth, so the delivery speed is less than the available bandwidth for a relatively long period of time, so if used A single HTTP/TCP connection to download, for example, the first fragment of the content to be broadcast, the channel change time can be large. The use of the FEC method has the same advantages as the no-FEC method, and has the additional advantage that not all of the requested data needs to arrive before the segment can be recovered, thus further reducing the channel change time and channel change time variability. By requesting a request on a different TCP connection, and requesting an overflow request for FEC repair data on at least one of the connections, the delivery time is, for example, sufficient to recover the amount of data of the first requested segment that enables the media broadcast to begin. The amount can be greatly reduced and can be made more consistent than if the coordinated TCP connection and FEC repair data were not used.

Figures 24(a) through 24(e) illustrate the delivery rate fluctuations of five TCP connections performed on the same link from the same HTTP web server on the simulated Evolutionary Data Optimized (EVDO) network to the same client. An example. In Figures 24(a) through 24(e), the X-axis illustrates the time in seconds, and the Y-axis illustrates the rate at which the client receives the bit on each of the five TCP connections, The rate on each connection is measured over the 1 second interval. In this particular simulation, a total of 12 TCP connections are executing on the link, and thus the network is relatively loaded during the time shown, which is the same cellular service in which more than one client client is operating the network. It is typical when streaming in the area. Note that although the delivery rate is somewhat related over time, the delivery rate of the five connections is vastly different at many points in time.

Figure 25 illustrates a possible request structure for a fragment of size 250,000 bits (approximately 31.25 kilobytes), where 4 different HTTP byte range requests, i.e., first HTTP connections, are made in parallel for different portions of the fragment The request is 50,000 bits, the second HTTP connection requests the next 50,000 bits, the third HTTP connection requests the next 50,000 bits, and the fourth HTTP connection requests the next 50,000 bits. If FEC is not used, ie there is no FEC method, there are only 4 requests for this segment in this example. If FEC, the FEC method, is used, then in this example there is an additional HTTP request for requesting additional 50,000 bit FEC repair data for the repair segment generated from the segment.

Figure 26 is an enlargement of the first few seconds of the five TCP connections shown in Figures 24(a) through 24(e), wherein in Figure 26, the X-axis illustrates time at intervals of 100 milliseconds, and the Y-axis diagram The rate at which the client receives the bit on each of the five TCP connections is measured, the rate being measured over a 100 millisecond interval. a line The amount of aggregate bits received at the client from the first 4 HTTP connections (excluding the HTTP connection from which the FEC data was requested) is received, that is, the amount of aggregate bits arrived using the FEC-free method. The other line shows the amount of aggregate bits received by the client for all of the five HTTP connections (including the HTTP connection from which the FEC data was requested), that is, the aggregate bits arriving using the FEC method. the amount. For the FEC method, it is assumed that the segment can be FEC decoded from the reception of any 200,000 bits of the 250,000 requested bits, which can be achieved, for example, using a Reed-Solomon FEC code, and for example using Luby IV In the case of the RaptorQ code described in the above, it can be realized in essence. For the FEC method in this example, data sufficient to recover the segment using FEC decoding is received after 1 second, allowing 1 second channel change time (assuming that the first clip can be requested and received before it is fully broadcast) Subsequent pieces of information). For the FEC-free method in this example, all the data for the 4 requests had to be received before the segment could be recovered, which occurred after 1.7 seconds, resulting in a channel change time of 1.7 seconds. Therefore, in the example shown in FIG. 26, the FEC-free method is 70% worse than the FEC method in the sense of the channel change time. One reason for the advantages exhibited by the FEC method in this example is that for the FEC method, any 80% of the requested data is allowed to be recovered, and for the FEC-free method, the requested data is required to be received. 100%. Therefore, the FEC-free method has to wait for the slowest TCP connection to complete the delivery, and due to the natural variation of the TCP delivery rate, the delivery speed of the slowest TCP connection may be quite large compared to the average TCP connection. Under the FEC method in this example, a slow TCP connection does not determine when the fragment can be recovered. Conversely, for the FEC method, the delivery of sufficient data is more dependent on the average TCP delivery rate than the worst case TCP delivery rate.

There are many variations of the FEC-free and FEC methods described above. For example, a coordinated HTTP/TCP request can be used for the first few fragments after a channel swap occurs, and thereafter only a single HTTP/TCP request is used to download further fragments, multiple fragments, or entire segments. As another example, the number of coordinated HTTP/TCP connections used may be a function of the urgency of the fragment being requested (i.e., how urgent the broadcast time of the fragments is) and the current network conditions.

In some variations, multiple HTTP connections may be used to request repair material from the repair segment. In other variations, different amounts of data may be requested on different HTTP connections, such as depending on the current size of the media buffer and the data reception rate at the client. In another variation, the source representations are not independent of one another, but rather represent layered media coding, where for example an enhanced source representation may depend on the base source representation. In such a case, there may be a repair representation corresponding to the base source representation and another repair representation corresponding to the combination of base and enhancement source representations.

All of the additional elements enhance the advantages that can be achieved by the methods disclosed above. For example, the number of HTTP connections used may vary depending on the current amount of media in the media buffer and/or the rate of reception in the media buffer. Coordinated HTTP requests utilizing FEC, ie, the FEC method described above and variations of the method, such as making more coordinated HTTP in parallel for different portions of the first segment, are aggressively used when the media buffer is relatively empty Requesting, requesting all of the source fragments and the relatively large fraction of the repair data from the corresponding repair fragment, and then as the media buffer grows, transitions to a reduced number of concurrent HTTP requests, each requesting a larger portion of the media material, And request is smaller The repair information for the score, for example, is converted to 1, 2 or 3 concurrent HTTP requests, converted to a request for a full number of fragments or multiple consecutive segments per request, and converted to not requesting repair data.

As another example, the amount of FEC repair data may vary as a function of media buffer size, ie, when the media buffer is small, more FEC repair data may be requested, and as the media buffer grows, it may gradually The amount of FEC repair data requested is reduced, and at some point in time when the media buffer is sufficiently large, the FEC repair data may not be requested, and only the data from the source segment indicated by the source may be requested. The benefit of such enhanced technologies is that they allow faster and more consistent channel change times, and greater resilience against potential media stagnation or stagnation, while reducing request message traffic and FEC repair data. Both minimize the amount of extra bandwidth used that exceeds the amount that media in the delivery source segment should consume, while enabling the highest possible media rate for a given network condition.

Additional enhancements when using concurrent HTTP connections

If the appropriate conditions are met, the HTTP/TCP request can be discarded and another HTTP/TCP request can be made to download the material that can be substituted for the material requested in the abandoned request, where the second HTTP/TCP request can be requested with the original request. The exact same information, such as source material; or overlapping data, such as some of the same source data and repair data that have not been requested in the first request; or completely disjoint data, such as repair data that has not been requested in the first request. An example of a suitable condition is to request that there is no response from the Block Server Infrastructure (BSI) within the specified time, or that a failure occurs while establishing a transmission connection with BSI, or an explicit failure message is received from the server. ,or others Failure condition failed.

Another example of a suitable condition is based on a measure of the connection speed (in response to the requested data arrival rate) and the expected connection speed, or the broadcast time of the media material contained in the response or depending on the time. The comparison of the connection speeds required to receive the response before other times is compared, and the reception of the data proceeds abnormally.

This approach has advantages in situations where BSI sometimes exhibits malfunction or poor performance. In this case, the above method improves the probability that the client can continue to reliably broadcast the media material even if there is a fault or bad performance in the BSI. Note that in some cases, there may be advantages to designing a BSI in such a way that BSI sometimes does exhibit such failure or poor performance, such as such a design may have less or less frequent or less frequent performance than does not manifest such failure or poor performance. A low cost of replacement design that exhibits such failure or poor performance. In such a case, the method described herein has further advantages because it allows for the use of such lower cost designs for BSI without the consequences of user experience degradation.

In another embodiment, the number of requests issued for material corresponding to a given block may depend on whether suitable conditions for the block are met. If the condition is not met, the successful completion of all currently unfinished data requests for the block will allow the block to be recovered with a high probability, and the client may be prohibited from making further requests for the block. If this condition is met, a larger number of requests for the block may be issued, i.e., the above prohibition does not apply. An example of a suitable condition is that the time up to the scheduled broadcast time of the block or other time depending on the time falls below the specified threshold. This method has the advantage that this is due to the fact that the additional request for the block data is included in the receipt of the block. The broadcast time of the media materials of the block was imminent and became more urgent. In the case of common transport protocols such as HTTP/TCP, these additional requests have the effect of increasing the share of the available bandwidth dedicated to the material that contributes to the reception of the negotiated block. This reduces the time required to complete the reception of the material sufficient to recover the block, and thus reduces the probability that the block cannot be recovered before the scheduled air time of the media material including the block. As described above, if the block cannot be recovered before the scheduled broadcast time of the media material including the block, the playout will be suspended, resulting in a poor user experience, so the method described herein advantageously Reduce the probability of such a bad user experience.

It should be understood that the reference to the scheduled playout time for a block throughout this specification refers to the time at which the encoded media material including the block may initially be available at the client to achieve a non-suspended playout of the presentation. As will be apparent to those skilled in the art of media presentation systems, this time is actually slightly earlier than the actual time at which the media including the block appears at the physical transducer (screen, speaker, etc.) for broadcast, as may A number of transformation functions need to be applied to the media material including the block to achieve actual playout of the block and the functions may take a certain amount of time to complete. For example, media material is typically transmitted in compressed form and a decompression transform can be applied.

Method for generating a file structure supporting a coordinated HTTP/FEC method

Embodiments of a file structure for generating a client that can be advantageously employed by a coordinated HTTP/FEC method are now described. In this embodiment, for each source segment, there is a corresponding repair segment generated as follows. The parameter R indicates how much FEC repair data is generated on average for the source material in the source segment. For example, R=0.33 indicates that if the source segment contains 1000 kilobytes of data, the corresponding repair The complex contains approximately 330 kilobytes of repair data. The parameter S indicates the symbol size in bits for FEC encoding and decoding. For example, S=64 indicates that the source material and the repair material include 64-byte symbols each of which is used for FEC encoding and decoding purposes.

A repair segment can be generated for the source segment as follows. Each segment of the source segment is considered a source block for FEC encoding purposes, and thus each segment is treated as a source symbol sequence from which the source block of the repair symbol is derived. The total number of repair symbols generated for the first i segments is calculated as TNRS(i)=ceiling(R ^* B(i)/S), where ceiling(x) is a function for outputting the smallest integer with a value of at least x. Therefore, the number of repair symbols generated for the segment i is NRS(i)=TNRS(i)-TNRS(i-1).

The repair segment includes a cascade of repair symbols for the segments, wherein the order of the repair symbols within the repair segment is in the order in which the segments of the repair symbol are generated, and within the segment, the repair symbol is encoded by its symbol identifier ( The order of ESI). The repair segment structure corresponding to the source segment structure is illustrated in FIG. 27 and includes a repair segment generator 2700.

Note that by defining the number of repair symbols for a segment as described above, the total number of repair symbols for all previous segments and thus the byte index of the repair segment depends only on R, S, B(i-1), and B(i). Without depending on any previous or subsequent structure of the segment within the source segment. This is advantageous because it allows the client to quickly calculate the position at which the repair block within the repair segment begins, using only local information about the structure of the corresponding segment from which the source segment of the repair block was generated, and also quickly calculates the Fix the number of repair symbols in the block. Therefore, if the client decides to download and play the clip from the middle of the source segment, the client can also quickly generate and access the corresponding repair block from the corresponding repair segment.

The number of source symbols in the source block corresponding to the segment i is calculated as NSS(i)=ceiling((B(i)-B(i-1))/S). If B(i)-B(i-1) is not a multiple of S, the last source symbol is filled with "0" bytes for FEC encoding and decoding purposes, ie, the last source symbol is padded with "0". The byte and thus the final source symbol size is the S-bit tuple for FEC encoding and decoding purposes, but the "0" padding byte is not stored as part of the source segment. In this embodiment, the ESI of the source symbol is 0, 1, ..., NSS(i)-1, and the ESI of the repair symbol is NSS(i), ..., NSS(i) + NRS(i) -1.

In this embodiment, the URL of the repair segment can be generated from the URL of the corresponding source segment by simply adding a post ".repair" to the URL of the source segment.

The repair index information and FEC information of the repair segment are implicitly defined by the index information of the corresponding source segment and from the values of R and S, as described herein. The time offset and the fragment structure including the repair segment are determined by the time offset and structure of the corresponding source segment. The byte offset to the end of the repair symbol in the repair segment corresponding to the segment i can be calculated as RB(i)=S ^* ceiling(R ^* B(i)/S). The number of bytes in the repair segment corresponding to the segment i is RB(i)-RB(i-1), and thus the number of repair symbols corresponding to the segment i is calculated as NRS(i)=(RB( i) - RB(i-1)) / S. The number of source symbols corresponding to the segment i can be calculated as NSS(i)=ceiling((B(i)-B(i-1))/S). Therefore, in this embodiment, the repair index information and the corresponding FEC information of the repair block in the repair segment can be implicitly derived from the index information of the corresponding segments of R, S and the corresponding source segment.

As an example, consider the example shown in FIG. 28, which illustrates a segment 2 starting at a byte offset B(1)=6410 and ending at a byte offset B(2)=6770. In this example, the symbol size is S=64 bytes, and the virtual separator number illustrates the byte offset corresponding to the multiple of S in the source segment. The total repair segment size as a fraction of the source segment size is set to R = 0.5 in this example. The number of source symbols in the source block of segment 2 is calculated as NSS(2)=ceiling((6,770-6,410)/64)=ceil(5.625)=6, and the 6 source symbols have ESI 0,... 5, wherein the first source symbol is the first 64 bytes of the segment 2 starting at the byte index 6410 in the source segment, and the second source symbol is the bit of the segment 2 starting from the source segment The next 64 bytes at the tuple index 6474, and so on. The end byte offset of the repair block corresponding to the segment 2 is calculated as RB(2)=64 ^* ceiling(0.5 ^* 6,770/64)=64 ^* ceiling(52.89...)=64 ^* 53=3,392 And the starting byte offset of the repair block corresponding to the segment 2 is calculated as RB(1)=64 ^* ceiling(0.5 ^* 6,410/64)=64 ^* ceiling(50.07...)=64 ^* 51 = 3,264, so in this example, there are two repair symbols with ESI 6 and 7 respectively in the repair block corresponding to segment 2, starting at the bit offset 3264 within the repair segment and ending at the bit The tuple offset is 3392.

Note that in the example shown in FIG. 28, although R=0.5 and there are 6 source symbols corresponding to the segment 2, the number of repair symbols is not as simple as the number of source symbols used to calculate the number of repair symbols. Three can be expected, but the number is 2 according to the method described herein. Unlike the number of source symbols that simply use the fragment to determine the number of repair symbols, the embodiments described above enable the positioning of the repair block within the repair segment to be calculated from the index information associated with the corresponding source block of the corresponding source segment. . Furthermore, as the number of source symbols K in the source block increases, the number of repair symbols KR of the corresponding repair block is closely approximated as K ^* R, since generally KR is at most ceil(K ^* R) and KR is at least floor(( K-1) ^* R), where floor(x) is the largest integer up to x.

There are many variations of the above embodiments for generating a file structure that can be advantageously used by clients using the coordinated HTTP/FEC method, as will be appreciated by those skilled in the art. As an example of an alternate embodiment, the original segment of the representation may be divided into N > 1 parallel segments, where for i = 1, ..., N, the specified fraction F _{i of the} original segment is included in the ith parallel segment And wherein the sum of F _i of i = 1, ..., N is equal to 1. In this embodiment, there may be one main segment map that is used to derive segment maps for all parallel segments, similar to how the repair segment map is derived from the source segment map in the embodiment described above. For example, if the active media material is not divided into parallel segments but instead is included in one original segment, the primary segment mapping may indicate the segment structure, and then the segment mapping of the i-th parallel segment may be as follows from the primary segment. Mapping derivation: If the amount of media data in the first code of the segment of the original segment is L byte, the total number of bytes in the first i parallel segment where the first code is aggregated is ceil(L ^* G _i ) Where G _i is the sum of F _j at j=1,...,i. As another example of an alternate embodiment, the segment may include a combination of the original source media material for each segment followed by the repair material of the segment, thereby obtaining the source media material and the FEC code generated from the source media material. Fix the mixed segments of the data. As another example of an alternate embodiment, a mixed segment comprising source media material and repair material may be divided into a plurality of parallel segments comprising a mixture of source media material and repair material.

Method for handling low latency streaming

In some deployment scenarios, low latency streaming to live services may be desirable. For example, in the case of distribution in a local venue of an event, such as a sports event or concert, it is expected that the live event is in the guest with the live service The delay between presentations on the client is as short as possible. For example, a maximum delay of 1 second may be expected.

As noted above, it may be advantageous to arrange each of the segments of the segment presented by the storage medium to begin with a random access point (RAP). Some profiles (especially ISO base media file format live profiles) require that each media segment begin with a RAP.

However, in environments where low end-to-end latency delivery is required, the duration of each segment must be shorter to minimize the delay between live activity and the presentation of the live event on the client. It is desirable to avoid inserting a RAP in each segment that will be used for low latency streaming. For example, RAP in video is usually implemented by an IDR frame. Encoding efficiency can be improved by avoiding the use of IDR frames within short segments expected for low latency streaming.

According to an embodiment, a representation of the media presentation conforming to the live profile and a low latency representation are generated. The representation conforming to the live profile has a relatively large media segment duration. Each media segment that conforms to the representation of the live profile has a RAP at the beginning of the media segment. Low latency indicates that there are relatively short segments (the segments may be referred to as "media segments"), which may not contain RAPs. Clients that support low latency streaming can receive media segments generated for low latency representations of media presentations, while clients that do not support low latency streaming can receive representations of media profiles that are compliant with live profiles. The resulting media segment.

Figure 30 illustrates the relationship between media segments and media segments for low latency streaming. The media segment 3002 generated for the live profile stream includes RAP 3004 ("mdat") at the beginning of the media material. Conversely, in media segments 3004, 3006, and 3008 generated for low latency streaming, Only media segment 3004 contains RAPs.

These media segments are generated during execution and are available for the client to download via HTTP. The media segments can be accumulated into media segments that conform to the ISO base media file format live profile without any modification to the media segments. For example, the media segments can be concatenated into media segments.

Both the media segment and the media segment can be created using the same encoding process. In this way, the media can be efficiently encoded for use by clients operating in environments that require low end-to-end latency and by clients that use protocols that require RAPs in each segment.

In some embodiments, a segment index (SIDX) is generated for each media segment. The SIDX can include a presentation time range within the media segment and a corresponding byte range in which the media segment is occupied by the media segment. In some embodiments, the SIDX indicates whether a RAP is present within the fragment. In FIG. 30, the "Include_RAP" field of the SIDX packet of the media segment 3004 is set to 1, indicating that the media segment 3004 contains the RAP. The "Include_RAP" field of the SIDX package for media segments 3006 and 3008 is set to 0, indicating that media segments 3006 and 3008 do not contain RAPs. The SIDX may further indicate the presentation time of the first RAP within the segment.

According to an embodiment, the media server may generate segments for low latency streaming and push the segments into the cache. The cache can cascade the segments to produce a media segment that is compatible with the live profile. After the media segment is generated, the cache can empty the media segments that have been concatenated to produce the media segment.

A single media presentation description (MPD) can store a first representation of a media segment with a media presentation that is compliant with a live profile and has low latency The second representation of the streamed media segment. Time shifting buffering using media segments and using media segments to handle viewing at the near live edge of the stream provides time shifted viewing. The client can switch between the representations, for example, starting in a time shift buffer and moving closer to the live edge via skipping the chapters presented by the media. Each representation of the MPD can be assigned an attribute to represent an array of representations that can be used for a single media presentation.

In storing an MPD with information about the first representation of the media segment and the second representation of the media segment, it may be advantageous to provide information indicating which of the second representations the media segment begins with the RAP. For example, the MPD can include an attribute to indicate the frequency at which RAPs occur within a plurality of media segments. In one embodiment, the MPD includes an attribute that indicates the frequency in a number of segments (ie, every xth media segment contains a RAP). In another embodiment, the attribute indicates the frequency in a manner that is adjacent to the distance between the RAPs in time.

Alternatively, information about the media clips may be stored in the first MPD, and information about the media segments may be stored in the second MPD.

In some embodiments, the MPD can signal a particular parameter that is applicable to a particular representation, such as the maximum duration of the represented media segment or media segment.

Other embodiments will be envisioned by those of ordinary skill in the art after reading this disclosure. In other embodiments, combinations or sub-combinations of the inventions disclosed above may be advantageously made. The example arrangement of elements is illustrated for purposes of illustration, and it is understood that combinations, additions, rearrangements, and the like are contemplated in alternative embodiments of the invention. Accordingly, while the invention has been described with respect to the exemplary embodiments, those skilled in the art will recognize that many modifications are possible.

For example, the processes described herein can be implemented using hardware components, software components, and/or any combination of the above. In such cases, the software component can be provided on a tangible, non-transitory medium for execution on a hardware that is or separate from the media. Accordingly, the specification and drawings are to be regarded as However, it is apparent that various modifications and changes can be made to the present invention without departing from the spirit and scope of the invention as set forth in the appended claims. And equivalent technical solutions.

Claims (15)

A method for constructing content material to be supplied using a media server, comprising the steps of: obtaining the content to be served; generating a plurality of media segments representing the content and encoded according to an encoding protocol, The encoding protocol includes a media presentation encoding one or more frames in each media segment, wherein a random access point is available in each media segment; generating a plurality of media segments encoded according to the encoding protocol (media fragments), wherein a media segment includes the plurality of media segments, and wherein at least some of the plurality of media segments comprise random access points and at least some of the media segments do not include random access points, a random An access point includes a location in a segment at which a decoder can decode subsequent fragments of the random access point without being affected by segments prior to the random access point; and generating an index for the media segment (segment index), the segment index includes: a presentation time range for each media segment in the media segment, and the media in the media segment Off a corresponding byte range occupied by the random access point, and a presence indicator, the random access point indicator indicating whether there exists a random access point within the media segment. The method of claim 1, wherein the media segment is generated by concatenating the plurality of media segments. The method as recited in claim 2, further comprising the steps of: The media segment is generated in the memory, and wherein the plurality of media segments used to generate the media segment are emptied from the cache memory after the media segment is generated in the cache. The method of claim 1, further comprising the steps of: generating a single media presentation description (MPD) file, the MPD file storing a first representation comprising the plurality of media segments and the media presentation regarding the media presentation Information including a second representation of the plurality of media segments. The method of claim 4, wherein the MPD includes an attribute to indicate a frequency at which a random access point occurs within the second representation. The method of claim 5, wherein the frequency is a time period. The method of claim 5, wherein the frequency is a number of media segments. A media server, comprising: a processor configured to: obtain the content to be served; generate a plurality of media segments representing the content and encoded according to an encoding protocol, the encoding protocol including a media presentation Encoding one or more frames into each media segment, wherein a random access point is available in each media segment; generating a plurality of media segments encoded according to the encoding protocol (media Fragments), wherein a media segment includes the plurality of media segments, and wherein at least some of the plurality of media segments comprise random access points and at least some of the media segments do not include random access points, a random access The point includes a position in the segment at which a decoder can decode the subsequent segment of the random access point without being affected by the segment before the random access point; and generate an index for the media segment (segment) Index), the segment index includes: a presentation time range for each media segment in the media segment, a corresponding byte range occupied by each media segment in the media segment, and a random access point existence An indicator, the random access point presence indicator indicates whether a random access point exists within each media segment. The media server as recited in claim 8, wherein the media segment is generated by concatenating the plurality of media segments. The media server as recited in claim 9, wherein the processor is further configured to: generate the media segment in a cache memory, and wherein when the media segment is generated in the cache memory, The plurality of media segments used to generate the media segment are emptied in the cache memory. The media server as recited in claim 8, wherein the processor is further configured to: generate a single media presentation description (MPD) file, the MPD file storing a first plurality of media segments including the plurality of media segments for the media presentation And indicating information about a second representation of the plurality of media segments presented by the media. A media server as recited in claim 11, wherein the MPD includes an attribute to indicate a frequency at which a random access point occurs within the second representation. The media server as recited in claim 12, wherein the frequency is a time period. The media server as recited in claim 13, wherein the frequency is a number of media segments. A non-transitory computer readable medium storing computer executable instructions to, when executed, cause one or more computing devices to: obtain the content to be served; generate a representation of the content and a plurality of media segments encoded according to an encoding protocol, the encoding protocol including a media presentation encoding one or more frames in each media segment, wherein each media segment has a random memory Taking a point; generating a plurality of media fragments encoded according to the encoding protocol, wherein a media segment includes the plurality of media segments, and wherein at least some of the plurality of media segments comprise random memory Taking a point and at least some of the media segments do not include a random access point, and a random access point includes a position in the segment at which a decoder can decode the subsequent segments of the random access point without being subject to the random The impact of the fragment before the access point; and generate a segment index for the media segment, the segment index package Included: a presentation time range for each media segment in the media segment, a corresponding byte range occupied by each media segment in the media segment, and a random access point presence indicator, the random storage The pick presence indicator indicates whether a random access point exists within each media segment.