GB2588930A - Multimedia system & method - Google Patents

Abstract

A client receiving live-streamed media at low latency is unable to determine whether a channel used for the streaming is able to support higher data rates which would enable receipt of higher quality streams. In one aspect of the invention, the client measures the average received data rate over a plurality of intervals, ignoring intervals with zero received data i.e. intervals corresponding to chunk creation at a streaming source. In a second aspect the client receives data relating to TCP properties, such as congestion window and round trip time (RTT), in HTTP headers or trailers sent e.g. by a content server in response to GET requests from the client. The client can then determine an available bandwidth of the link using one or both of these aspects, and thus determine whether to request streaming at a higher data rate.

Description

MULTIMEDIA SYSTEM & METHOD

BACKGROUND OF THE INVENTION

This invention relates to streaming of media, in particular media that is arranged as segments and created and made available in chunks that are delivered successively over HTTP.

Media which includes audio-video content may be provided via a variety of streaming technologies. So called "live" streaming involves the capturing of audio-video content and delivering onto a network as soon as possible after capture. The streaming is "live" in the sense that the media is delivered to a network, and ultimately to clients, with low latency. Such an approach may be contrasted to pre-stored or "on demand" content which is fully recorded and available for download prior to the download commencing.

When streaming media over unmanaged internet connections, variable network performance can be encountered. There is a trade-off between sound and picture quality, which requires the highest possible bitrate, and reliable performance, which needs the bitrate of the media to be lower than the capacity of the connection between the client and the media server. This capacity can vary due to other usage of the links, both near the ends of the connection (e.g. at the servers or in a home) and across the networks it is carried on. The variation means that during the period when a viewer may be watching a live stream, a higher quality may be possible sometimes but not at other times. One way to cope with this kind of variation is using an adaptive bitrate system in which the bitrate of the media stream being delivered can be altered according to the available capacity.

Modern adaptive bitrate systems such as M PEG DASH (Dynamic Streaming over HTTP) and HTTP Live Streaming (HLS) divide the media stream into segments (typically 2-10 seconds in duration) and make available a number of different representations of the content, encoded at different bitrates (and therefore, with different quality levels). The client then selects from the available representations based on its assessment of which quality level is likely to be playable without interruption, based on the current network conditions. The client typically chooses between the available representations at a segment boundary.

With on-demand content clients typically buffer a quantity of media data ahead of the current play position so that playback can continue if the network conditions deteriorate without necessarily immediately switching to a different quality level. Even with live content, if latency is not critical, buffer durations of 10-30 seconds are common. With this duration of media stored in a buffer, the client can make use of simple measurements of buffer occupancy and/or segment download time interval to decide whether the currently selected representation is passing through the network at a sustainable rate and playback can continue at the current quality level. The client can also estimate the maximum bitrate that the network could sustain by considering the rate at which the buffered duration increases or the ratio between the segment download time interval and the segment's media duration.

The client can then decide when it is safe to switch to a higher quality representation whilst minimising the risk of stalling.

SUMMARY OF THE INVENTION

We have appreciated problems in determining when to switch between differing qualities of representation of streaming content when retrieving live streaming content in contrast to on-demand content, especially when it is important to limit the latency of the live stream.

We have particularly appreciated problems in determining, at a client, whether to select a different representation of media that arise due to the reliability of information obtained at the client relating to prevailing network conditions. The network conditions include, in particular, the achievable download rate for the media content and the various factors that influence it.

We have further appreciated particular problems in relation to streamed media that is arranged as segments, each segment being divided into multiple chunks and delivered over HTTP. The invention is defined in the claims to which reference is directed. Preferred features are set out in dependent claims.

The inventive concept broadly resides in providing improved data at a client to allow the client to determine whether to select a different representation of streaming media whilst retrieving that media as successively received chunks. The invention is provided as two separate improvements.

According to a first improvement, there is provided a method operable at a client for selecting from multiple representations of streamed media content from a server, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: selecting a first representation of the streamed media content of a first bandwidth; requesting a segment of the first representation from the server; receiving data for the segment of the first representation in an HTTP response from the server; measuring, at an application level, a quantity of data for the segment received during each of a plurality of intervals; selecting intervals from the plurality of intervals during which data is received; determining an average data rate from the selected intervals; and based on the average data rate satisfying a condition, determining to select a second representation of a second bandwidth that is different from the first bandwidth and requesting a segment of the second representation from the server.

This improvement allows the client to more accurately determine an available download rate or bandwidth when chunks of a segment become available progressively as they are created and hence when the total time taken to receive a segment may no longer be representative of the available bandwidth. In this case, when the bandwidth exceeds that necessary to deliver the segment in real time, most of the segment typically still arrives in real time because the rate becomes limited by the rate at which the chunks are created. Measuring the bandwidth in a traditional manner over the whole segment duration results in a measurement that is lower than the actual bandwidth available in such circumstances. By obtaining a more accurate bandwidth measurement over the periods when data is actually being sent, the improvement allows the client to more appropriately select a suitable representation of streaming media. In particular, it allows a client to determine that greater bandwidth is available such that a higher quality higher bandwidth representation of audio-video may be selected instead of the currently received representation. Such an approach provides an advantage that a decision can be taken at the client to select a higher quality representation even when the client maintains only a small buffer such that it can play the content with minimum latency. It also provides a greater chance of success that selecting the higher quality representation and issuing a request for that representation will achieve a successful download based on the prevailing bandwidth actually available from server to client. Preferably, when the client determines that the average data rate as determined above is above a threshold, a higher bandwidth representation of the streaming media is selected for download. The threshold may be related to the bandwidth of the second representation and optionally varied by one or more other factors so as to improve the chances of successfully obtaining the higher quality higher bandwidth representation.

The second improvement provides an improved determination of whether a second representation of streaming media may be successfully selected, requested and retrieved based on one or more transport layer metrics. Such transport layer metrics may include a congestion window and round trip time. Such metrics are available at the transport layer, for example TCP, but, we have appreciated, are not usually utilised at the application layer by either the server providing responses or the client issuing requests. Some such metrics, such as the congestion window for the flow from the server to the client, are not available at the client at all, even at the transport layer.

According to the second improvement, there is provided a method operable at a client for selecting from multiple representations of streamed media content from a server, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: selecting a first representation of the streamed media content of a first bandwidth; requesting a segment of the first representation from the server; receiving data for the segment of the first representation in an HTTP response from the server; receiving from the server over HTTP one or more transport layer metrics obtained from a transport layer of the server and indicative of network conditions between the server and client; based on the one or more transport layer metrics, determining to select a second representation of a second bandwidth that is different from the first bandwidth and requesting a segment of the second representation.

The provision of the transport layer metrics obtained from the transport layer at the server to a client allows the client to obtain a measure of data throughput from those metrics. The transport layer metrics may directly indicate a bandwidth or measure of throughput that is available from server to client, or may be used to derive such a measure of throughput from the server to the client. The client itself may derive the measure or the server may be arranged to derive that measure and provide it to the client. In either approach, the client obtains a better measure of available data throughput derived from the transport layer metrics that are available at the server. Such metrics are a more accurate representation of the available throughput in comparison to the average data rate of entire segments received at the client, particularly in the case where the segments are made up of chunks that become available progressively as they are created.

In both the first and second improvement more accurate determination of when to select, request and download different representations of streaming media available at different qualities is achieved.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described in more detail by way of example with reference to the drawings, in which: Figure 1: is a graph showing an example transfer rate with time of media data from client to server for a first improvement; Figure 2: shows the count of transferred bytes of data at time intervals of the data rate shown in Figure 1; Figure 3: shows the count of bytes of Figure 2 indicating intervals during which no data is received; Figure 4: shows the count of bytes of Figure 2 with the intervals during which no data is received removed; Figure 5: shows the overall message flow of a second improvement; Figure 6: shows a client-server arrangement to which the invention may be applied.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention may be embodied in a method of selecting, retrieving or processing streamed media, devices for sending or receiving streamed media, transmitters, receivers and systems involving such a method.

The embodiments of the invention will be described in relation to the provision of streaming media from a server over a network to a client. Figure 6 is a schematic view of such an arrangement to which the invention may be applied. The system 1 of Figure 6 comprises a server 10 which is receiving audio-video content to be provided as streaming media over a network 12, which is typically a best effort network such as the Internet comprising one or more intermediate nodes, to a client. The network 12 may include for example various content delivery networks or other intermediate devices by which the streaming data is provided to the client 16. The client 16 may be any media streaming application operable on a client device such as a television, PC, tablet or mobile telephone. It may be an application built-in to such a device, installed on the device by a user, or transiently loaded on the device.

The arrangement embodying the invention preferably operates an adaptive bit rate system such as either MPEG DASH (Dynamic Streaming over HTTP) or HLS (HTTP Live Streaming). In such adaptive bitrate systems, data is provided by dividing the media stream into segments. The segments themselves may be formed from chunks. The terminology "chunk" is a division of a segment. For example, the format Common Media Application Format for Segmented Media (CMAF) defines a "CMAF Chunk" which is one example of the way in which segments are divided into chunks. The purpose of chunks is that they carry a consecutive series of media samples (such as video frames or audio frames) and can be created as soon as that series of media samples has been encoded. These chunks may exist as separate files available on a disc or server to be transferred over HTTP. A series of chunks forming one segment may preferably be concatenated to form a single HTTP response from the server to the client. Sending multiple chunks progressively in a single HTTP response is achieved using a technique such as the Chunked Transfer Encoding feature of HTTP/1.1 or the framing structures of HTTP/2 to send each chunk before the sizes of future chunks are known. The assembled segment of chunks is received by the client for playout on a device. As previously discussed, for on-demand or non-live streaming, the client is able to monitor a buffer level to determine whether segments are being received at a greater rate than they are needed at a client device. If so, this suggests the client may be able to attempt switching up to a higher bandwidth version of a representation being delivered. This is because the measurement of segments downloaded shows a genuine indication of the actual bandwidth available from server to client.

However, as will be discussed, in live streaming of media, segments may arrive in real time or near real time, each segment typically comprising somewhere between 2 and 10 seconds of media. The chunks, being divisions of segments, in for example, Common Media Application Format for Segmented Media (CMAF) are delivered from server to client as fast as they are created at the server from the incoming audio-video. This allows the chunks to be transmitted as soon as each chunk is available but before an entire segment is complete. This provides a progressive download arrangement so that chunks being produced in stages are transferred as soon as they are created and prior to the completion of each segment.

The server 10 and client 16 should be understood as referring to either the hardware or software available at these locations. In particular, the client 16 refers to the client application that receives a stream of media content over a reliable transport protocol such as TCP or QUIC and in response to a HTTP request. The arrangement of such a live streaming system 1 has the aim of providing a live media stream as quickly as possible to a client 16 and so to a viewer. However, the segment and chunk based approach described when used in prior systems does not adequately provide for the client to determine when to select an alternative representation of the streaming media.

Sometimes, a delay of 10-30 seconds is unacceptable for a live media stream. For example, viewers might be watching a football match over the Internet close to neighbours watching the same match via broadcast. The Internet viewer may find it very frustrating to hear someone else's reaction to a goal a long time before they see it themselves. It also matters for viewers watching a live event and also following others' reactions on social media. They prefer not to read comments about things they have yet to see.

Both MPEG DASH and HLS can be operated with lower latency by reducing the amount of data buffered by the client. When the size of the buffer gets close to or even less than the segment duration, it becomes necessary for a client to be able to receive and process the media segments progressively rather than as a complete unit. This can be achieved by structuring the segment in a way that it can be created and parsed progressively in chunks as previously discussed. The Common Media Application Formatfor Segmented Media (CMAF) defines a CMAF Chunk for this purpose. A media segment consisting of CMAF Chunks can then delivered using the Chunked Transfer Encoding feature of HTTP/1.1 or HTTP/2 framing.

If the segments are being received by the client when they are still being created at the server, the traditional ways of selecting the quality level to request no longer work because measurements of buffer occupancy and segment download time interval no longer provide information on excess bandwidth that might allow a higher quality representation to be selected. This is because the speed at which the data can be transferred is limited by the rate at which the data is being produced at the server: the client cannot get ahead of real time. Conventional approaches to detecting excess network throughput that could support a switch to a higher quality representation thus lead the client to believe that there is no excess throughput.

The embodiments of the invention will be described as a first improvement and a second improvement. The separation of the description is for simplicity of understanding, but it should be noted that the two improvements may be used in combination together. In particular, the determination at the client as to whether to select an alternative representation according to the first improvement may be combined with the additional data relating to available throughput provided by the second improvement. In this way, the client would be able to operate both the methods of first and second improvement and combine the results so as to provide an even better determination of whether to select an alternative representation.

Various ways in which the first and second improvement may be combined will be discussed later.

First Improvement The first improvement provides a way in which a streaming client can measure the approximate network throughput whilst progressively downloading a "media segment" consisting of a number of "chunks" that are transferred sequentially in a single HTTP response.

Because the chunks are generated in real time by the encoder, the traditional approach of measuring the time taken to download the media segment as a whole will never give a throughput figure significantly higher than the bitrate of the encoded media. This means that the client can't tell when there is excess bandwidth that might sustain a higher quality stream without trying it. In a low latency situation, trying a higher quality stream when there isn't sufficient throughput will very rapidly lead to the stream stalling.

The first improvement provides an arrangement at a client that measures the throughput in a different way by counting the bytes transferred in each of a number of short measurement intervals. Observing that there will typically be a burst of data transferred each time a chunk is produced by the encoder, followed by a gap until the next chunk, the algorithm determines an estimate of the real network throughput by considering only the measurement intervals in which some data is received, ignoring those where no data is received. This has the effect of measuring the throughput when data is being transferred whilst ignoring the periods when the link is idle awaiting the next chunk.

Figure 1 illustrates how the transfer rate of an example DASH media segment might vary over its delivery when (a) the client is streaming with low latency and (b) network throughput is greater than the segment's encoded bitrate. In this example, the segment consists of four CMAF chunks delivered via a single HTTP response using chunked transfer encoding. The time that the first byte of the response is received by the client is IF and the time that the last byte is received is TL.

As can be seen in Figure 1 the transfer rate shows a burst of activity from the time that the first byte is received followed by a gap and then subsequent second, third and fourth bursts of activity during which time the successive four CMAF chunks are delivered in a single HTTP response. The gaps between the chunks are because, in live streaming, the subsequent chunks have not been created at the point that the previous chunk has been completely transferred from server to client. This shows that the bandwidth available for transmission is greater than the data rate needed to transfer the streaming media at this quality of representation. In essence, each chunk is delivered faster than real time. The client wishes to determine, therefore, both the fact that the transfer rate is faster than that required of this quality of representation and also whether a change to select a higher quality representation is likely to be successful.

As a client's network interface receives TCP packets in response to a request for a media segment the data from those packets are passed to the application level by the client's network stack. In this arrangement, the client records the number of bytes of data that it receives within each consecutive equal-duration time interval, or bin, of the segment transfer. The time intervals are preferably of equal duration as this simplifies the calculation of the data rate, but the intervals could feasibly be of differing durations as long as the pattern of durations is known. The time intervals will be referred to as bins in subsequent description for simplicity.

Figure 2 shows the number of bytes received in each consecutive bin of the segment transfer shown in Figure 1. Each bin in this example has a duration d; this is a parameter whose value can affect the accuracy of the network throughput estimated by this approach.

The client then identifies which of the bins have zero bytes in them and removes them from its data set, as shown in Fig 3, leaving a data set consisting of only the bins containing a non-zero number of bytes, as shown in Figure 4.

The client can calculate an average transfer rate for the "active'. portions of the response represented by the data set in Figure 4 using the following calculation: 8T R = Nd Where: R is the average active transfer rate in bits per second, T is the total number of bytes in all bins, N is the number of non-empty bins, d is the bin duration in seconds, 8 is the factor to convert from bytes to bits.

This approach is very convenient to implement and uses measurements that low latency DVB DASH clients need to be able to make anyway for the "metrics reporting" feature required in that specification. It gives a reasonable estimate of the throughput that is good enough to be useful for bitrate adaptation.

The average data transfer rate may then be used within the client to determine whether to select a second representation of a second bandwidth that is different from the first bandwidth. Such an arrangement may be performed by comparing against a threshold and determining if the threshold is satisfied. The threshold may be variable and may depend upon the bandwidths of representations available. In particular, the threshold may relate to the next bandwidth of a representation available at higher quality and may be set so that it is likely that the bandwidth of the second representation may be reliably used and downloaded.

The choice of bin duration affects the accuracy of throughputs calculated by this arrangement. The bin duration should be shorter than the duration of media contained within each chunk and longer than twice the round trip time, RTT, of the connection. The latter condition avoids the removal from the throughput calculation of periods where no data is being received due to typical TCP retransmissions.

The first improvement provides advantages over other approaches, including for example: (1) Making a throughput calculation each time some data is received by the player and discarding those measurements that are close to the average throughput of the entire media segment.

(2) Inspecting the data being received to precisely identify the boundaries between chunks, measuring the throughput from the first byte of a chunk until the last. This approach has been disclosed in outline in the DVB DASH specification.

The value of the first improvement compared to these is that it will likely perform better than the first approach and is significantly less complex than the second approach.

Second Improvement The second improvement addresses the same problem of lack of reliable information at the client, but rather than trying to measure the data throughput as actually received at the client, takes a different approach in which information available at the transport layer is made available from the server to the client. This can either be performed by providing the information from the transport layer, such as TCP, from server to client for the client to then perform calculations, or calculations can be performed at the server and the result of those calculations provided to the client.

The particular implementation of the second improvement uses TCP, and provides information from the TCP layer to the client using HTTP. These protocols are very well known to the skilled person, but are described here briefly for completeness.

HTTP is a request/response protocol which exists in a number of versions.

Versions up to HTTP/1.1 are text based, where every message has the general form: START_LINE <CRLF> MESSAGE_HEADER <CRLF> MESSAGE HEADER <CRLF> MESSAGE HEADER <CRLF> <CRLF> MESSAGE BODY <CRLF> where <CRLF> stands for carriage-return+line-feed. Later versions, from HTTP/2 onwards, are specified with a binary syntax.

TCP is a transport protocol and provides the most common mechanism by which the HTTP requests and responses may be delivered. It provides for reliable in-order delivery of data which adapts to the available path capacity. It adapts to the path capacity by sending packets into the network and then reacting to the receipt of, or lack of, corresponding acknowledgement packets sent by the receiver. This technique for adapting to the available capacity is known as congestion control.

Whilst there are a number of congestion control algorithms the basic goal of TCP congestion control is for each source to determine how much capacity is available in the network, so that it knows how many packets it can safely have in transit. The number of packets it has in transit is maintained as a list, known as the congestion window. The receiver also maintains its own window to control the flow of data, whose maximum size is determined by its configuration and resource limits. The receiver window size is conveyed to the sender which may limit the usable portion of the congestion window used by the sender. TCP increases the congestion window upon the arrival of corresponding acknowledgement packets, whilst it decreases its size upon the detection of loss. The loss of packets is mainly detected through the use of timeouts with respect to reception of acknowledgement packets, though it may also be based upon packet delay measurements.

The approach of the second improvement is equally applicable to other window-based transport protocols, such as QUIC which is proposed as the underlying transport protocol for HTTP/3. The concepts of congestion window and other such parameters are also found in these newer transport protocols.

Adaptive streaming, be it using the HLS or DASH standards, transfers media over HTTP which in turn uses a transport protocol. The transport protocol, TCP in this case, uses the approach called congestion control to avoid sending data faster than the network can deliver it. As noted above, it does this by the sender calculating and maintaining a "congestion window" that limits how much data it will send whilst waiting for it to be acknowledged by the receiver. Combining the congestion window with an estimate of the round-trip time (also calculated by TCP) gives an estimate of the throughput of the connection. Certain aspects of this information, such as the congestion window, are only known to the sending end TCP implementation but not the receiving end.

The second improvement is to modify the edge-server that sends media segments to the client so that it passes to the client an estimate of the connection throughput derived from the TCP properties. This approach is illustrated in Figure 5. In this case the TCP metrics are passed by means of a 'Transport-Info' HTTP header. An HTTP trailer could be used as well or instead. The client can then use the metrics as part of its strategy for determining whether to switch to a higher or lower quality stream, as appropriate to the current network conditions.

The TCP transport information may be delivered to the client in the form of an HTTP header or trailer, nominally named 'Transport-Info', and returned in response to an HTTP GET, HEAD or OPTIONS request, or in a message body response from the edge server in reply to a request for a designated specific URL (e.g. /transport-info). The header or response may contain single or multiple timestamped entries, where each entry starts with a string identifier generated by the edge server that inserted the header. The contents of an entry are detailed as follows: * Exactly one parameter whose name is "ts", and whose value is a float indicating the measurement timestamp in seconds since UNIX epoch.

* Optionally one parameter whose name is "rtt", and whose value is a float, in milliseconds, indicating the server's estimate of the Round-Trip Time from its transport layer.

* Optionally one parameter whose name is "rttvar", and whose value is a float, in milliseconds, indicating the server's estimate of the variation of the Round-Trip Time from its transport layer.

* Optionally one parameter whose name is "cwnd", and whose value is an integer, conveying the size of the server's congestion window in packets.

* Optionally one parameter whose name is "rcv_space", and whose value is an integer, conveying the size of the receiver's window in bytes.

* Optionally one parameter whose name is "mss", and whose value is an integer, conveying the size of the server's Maximum Segment Size in bytes.

* Optionally one parameter whose name is "dstport", and whose value is an integer, conveying the server's destination port of this connection for correlation of measurements between requests.

* Optionally one parameter whose name is "send_rate", and whose value is a float, in bits per second, conveying the server's calculation of the sending rate for this connection.

* Optionally one parameter whose name is "cc_algo", and whose value is string, conveying the name of congestion control algorithm used by the server for this connection.

Here is an example of a header with a single entry: Transport-Info = ExampleEdge; ts=1567176968.69; cwnd=24; rtt=250; mss=1460; rttvar= 10; dstport=12345 Here is an example of a header with multiple entries, containing server calculated send rate: Transport-Info = ExampleEdge2; ts=1567176968 69, send_rate=710000, 20 dstport=12345, ExampleEdge2; ts=1567176969.97; send_rate=720000; dstport=12345 This approach has the advantage of being backwards compatible at both ends: a server can safely include the information as an HTTP header, without affecting clients that don't understand it, and clients can be made to understand this header and use it where available but use other methods where it isn't.

Other approaches where the client is given additional information about the network are known but these involve a network operator providing the information as a separate service and are not appropriate for use in an unmanaged network.

Whilst it is understood that transport layer metrics may only provide an instantaneous view on the transport state, the Transport-Info header is designed to allow for delivery of multiple timestamped entries in a single header.

Another technique is for the client to perform HEAD or OPTIONS * requests for a resource at the same origin, or utilise a specific Transport-Info query location on the server that provides a minimal response containing a Transport-Info header with the current metrics, as may be seen in the optional box in Figure 5. This can be particularly advantageous when used in conjunction with HTTP/2 where it can increase temporal coverage by allowing a client to perform multiple requests on the same transport connection in parallel with an existing ongoing session of interest. For example, whilst a video segment is downloaded using HTTP/2, it is possible to make multiple queries for the Transport-Info header to provide for the desired level of temporal coverage. Whilst the HTTP/2 priorities can affect the allocation of capacity between streams, the main use-case for the Transport-Info header is for information regarding sustained flows such as media delivery which tend to consist of a known limited number of flows to the same origin so the priorities would not affect the calculations.

Optionally, an HTTP/2 server could use the Server Push feature of HTTP/2 to push the Transport-Info responses automatically to avoid the client needing to explicitly request them.

Preferably, if the client makes a separate HTTP request for the Transport-Info header, it does so using the same transport connection as that used to deliver the media. This makes it unambiguous to the server which transport connection metrics are being requested. Alternatively, the client can make reference to the transport connection explicitly by including client-end port number in the HTTP request and the server can be arranged to interpret this and return the transport metrics for the requested connection.

The arrangement shown in Figure 5 will now be described in greater detail. The edge CDN is a server that provides streaming media to a client. This could be the original source of the media but in the example given is an intermediate server between the origin server and the client. The arrangement in Figure 5 shows the message flows. First, a GET request is sent over HTTP from the client to the edge CDN server. The GET request may then be transferred to the origin server and a HTTP response received. At this point, the edge CDN server obtains one or more transport layer metrics from a transport layer of the server that indicate available bandwidth from the server to the client. The metrics may comprise at least a congestion window and round trip time or metrics derived from the congestion window and round trip time. The transport layer metrics are then included in an HTTP response to the client. The HTTP response then continues with HTTP chunked delivery, until that complete segment has been delivered. That HTTP connection is usually kept open so that subsequent requests can utilise it.

Some key points may be noted in relation to the second improvement. The transport information included in the HTTP response may be either the congestion window and round trip time derived directly from the TCP layer and provided from server to client. Alternatively, the transport information may comprise a measure of data throughput between client and server derived at the server from congestion window and round trip time and provided to the client.

The improvement addresses the fact that in modern web clients it is not currently possible to obtain such low level information about transport-layer connections. As a result, clients often resort to application level measurements, to infer such things as the current delivery rate, for example, in streaming media adaptive bitrate (ABR) algorithms implemented in Javascript. Although the Web Hypertext Application Technology Working Group's Fetch API, in conjunction with the Streams API, allows for progressive downloading of objects, it provides limited information on the timing of data received so the accuracy with which the download speed can be calculated is limited. Whilst this is not so much of an issue for fixed length content, it is of particular relevance where content is progressively made available in chunks within a single HTTP response, as is done for low latency video delivery.

Provision of the Transport-Info header metrics is possible by utilising operating system support on Linux and BSD based systems. Specifically, it is possible to obtain the 'tcp_info' data structure, which contains the aforementioned TCP metrics, by calling the getsockopt() function with the network socket file descriptor of interest, at IPPROTO_TCP level, with the TCP_INFO option name. This function would be called by the edge-server with the network socket file descriptor of the connection between the it and client.

There is in-built support to access this information at the server in the widely used Nginx/Openresty server using its variables var.tcpinfo_rtt etc. Apache Traffic Server provides support using the TCPlnfo plugin. Varnish provides access to tcp_info using their vmod_tcp module. Node.js has libraries such as nodejs_tcpinfo which provide support. Whilst most of the implementations do not provide access to the TCP MSS it is available from the tcp_info data structure so it would be straightforward to provide access to such information.

Calculation of the TCP transmission rate is possible using the information provided by the metrics Congestion Window (CWND) in packets, Receiver Window (RCV_SPACE) in bytes, Round Trip Time (RTT) in seconds, and knowledge of the TCP Maximum Segment Size (MSS) in bytes, the equation being as follows: 8 * send window se nd_rate

RTT

Where: send_rate is the TCP sending rate in bits per second send_window = MIN(RCV_SPACE, CWND*MSS) Preferably, this calculation is performed using the actual MSS value being used. Some server software architectures make it harder to obtain this value than the other parameters. A less accurate estimate of send_rate can be calculated without having access to the actual MSS value in use if a default value is assumed.

Because common values of MSS lie within a fairly small range, an estimate of send_rate calculated with such a default value may still be useful. It is preferable in such circumstances to use a low estimate of MSS so as not to over-estimate the send rate. The send_window is preferably calculated using both CWND and RCV_SPACE as shown, but if the RCV_SPACE is not available the send_window may be approximated by just using the CWND, based on the assumption that the send rate is not being limited by the receiver.

For other window-based transport protocols such as QUIC, the send rate can be calculated using the same principles using the corresponding parameters for the transport protocol in use.

Looked at from the point of view of the client in Figure 5, the arrangement comprises selecting a first representation of streamed media content and issuing a request for a segment as the GET request to the server that can provide that media representation. The client is unaware of whether this is an edge CDN server or the origin server but simply receives a HTTP response that includes transport information. If the transport information comprises the congestion window and round trip time, then the client will perform a calculation of the data throughput based on those transport layer metrics. On the other hand, if the server has performed the calculation of data throughput using the congestion window and round trip time, then the transport layer metrics provide a measure of data throughput from the edge CDN server to the client. The client then receives the requested representation via HTTP chunked delivery.

The client may determine based on the transport layer metrics received in the HTTP response that a higher bandwidth representation of the streamed media content may be retrieved. If so, the client will determine that fact and request a segment of the second representation. The second representation would typically be a higher bandwidth representation of the media. The ability to download the higher bandwidth representation would be determined if the data throughput indicated by the transport layer metrics is such that the measure of data throughput satisfies a threshold for which the second representation may be successfully streamed. The threshold may be a direct measure of the bandwidth of the second representation or may include a factor based on reliability.

The information relating to the measure of data throughput may be used by the client in a variety of further ways. For example, if the measure of data throughput shows that the data throughput is increasing this may be used as a trigger to select a higher bandwidth representation. If it is decreasing this may be used as a trigger to select a lower bandwidth representation.

Another more general transport level metric that may be conveyed is the type and features of the current congestion control algorithm being utilised by the server. These may be used to further inform the client's calculations of thresholds and behaviour. For example, TCP Cubic congestion is considered a more efficient protocol for high speed flows, and its reaction to packet loss is not as severe as TCP Reno. A congestion control feature that could impact behaviour is knowledge of use of Explicit Congestion Notification (ECN) which could indicate that the flow is less likely experience loss.

Information regarding the variability of the RTT, in the form the RU variation (RTTVAR) TCP metric may also be useful to the client in deciding how to behave with respect to thresholds. For example, if the RTTVAR is high then it may be more cautious in switching representation.

The calculation of the TCP transmission rate, this being a measure of data throughput, may be undertaken either at the client or at the server as previously described.

Combining the First and Second Improvements The first and second improvements can both be used to estimate the achievable throughput of media data but each improvement determines the estimate in a different way: the first improvement is based on recent rates of data flow at an application level, whereas the second improvement uses information from the transport layer from one or more points in time and is indicative of both past and present rates of transmission from the server.

Estimates from the two improvements can be combined by a client. By taking into account estimates made in both ways, improved reliability can be achieved by minimising the chances of selecting a second representation whose bandwidth exceeds the available throughput of the network.

The improvements can be combined in a number of ways: A client wishing to minimise the possibility of an interruption to the presentation of media can take the lower value of the throughput estimate from each of the improvements and based on that, make a decision about which representation to select for the second representation.

Alternatively, a client could use the average of the throughput estimates from each improvement.

The throughput estimates from each improvement may be more accurate for some clients than for others. For example, a client that only receives data at the application layer at infrequent intervals will have to use a larger measurement interval with the first improvement and may therefore underestimate the throughput actually available.

A client that is aware of the likely accuracy of each estimate in the particular circumstances existing at the time (such as the manner in which the data passes up to the application layer as described above) can use that information to apply a weighting, taking a greater or lesser account of the estimate from one improvement according to the likely accuracy.

Different weightings can be applied to estimates that fall below the bandwidth of the first representation (and indicate a need to switch to a lower quality representation) and those that are higher than the bandwidth of the first representation.

It is also possible to use information from the second improvement to assist in the calculation of the estimate in the first improvement. As described previously, the intervals used in the calculation of the estimate should be no less than twice the network round trip time. Metrics available to the client using the second improvement can be used to determine the round trip time value to use.

Claims (75)

1. CLAIMS1. A method operable at a client for selecting from multiple representations of streamed media content from a server, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: - selecting a first representation of the streamed media content of a first bandwidth; -requesting a segment of the first representation from the server; -receiving data for the segment of the first representation in an HTTP response from the server; -measuring, at an application level, a quantity of data for the segment received during each of a plurality of intervals; -selecting intervals from the plurality of intervals during which data is received; - determining an average data rate from the quantity of data received during the selected intervals; and - based on the average data rate satisfying a condition, determining to select a second representation of a second bandwidth that is different from the first bandwidth and requesting a segment of the second representation from the server.
2. A method according to claim 1, wherein satisfying a condition comprises determining whether the average data rate satisfies a threshold.
3. 3. A method according to claim 2, wherein if the average data rate is above a threshold, the second representation selected is a higher bandwidth representation than the first representation.
4. 4. A method according to claim 3, wherein the threshold is related to the bandwidth of the second representation.
5. 5. A method according to and of claims 2 to 4, wherein the threshold is determined based on a bandwidth of the second representation adjusted by a safety factor.
6. 6. A method according to any preceding claim, wherein the duration of the intervals is variable based on one or more network factors, preferably including network round trip time.
7. 7. A method according to any preceding claim, wherein the duration of the intervals is variable based on the duration of chunks in the first representation.
8. 8. A method according to any preceding claim, wherein the duration of the intervals is the same for all intervals.
9. 9. A method according to any preceding claim, wherein the duration of the intervals is shorter than the duration of media contained within each chunk.
10. 10. A method according to any preceding claim, wherein the duration of the intervals is longer than twice the round trip time, RTT, of the connection between the server and the client.
11. 11. A method according to any preceding claim, wherein the selected intervals do not include intervals for which one or more network conditions occur, preferably including drop outs, local effects or network effects.
12. 12. A method according to any preceding claim, wherein the streamed media content comprises real time or low latency media content.
13. 13. A client configured to select from multiple representations of streamed media content from a server, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: -means for selecting a first representation of the streamed media content of a first bandwidth; -means for requesting a segment of the first representation from the server; -means for receiving data for the segment of the first representation in an HTTP response from the server; -means for measuring, at an application level, a quantity of data for the segment received during each of a plurality of intervals; -means for selecting intervals from the plurality of intervals during which data is received; -means for determining an average data rate from the quantity of data received during the selected intervals; and -based on the average data rate satisfying a condition, means for determining to select a second representation of a second bandwidth that is different from the first bandwidth and requesting a segment of the second representation from the server.
14. 14. A client according to claim 13, wherein satisfying a condition comprises determining whether the average data rate satisfies a threshold.
15. 15. A client according to claim 14, wherein if the average data rate is above a threshold, the second representation selected is a higher bandwidth representation than the first representation.
16. 16. A client according to claim 14, wherein the threshold is related to the bandwidth of the second representation.
17. 17. A client according to and of claims 14 to 16, wherein the threshold is determined based on a bandwidth of the second representation adjusted by a safety factor.
18. 18. A client according to any of claims 13 to 17, wherein the duration of the intervals is variable based on one or more network factors, preferably including network round trip time.
19. 19. A client according to any of claims 13 to 18, wherein the duration of the intervals is variable based on the duration of chunks in the first representation.
20. 20. A client according to any of claims 13 to 19, wherein the duration of the intervals is the same for all intervals.
21. 21. A client according to any of claims 13 to 20, wherein the duration of the intervals is shorter than the duration of media contained within each chunk.
22. 22. A client according to any of claims 13 to 21, wherein the duration of the intervals is longer than twice the round trip time, RTT, of the connection between the server and the client.
23. 23. A client according to any of claims 13 to 22, wherein the selected intervals do not include intervals for which one or more network conditions occur, preferably including drop outs, local effects or network effects.
24. 24. A client according to any of claims 13 to 23, wherein the streamed media content comprises real time or low latency media content.25. A client according to any of claims 13 to 24, wherein the client is one of a television, PC, tablet or mobile device.
25. 25. A computer program comprising code which when executed on a device undertake the method of any of claims 1 to 12.
26. 26. A method operable at a client for selecting from multiple representations of streamed media content from a server, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: -selecting a first representation of the streamed media content of a first bandwidth; -requesting a segment of the first representation from the server; -receiving data for the segment of the first representation in an HTTP response from the server; -receiving from the server over HTTP one or more transport layer metrics obtained from a transport layer of the server and indicative of network conditions between the server and client; -based on the one or more transport layer metrics, determining to select a second representation of a second bandwidth that is different from the first bandwidth and requesting a segment of the second representation.
27. 27. A method according to claim 26, wherein the one or more transport layer metrics include at least a congestion window and round trip time, and wherein the client is arranged to derive a measure of data throughput between client and server from one or more transport layer metrics for use in the determining whether to select the second representation.
28. 28. A method according to claim 26, wherein the one or more transport layer metrics comprise a measure of data throughput between client and server derived at the server from at least a congestion window and round trip time.
29. 29. A method according to any of claims 27 or 28, wherein if the measure of data throughput indicates a decreasing throughput of data from server to client or a variation in throughput above a threshold, the second representation selected is a lower bandwidth representation than the first representation.
30. 30. A method according to any of claims 27 to 29, wherein if the measure of data throughput indicates an increasing throughput of data from server to client, the second representation selected is a higher bandwidth representation than the first representation.
31. 31. A method according to any of claims 27 to 30, wherein if the measure of throughput is above a threshold, the second representation selected is a higher bandwidth representation than the first representation.
32. 32. A method according to any of claims 27 to 31, wherein if the measure of throughput is below a threshold, the second representation selected is a lower bandwidth representation than the first representation.
33. 33. A method according to claim 31 or 32, wherein the threshold is dependent on the bandwidth of the second representation.
34. 34. A method according to any of claims 26 to 33, wherein the threshold is chosen based upon one or more of the transport layer metrics.
35. 35. A method according to any of claims 26 to 34, wherein the one or more transport layer metrics are received in an HTTP header with the HTTP response carrying data for the segment.
36. 36. A method according to any of claims 26 to 35, wherein the one or more transport layer metrics are received in an HTTP trailer with the HTTP response carrying data for the segment.
37. 37. A method according to any preceding claim, wherein receiving from the server over HTTP one or more transport layer metrics involves making one or more separate HTTP requests using the same transport layer connection as the HTTP response carrying data for the segment.
38. 38. A method according to claim 37, wherein the separate HTTP requests are made using a version of HTTP that supports multiplexing and occur during the receiving of data for the segment. 39. 40. 41. 42. 43. 44. 45.
39. A method according to any of claims 26 to 38, wherein the streamed media content comprises real time or low latency media content.
40. A method operable at a server for delivering representations of streamed media content, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: -delivering data to the client for a segment of a first representation in an HTTP response; -obtaining one or more transport layer metrics from a transport layer of the server indicative of network conditions between the server and client; -delivering the one or more transport layer metrics from the server to the client over HTTP.
41. A method according to claim 40, wherein the one or more transport layer metrics include at least a congestion window and round trip time.
42. A method according to claim 40, wherein the one or more transport layer metrics comprise a measure of data throughput between client and server derived at the server from at least a congestion window and round trip time.
43. A method according to any of claims 40 to 42, wherein the one or more transport layer metrics are sent in an HTTP header with the HTTP response carrying data for the segment.
44. A method according to any of claims 40 to 43, wherein the one or more transport layer metrics are sent in an HTTP trailer with the HTTP response carrying data for the segment.
45. A method according to any of claims 40 to 44, wherein delivering the one or more transport layer metrics occurs in response to a separate HTTP request.
46. 46. A method according to claim 45, wherein the separate HTTP request is made using the same transport layer connection as the HTTP response carrying data for the segment.
47. 47. A method according to claim 46, wherein the delivering uses use a version of HTTP that supports multiplexing and wherein delivering the one or more transport layer metrics occurs during the delivering of data for the segment.
48. 48. A method according to claim 46 or 47, wherein delivering the one or more transport layer metrics uses server push.
49. 49. A method according to any of claims 45 to 48, wherein delivering the one or more transport layer metrics comprises more than one HTTP response.
50. 50. A client configured to select from multiple representations of streamed media content from a server, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: -means for selecting a first representation of the streamed media content of a first bandwidth; - means for requesting a segment of the first representation from the server; - means for receiving data for the segment of the first representation in an HTTP response from the server; -means for receiving from the server over HTTP one or more transport layer metrics obtained from a transport layer of the server and indicative of network conditions between the server and client; -based on the one or more transport layer metrics, means for determining to select a second representation of a second bandwidth that is different from the first bandwidth and requesting a segment of the second representation.
51. 51. A client according to claim 50, wherein the one or more transport layer metrics include at least a congestion window and round trip time, and wherein the client is arranged to derive a measure of data throughput between client and server from one or more transport layer metrics for use in the determining whether to select the second representation.
52. 52. A client according to claim 50, wherein the one or more transport layer metrics comprise a measure of data throughput between client and server derived at the server from at least a congestion window and round trip time.
53. 53. A client according to any of claims 51 or 52, wherein if the measure of data throughput indicates a decreasing throughput of data from server to client or a variation in throughput above a threshold, the second representation selected is a lower bandwidth representation than the first representation.
54. 54. A client according to any of claims 51 to 53, wherein if the measure of data throughput indicates an increasing throughput of data from server to client, the second representation selected is a higher bandwidth representation than the first representation.
55. 55. A client according to any of claims 51 to 54, wherein if the measure of throughput is above a threshold, the second representation selected is a higher bandwidth representation than the first representation.
56. 56. A client according to any of claims 51 to 55, wherein if the measure of throughput is below a threshold, the second representation selected is a lower bandwidth representation than the first representation.
57. 57. A client according to claim 51 or 56, wherein the threshold is dependent on the bandwidth of the second representation.
58. 58. A client according to any of claims 51 to 57, wherein the threshold is chosen based upon one or more of the transport layer metrics.
59. 59. A client according to any of claims 50 to 58, wherein the one or more transport layer metrics are received in an HTTP header with the HTTP response carrying data for the segment. 60. 61. 62. 63. 64. 65. 66.
60. A client according to any of claims 50 to 59, wherein the one or more transport layer metrics are received in an HTTP trailer with the HTTP response carrying data for the segment.
61. A client according to any of claims 50 to 60, wherein receiving from the server over HTTP one or more transport layer metrics involves making one or more separate HTTP requests using the same transport layer connection as the HTTP response carrying data for the segment.
62. A client according to claim 61, wherein the separate HTTP requests are made using a version of HTTP that supports multiplexing and occur during the receiving of data for the segment.
63. A client according to any of claims 50 to 62, wherein the streamed media content comprises real time or low latency media content.
64. A server configured to deliver representations of streamed media content, each representation comprising segments that are created and made available in chunks and delivered successively over HTTP as the chunks are made available, comprising: - means for delivering data to the client for a segment of a first representation in an HTTP response; -means for obtaining one or more transport layer metrics from a transport layer of the server indicative of network conditions between the server and client; - means for delivering the one or more transport layer metrics from the server to the client over HTTP.
65. A server according to claim 64, wherein the one or more transport layer metrics include at least a congestion window and round trip time.
66. A server according to claim 64, wherein the one or more transport layer metrics comprise a measure of data throughput between client and server derived at the server from at least a congestion window and round trip time.
67. 67. A server according to any of claims 64 to 66, wherein the one or more transport layer metrics are sent in an HTTP header with the HTTP response carrying data for the segment.
68. 68. A server according to any of claims 64 to 67, wherein the one or more transport layer metrics are sent in an HTTP trailer with the HTTP response carrying data for the segment.
69. 69. A server according to any of claims 64 to 68, wherein delivering the one or more transport layer metrics occurs in response to a separate HTTP request.
70. 70. A server according to claim 69, wherein the separate HTTP request is made using the same transport layer connection as the HTTP response carrying data for the segment.
71. 71. A server according to claim 70, wherein the delivering uses use a version of HTTP that supports multiplexing and wherein delivering the one or more transport layer metrics occurs during the delivering of data for the segment.
72. 72. A server according to claim 64 to 71, wherein delivering the one or more transport layer metrics uses server push.
73. 73. A server according to any of claims 64 to 72, wherein delivering the one or more transport layer metrics comprises more than one HTTP response.
74. 74. A client according to any of claims 50 to 63, wherein the client is one of a television, PC, tablet or mobile device.
75. 75. A computer program comprising code which when executed on a device undertake the method of any of claims 26 to 49.