GB2480424A

GB2480424A - A system varies the bit rate of a media stream to a device in dependence upon the type of device and connection.

Info

Publication number: GB2480424A
Application number: GB1006015A
Authority: GB
Inventors: Jeremy Mayo Boswell; Jonathan Mark Kendrick
Original assignee: ROK INVESTMENT GROUP Ltd
Current assignee: ROK INVESTMENT GROUP Ltd
Priority date: 2010-04-10
Filing date: 2010-04-10
Publication date: 2011-11-23
Also published as: GB201006015D0

Abstract

A system for streaming media data to a mobile device 22 comprises: a streaming server 16 with a plurality of stream sources 34, 35, 36 streaming the same content with different stream configurations; a processor which connects a first stream source, streaming data with a first stream configuration to the mobile device; the processor calculating a stream performance factor based on the bandwidth utilised by the mobile device and preferably based on the amount of data received by the device buffer and/or upon a TCP burst test; wherein the processor determines if the stream performance factor is within a predetermined limit, and if not connects one or more different stream sources with different stream configurations to the device until such time that the stream performance factor is within the predetermined limit. The system may further comprise a database of mobile device performance factors wherein a device performance factor may be based one or more of: the device the system streams to, the device specification, a blit test.

Description

Variable bit rate streaming

Technical field

The present invention relates to the streaming of multimedia data, such as a video stream, to portable devices such as mobile telephone devices. In particular the invention relates to the apparatus arid method used to dynamically vary the bit rate of data streamed to take into account the device used and the strength of the connection.

Background

It is known to stream data to mobile telephone devices over data connections such as GPRS, UMTS, EGPRS etc. It is also known that these connections may vary over time resulting in a non-optimised data stream. For example, if a user switches from a GPRS network connection to a UMTS connection they would be able to receive streamed data at a higher bit rate, but the bitrate of the sent stream does not adjust accordingly.

Certain video standards also allow for changes in the bit stream rate. The H264 video standard used in many applications such as those from well known content providers such as YouTube (RTM). H264 allows for changes in resolution by marking certain video frames (inter coded frames such as P or B frames) as "discardable". When the connection from the streaming server is below a predetermined limit the frames that are marked as discardable are ignored thereby reducing the bandwidth requirement.

However, it is sometimes undesirable to discard frames as it may result in "jerky" video.

Another well known video encoding standard is the RealMedia (RTM) standard. This standard allows for files to be stored at a variable bitrate in the RMVB format.

When changing the bitrate of the stream, the user may notice a brightening or darkening of the video data as the bitrate increases or decreases respectively. Constant changes in the bitrate are distracting and undesirable as they impair user enjoyment of the streamed data. p

Therefore, it is desirable to be able to change the bitrate of data transfer according to the present conditions in a maimer that does not adversely affect user enjoyment of the multimedia.

The present invention is defined by the appended set of claims. In the context of this application, the term "multimedia data" embraces not only audiovisual data but also audio data and visual data separately.

Brief description of the drawings

Embodiments of the invention are now described, by way of example only, with reference to the accompanying drawings in which: Figure 1 is a schematic representation of the apparatus according to an aspect of the invention; Figure 2 is a schematic representation of the streaming server according to an aspect of the invention; and Figure 3 is a flow chart of the process used to determine the bitrate at which to stream data.

Detailed description of an embodiment

Figure 1 is a schematic representation of the apparatus according to an aspect of the invention.

There is shown the system 10 comprising: a content server 12; encryption server 14; streaming server 16; authentication server 18; database 20; mobile device 22; source update module 24; and service provider interface 26.

In a preferred embodiment of the invention, a user obtains multimedia data to their mobile device 22. The mobile device 22 is typically a mobile telephone but may be any portable device such as a PDA, tablet computer, etc which uses a mobile connection. The user has a subscription account so that they can access the content, * . though in other embodiments there are no restrictions as to whom may access the multimedia data. The multimedia data is typically a video stream, though it may also be any other media data such as audio.

The multimedia content is stored on the content server 12. The content server 12 may be a known commercially available server.

In the preferred embodiment, the multimedia content is encrypted so that only subscribers may access the content. The multimedia content is encrypted at the encryption server 14. The encryption server 14 may use known encryption techniques.

The encrypted content is passed from the encryption server 14 to the streaming server 16. The streaming server 16 is enabled to stream the encrypted content to one or more mobile devices 22 via a plurality of stream sources. Figure 2 is a schematic representation of the streaming server 16.

In Figure 2 there is shown the streaming server 16. The streaming server 16 has a number of ports that can be used for streaming media data to the mobile devices 22, Ports 31, 32 and 33 are examples of such. For ease of illustration, only three such ports are shown, it being understood that in practice the number of such ports could be lower or higher (usually the latter). A mobile device 22 is directed to connect to one of these ports to receive streamed media content. The streaming server contains a number of stream sources each streaming the same content (as identified by a common stream 1D) but at a different bit rate. Sources 34, 35 and 36 are examples of such. For ease of illustration, only three such sources are shown, it being understood that in practice the number of such sources could be lower or higher (usually the latter). The streaming server 16 contains a multiplexer 37 capable of feeding each port with a stream from one of the sources. In this way, a mobile device connected to a particular one of the ports can receive a particular one of the streams from a particular one of the stream sources, as selected by the multiplexer 37. The manner in which the multiplexer is controlled to make these connections will be described later.

The streaming server 16 is in communication with the authentication server 18 and the mobile devices 22. A mobile device 22 communicates with the streaming server 16 in a preferred embodiment using known TCP protocols. The authentication server 18 contains the details of users who are have subscribed to the service or have access rights to the content. In a preferred embodiment, a mobile device 22 connects to the streaming server 16 which passes the request to the authentication server 18. The authentication server 18 queries a database 20 of subscribers to see if the mobile device 22 is a subscriber's device. If the user is authenticated, the authentication server 18 informs the streaming server 16, which allows the connection between the streaming server 16 and the mobile device 22. If not, the streaming server 16 refuses the connection. The refusal or acceptance of the connection is made using the standard TCP initiation handshake protocol.

Furthermore, when device 22 is authenticated by the database 20, a stream source e.g. 34, 35, 36 is assigned to the device 22 by the source update module 24 and the stream from that source is delivered via the muliplexer 37 to the port to which the mobile device is connected. Therefore, the device 22 will receive streamed data from a selected one of the stream sources. In use, the source update module 24 may assign a different stream source to the device 22 and this process will be described later with reference to Figure 3. If the source update module 24 assigns a different source to the device 22, then the multiplexer 37 directs the stream from the newly selected source to the port to which the mobile device 22 is connected.

Optionally, there is a service provider interface 26 where the service provider can access the authentication server 18 and update subscriber information and retrieve information regarding the number of authentications, usage etc. Therefore, the apparatus provides a user the ability to view encrypted content from a particular source e.g. 34, 35, 36 of the streaming server 16 using a mobile device 22.

The streaming server 16 has several groups of stream source. For example, sources 34 and 35 might belong to one of these groups and source 36 might belong to another.

These groups are broadly based on the type of mobile device 22 that can be used by & * .1 the user to view the content. For example, a first group of said stream sources may be for high end devices that support Flash (RTM) or html5 video streams, whereas another group of said stream sources may be for devices that use Portable Media Center (RTM).

Within each of the groups, the stream sources stream the same content at different parameters. Each stream source in a group may have different audio and/or video component bitrates, frame rates, key-frame intervals, resolutions etc. Given the difference in these parameters, the device 22 may require one or more different codecs to play the different streams from different stream sources within the same group.

Therefore each stream is understood to have different configurations.

The source update module 24 monitors the bandwidth utilised by the device 22 and is enabled to decide if the source to which the device is currently connected, say source 34, is the optimal source. If the bitrate of the stream from source 34 is deemed to be too high or low, then the source update module 24 will assign the device 22 to another source, say source 35, which streams the same content but with different stream parameters. The process of the source update module 24 determining if the stream source needs to be changed is discussed in detail with respect to Figure 3.

When the source update module 24 determines that the stream source needs to be changed the streaming server 16 continues to stream data from current source until a key frame is reached. Once the key frame is reached, the source feeding the mobile device 22 is changed to the newly selected one. As the stream switches on key frames, which are encoded independently of other frames and can be used as the basis of encoding for subsequent frames of data, there are no lost frames of video data.

Therefore, the present apparatus allows for dynamic changing of the bitrate of the streamed data.

Figure 3 shows the flow chart of the process of selecting the initial stream source and making the decision as to whether the device 22 stays on the same stream source or is allocated to different stream source.

I

At step S102 a device performance factor for each device 22 is determined. In an embodiment, the device performance factor is predetermined for each type of mobile device 22. The device performance factor is dependent on a number of factors such as operating system used, data connections available, processor etc. In a preferred embodiment the device performance factor is determined by using a display blit performance test to determine the performance of the player. But testing is known in video rendering and measures the ability of the device 22 to render the images on its display. In an embodiment, the device performance factor for each device 22 is predetermined by the supplier, alternatively it is measured for each individual device 22.

Based on the device performance factor obtained at step Si 02, a stream source group is selected at step Si 04. Each group preferably relates to devices which have similar capabilities as determined by their device performance factors. This ensures that one or more devices 22 are not connected to a stream source, say stream source 34, which is streaming at a rate which the device 22 cannot process due to the constraints of the device 22 (e.g. a relatively slow processor).

Once the stream source group has been selected, a stream source within the group is selected at step Si 06. As stated above, each of these sources streams the same content but at different bitrates and preferably different video/audio resolutions. The stream source initially selected may be selected in a number of different manners. In an embodiment, the stream vendor (i.e. the entity that supplies the streamed content to users) may have an initial setting e.g. a UMTS bitrate. In further embodiments, the initial stream source may be based on the current network connection of the device 22.

For example, if the device's network connection is a GPRS connection then stream source 34 may be used. Alternatively, if the device's network connection is a UMTS connection then stream source 35 may be used. In yet another embodiment, the highest, median or lowest bitrate stream source may be selected. The initial stream source may be selected by the source update module 24. -7..

Once the stream source has been selected, the streaming server 16 streams the data to the device 22 at step S 108.

An aspect of the invention is the dynamic selection of the stream source so that the optimal bitrate stream is selected. At step SilO one or more stream performance factors are determined. Examples of useful stream performance factors are discussed later.

Once the stream performance factor or factors have been determined they are transmitted to the streaming server 16, and preferably the source update module 24.

This information is used to determine if a change in stream source is required at step Si 14.

It is found that constantly changing the stream source (and potentially bitrate) is undesirable. Constant changes in the stream quality result in brightening and darkening of the screen which is found to result in an unsatisfactory experience for the user. To overcome this problem at step S 112 the stream performance factor as determined at step SilO is checked to see if it remains within a predetermined range.

If the stream performance factor is determined to be within the acceptable limits at step S 112, then the process moves to step S116 and the mobile device continues to be fed from the same stream source as this indicates that the bitrate of the stream is within an optimal limit. The process then returns to step SilO whereupon the stream performance factor is recalculated.

If at step SI 12 it is determined that the stream performance factor indicates that a change of bitrate is required then the process moves to step Si 14 where a stream source is selected to feed the mobile device.

The selection of a new stream source at step Si 14 may be for the purpose of increasing or decreasing the bitrate depending on the stream performance factor. If the stream performance factor is determined to be below the accepted limit at step SI 12 a lower bitrate stream source is selected by the source update module 24 to feed the port

I

to which the mobile device 22 is connected. If the stream performance factor is above a predetermined size this may indicate spare capacity and that a high bitrate stream may be used, in which case, the source update module 24 selects a stream source of the appropriate higher bitrate.

As the streams have the same stream ID, the user will continue to view the same content despite a change in stream source. As described above, the switch between stream sources at step SI 14 occurs on a key frame to ensure that there is no break in the stream.

Preferably, once the source update module 24 has determined that a change of stream source is required, the source update module 24 informs the device 22 that a different stream source is being used and the device 22 can prepare accordingly i.e. initiate connection with TCP handshake protocol and load any codecs if required.

After step S 114, the process returns to step SilO whereupon the stream performance factor is recalculated.

Some examples of stream performance factors, and how they are measured, will now be discussed.

Consider the case where a mobile device 22 is consuming a stream from one particular stream source of the streaming server 16. There is a TCP transmit buffer in the streaming server 16 into which stream packets are queued until the streaming server is ready to transmit them to the mobile device 22 over the relevant port. The mobile device 22 has a TCP receive buffer for queuing received stream packets until the mobile device is ready to process them. In the ordinary course of streaming, the stream will be operating at some average bitrate. In order to measure one type of stream performance factor, the streaming server 16 will arrange a burst to appear in the stream. The burst is arranged to have the maximum bitrate that appears amongst the collection of stream sources. This is achieved by temporarily withholding data from the stream and then releasing it to the mobile device 22 as a burst at the aforementioned maximum bit rate. The burst does not affect the average bitrate of the stream (since it is created by holding back data that would otherwise have been delivered earlier) so that the burst does not unduly perturb the mobile device's attempt to play/present the stream to its user.

The stream packets are time stamped at entry to the streaming server's TCP transmit buffer and from these time stamps the bitrate of transmission from the streaming server can be calculated. Moreover, the stream packets are time stamped when they are read from the mobile device's TCP receive buffer and from these time stamps the bitrate at which the mobile device consumes the stream is calculated. The value of this latter bitrate for the stream packets of the burst is the stream performance factor in this example and is reported back to the source update module 24. It will be apparent to the skilled person that the bitrate at which the stream packets of the burst are consumed may be higher than the currently prevailing average bitrate, which would indicate to the source update module 24 that the mobile device 22 could be directed to switch to using a higher bit rate stream source of the streaming server 16.

Another, simpler stream performance factor is the rate at which stream packets are received at the mobile device 22. This is periodically reported back to the source update module. However, this stream performance factor can only indicate whether the mobile device is coping with the nominal bitrate of the currently selected stream source and caimot assess whether the mobile device could in fact presently cope with a switch to a high bitrate stream source.

A further stream performance factor, again of a simplistic form, is offered by the so-called "panic messages" that the mobile device 22 can issue if it finds itself unable to cope with the bitrate of the presently utilised stream source. Information contained in such messages will indicate to the source update module 24 that the mobile device 22 should switch to using a lower bitrate stream source of the streaming server 16.

Again, this stream performance factor cannot assess whether the mobile device could in fact presently cope with a switch to a high bitrate stream source.

Another straightforward stream performance factor is the state of fullness of the TCP transmit buffer that serves the port currently feeding the mobile device. If the amount

I I

of date queued in this buffer exceeds some predetermined threshold, then it can be assumed that the mobile device is not coping with the bitrate of the currently selected stream source. Again, this stream performance factor cannot assess whether the mobile device could in fact presently cope with a switch to a high bitrate stream source.

A number of stream performance factors have now been discussed and it will be apparent to the skilled person that these could be used individually or in some combination in deciding whether to switch a mobile device 22 to consuming streamed content from a different stream source having a different bitrate.

The first stream performance factor that was discussed above utilised time stamps of stream packets to measure bit rates. Often, TCP schemes use efficiency measures whereby several stream packets are put into the same TCP packet. When two stream packets are put into the same TCP packet, both stream packets then receive the same time stamp, which reduces the time resolution of the bitrate calculation founded on these time stamps. Accordingly, in one variant, the streaming server 16 disables these efficiency measures such that the stream packets of the measurement burst are each sent in a separate TCP so that each has its own time stamp.

In certain circumstances, the bandwidth receivable by the mobile device can drop drastically. If this happens the TCP transmit buffer feeding the output of the current stream source to the mobile device will accumulate data that will only slowly be released to the mobile device. Accordingly, if the mobile device then sends to the streaming server 16 a panic message or other stream performance factor indicating that it is failing to cope with the current stream, the mobile device will still face the task of handling the media data that is jammed in the TCP transmit buffer with the result than the streaming process becomes liable to fail.

To guard against this, the TCP transmit buffer can be kept short, so that the amount of data that can become jammed there is reduced, in turn reducing the chance of stream failure. As a further protective measure, the source update module 24 can be arranged to switch the mobile device 22 to the lowest bitrate stream source upon detecting the

C

TCP transmit buffer becoming highly occupied. This switch to the lowest bitrate stream source can then be followed by an analysis of, for example, the burst-based stream performance factor discussed above in order to determine if a higher bitrate stream source can once again be selected and if so, to determine which of the available higher bitrate stream sources is appropriate.

Claims

IClaims 1. A system for streaming media data to a mobile device, the system comprising: a streaming server with a plurality of stream sources, said stream sources streaming the same content with different stream configurations; a processor suitable for selectively connecting a first stream source, streaming data with a first stream configuration, of the streaming server to the mobile device; the processor calculating a stream performance factor based on the bandwidth utilised by the mobile device; wherein the processor determines if the stream performance factor is within a predetermined limit, and in the event the stream performance factor is outside the predetermined limit the processor selectively connects one or more different stream sources with different stream configurations to the device until such time that the stream performance factor is within the predetermined limit.
2. The system of claim I further comprising: a database comprising predetermined device performance factors for a plurality of mobile devices, wherein a device performance factor is selected based on the device the system streams to; and wherein the choice of first stream source is based on the device performance factor selected.
3. The system of claim 1 or 2 wherein the device performance factor is based onthe specification of the device.
4. The system of any preceding claim wherein the device performance factor is determined by a blit test.
5. The system of any preceding claim wherein the stream performance factor is based on a measure of the amount of data received by the device a buffer.
6. The system of any preceding claim wherein the stream performance factor is based on a TCP burst test. I c
7. The system of any of claims 3 to 6 wherein the streaming server further comprises a plurality of device groups wherein the device group selected is based on device performance factor.
8. The system of any preceding claim further comprising an authentication server.
9. The system of any preceding claim further comprising a content server and preferably a content encoding server.
10. The system of any preceding claim wherein when the stream performance factor is below the predetermined limit, the device is queried to determine the received bitrate from the connected stream source and the stream source selected is the one with a similar or equal bitrate.
11. The system of claim 6 to 10 when dependent on claim 6 wherein the TCP burst test determines the maximum bandwidth as a measure of the packet read interval using the TCP timestamps.
12 The system of any of claims 6 to 11 when dependent on claim 6 wherein each data packet in the burst is sent as an individual TCP packet.
13. The system of any preceding claim wherein the switch between a first and a second port occurs on a key frame of data.15. The system of any preceding claim wherein the switch between a first and a second stream source occurs when a change in the data connection type is detected.16. The system of any preceding claim wherein the stream configuration changes with the stream bitrate.17. The system of any preceding claim wherein the stream configuration changes with one or more of video component bitrate, audio component bitrate, video resolution, frame rate, and key frame interval.