WO2014110670A1

WO2014110670A1 - Media server

Info

Publication number: WO2014110670A1
Application number: PCT/CA2014/000038
Authority: WO
Inventors: Varadhan Venkataseshan; Seyed M. Sharif-Ahmadi
Original assignee: Mimik Technology Inc.
Priority date: 2013-01-21
Filing date: 2014-01-21
Publication date: 2014-07-24
Also published as: CA2804741A1

Abstract

A RTMP Server initiated bit rate switching in RTMP live media streaming is provided, directed to fully RTMP server originated Adaptive bit rate switching of live media stream containing H.264 video and AAC audio codecs served to RTMP based client players.

Description

MEDIA SERVER

TECHNICAL FIELD

[0001] The present disclosure is directed at a system and method for managing streaming media content delivery, and more particularly for managing the bit rate of such media content delivery between a server and a client.

BACKGROUND

[0002] The Internet allows the transmission and play of media content, and typically a media server is responsible for obtaining the media content from a content source, and then transmitting the content to a client. Media servers may use Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), Real Time Streaming Protocol (RTSP), or more commonly, Real Time Messaging Protocol (RTMP) or User Datagram Protocol (UDP), to deliver streaming content.

[0003] Media content is transmitted to mobile playback devices such as smart phones, tablets, laptop computers, and the like. Transmission of media content to such mobile devices offers additional challenges, as the bandwidth of the communication channel is typically reduced, and the processing power of the device itself is typically less than that of an ordinary computer or a special purpose device.

[0004] In live media content streaming, the throughput of the communications channel used to transmit the media content is highly variable and difficult to predict. Accordingly, media servers typically store several different versions of each available media content, with each different version optimized for different communications channel throughput, and the appropriate version is selected for transmission based upon the particular communications channel throughput.

[0005] However, while the bit rate of most streamed media content is temporally constant, this is not the case where variable bit rate encoding is employed. In the case of variable bit rate encoding, the media content is encoded using a bit rate that varies with the program material, and hence, varies with time as well. These temporal variations of the bit rate can give rise to situations wherein the bandwidth of the communications channel is sufficient, or more than is required, for portions of the media content stream, but insufficient for other portions. Since decisions are made with respect to current throughput and the average bit rate of the media content stream, such variations cause two problems: (1) the buffer becoming empty during the bitrate peak, resulting in choppy playback, and (2) the media program player may not switch up to a higher bitrate during a valley, which results in lower quality video displayed to the user who should be able to view a higher quality stream.

[0006] RTMP is a system for delivering on-demand and live media content to Adobe Flash applications or to any RTMP complaint players. RTMP uses TCP as the underlying protocol and maintains persistent connections between the server and a client. To deliver streams smoothly and transmit as much information as possible, RTMP divides streams into "chunks" with a size negotiated dynamically between the client and media server. Fragments from different streams may be interleaved and multiplexed over a single connection.

[0007] RTMP defines several virtual channels on which packets may be sent and received, and which operate independently of each other. For example, there is a channel for handling Remote Procedure Call (RPC) requests and responses, a channel for a video stream, a channel for an audio stream and a channel for out-of-band control messages. During a typical RTMP session, several channels may be active simultaneously at any given time. When RTMP data is encoded, a packet header is generated. The packet header includes the identification (ID) of the channel on which the packet is to be sent, a timestamp of when the packet was generated (if necessary), and the size of the packet's payload. This header is then followed by the actual payload content of the packet, which is fragmented according to the currently agreed-upon fragment size before it is sent over the connection. The packet header itself is never fragmented, and its size does not count towards the data in the packet's first fragment. In other words, only the actual packet payload (the media content data) is subject to fragmentation.

[0008] At a higher level, the RTMP encapsulates Advanced Audio Coding (AAC) audio and H.264 video multimedia streams with appropriate timing information. The RTMP includes RPCs made using the Action Message Format (AMF).

[0009] Adaptive multi bit rate media content streaming is known in the art and is the process of delivering streaming video to users by dynamically switching among different streams of varying quality and size without disrupting the continuous playback. Adaptive streaming is very important in delivering the optimum quality of service to a diverse range of viewers and devices.

[00010] In a typical session for RTMP based dynamic adaptive multi bit rate streaming for an RTMP based server and client, as shown in Figure 1 , a streaming media player 210 is installed on a client device 240, typically as a web browser plugin. The RTMP client 200 requests streaming media content from a media server 1 10, which may be part of a gateway 150. The media server 1 10 provides the streaming media player 210 operating on the client 240, an initial request for the appropriate configuration file, such as Synchronized Multimedia Integration Language (SMIL), or for information including the RTMP connection URL and the available video bit-rates for the client device.

[0001 1] An example of a small section from a SMIL sample showing the RTMP:URL and available bit rates follows:

<smil>

<head>

</head>

<body>

</switch>

</body>

</smil>

[00012] As shown in Figure 2, the RTMP client 200 first directly establishes a TCP session with the RTMP server 120. After establishing the TCP connection, the RTMP client 200 initiates the RTMP Handshake process. Following a successful RTMP specific handshake process, the client and server exchanges a series of complex commands and RPCs, which are mostly AMF encoded. AMF is a compact binary format that is used to serialize ActionScript object graphs.

[00013] The RTMP client 200 then sends a RTMP Netconnection::connect() Command Message to the server requesting connection to a server application instance (step A). The RTMP server 120 sends the _result command message (step B) informing the RTMP client 200 of the connection status (success/fail) (step C).

[00014] On a successful RTMP connect response, the RTMP client 200 then sends a Netconnection::createStream command (step D) to the RTMP server 120 to create a logical channel for message communication. Audio, video, and metadata flows over this stream channel created using the createStream command.

[00015] Then, on a successful createStream response from the RTMP server 120 (step E), the RTMP client 200 sends the NetStream: :play command (step F). The NetStream defines the channel through which the streaming audio, video, and data messages can flow over the NetConnection that connects the RTMP client 200 to the RTMP server 120. The "play" command object contains a Stream Name field, which has the live channel or the media that has to be streamed by the RTMP server 120. On receiving the play command, the RTMP server 120 sends a protocol message to set the chunk size (step G). [00016] Then RTMP server 120 sends a protocol message (user control) specifying the event 'StreamBegin', to indicate beginning of the streaming to the RTMP client 200 (step H). Then RTMP server 120 sends OnStatus command messages NetStream.Play. Start if the play command sent by the RTMP client 200 is successful (step I) and also sends Notify: onMetadata (step J) RPC to the Client 200. Then RTMP server 120 sends the relevant audio and video config (step K) and then starts streaming the media content by continuously sending video and audio messages in chunks to RTMP client 200 (steps L and M).

[00017] Streaming Media Player 210 and RTMP client 200 use heuristics based on factors such as device capabilities, estimated buffer occupancy and the estimated network bandwidth available, when deciding to switch to a different bit rate stream. RTMP client also chooses the appropriate bandwidth channel as provided in the initial configuration.

[00018] RTMP client 200 then sends NetStream::play2 command requests (step N) to RTMP server 120. Unlike the play command, play2 can switch to a different bit rate stream without changing the timeline of the content played. The command structure of play2 from RTMP client 200 contains "streamName" that has the name of new bit rate stream.

[00019] The RTMP server 120 receives the RTMP client 200 originated play2 bit rate change request. Based on the client rate change request, the RTMP server 120 handles the play2 request (steps O and P) and then appropriately switches to the new bit rate stream for the video messages (the audio bit rate remains fixed) at the optimal point with minimal playback impact (steps Q and R). Then the RTMP server 120 continues to send the audio and video in chunks to RTMP client 200.

[00020] What is needed is a method and apparatus allowing selection, by a RTMP server, of media content bit-streams according to time-varying bit rate requirements. SUMMARY OF THE INVENTION

[00021] A method of streaming multi bit rate media content to a RTMP client is provided. In the system according to invention, a RTMP server implements a fully Adaptive Dynamic Multi Bitrate by not using the client initiated RTMP "NetStream::play2" invocation bit rate switching request.

[00022] In the system according to the invention, the RTMP streaming server completely initiates the bitrate switching of live streaming media, unlike the usual method wherein the RTMP client player originates a bit rate switch request for RTMP.

[00023] In the system according to the invention, the RTMP client is not provided with a multibit rate option to choose a bit rate and the RTMP client has no influence on the bandwidth requested.

[00024] As an example, suppose the RTMP client is provided with one simple connection URL such as RTMP://mediaserver_ip/live/ch[Channelid]. The client or RTMP player is completely unaware of the available video quality or bit rates of the media content before the request for the content is made.

[00025] The use of fully server initiated adaptive multi bit rate switching for serving RTMP clients has some benefits for the RTMP media server. The advantages of this approach for the server include:

[00026] 1. Usually the configuration files involving adaptive multi bit rate support are very elaborate or complicated, and vary for different media player's plug-ins. By using a server initiated, multi bit rate switching technique, media servers only need to provide a simple RTMP stream access URL to the client.

[00027] 2. The system works with any RTMP players or flash based players (clients). [00028] 3. The system provides improved Quality of Service (QoS) to the clients and a fair share of network bandwidth utilization by servers distributed in a carrier network.

[00029] 4. The RTMP streaming server need not have to rely on the client to choose the right bit rate stream and instead can decide the appropriate bit rate bandwidth based on its own heuristics.

BRIEF DESCRIPTION OF THE FIGURES

[00030] Figure 1 is a block diagram showing an environment in which the RTMP media server and RTMP media player operate.

[00031] Figure 2 is a state diagram showing the prior art communication process between a prior art RTMP server and a RTMP client.

[00032] Figure 3 is a state diagram showing the communication between a RTMP server and RTMP client according to the invention.

[00033] Figure 4 is a block diagram of a media server and RTMP server according to the invention.

DETAILED DESCRIPTION

[00034] The following details the work flow of the system according to the invention to allow media server 110 initiated adaptive dynamic multi bitrate switching of a live media content stream that is transparent to the RTMP client 200.

[00035] As shown in Figures 1 and 3, streaming media player 210, installed usually as a web browser plug in with a RTMP Client 200, requests streaming media content from Media Server 1 10. The RTMP Client 200 refers to an Adobe Flash Player or any other equivalent player that supports playback of audio and video streamed from RTMP servers. Media Server 1 10 provides the Streaming Media Player 210 a simple RTMP media request URL, such as RTMP://medisaserver.com/live/chl . Media server 110 may be within an Internet based server, or may be within a gateway device 150 connecting a Client 200, such as user device, to a local network, which in tum is connected to the Internet.

[00036] The steps taken in the communications between RTMP server 120 and RTMP client 200 being with RTMP client 200 first directly establishing a successful TCP session with RTMP Server 120 and initiating the RTMP Handshake process. Upon a successful handshake process, RTMP client 200 and RTMP server 120 exchanges a series of complex commands and RPCs, which may be AMF encoded.

[00037] RTMP client 200 then sends the RTMP Netconnection::connect() Command Message (step 1) to RTMP server 120 requesting connection to a server application instance. RTMP server sends the _result command message (step 2) informing the RTMP client 200 of the connection status (success/fail) (step 3).

[00038] On a successful RTMP connect response, the RTMP client 200 then sends Netconnection::createStream command (step 4) to the RTMP server 120 to create a logical channel for message communication. Audio, video, and metadata flows over this stream channel created using the createStream command. On a successful createStream response (step 5) from RTMP server 120, the RTMP client 200 sends the NetStream::play command (step 6). The NetStream defines the channel through which the streaming audio, video, and data messages can flow over the NetConnection that connects the RTMP client 200 to the RTMP server 120.

[00039] The "play" command object contains a Stream Name field, that has the live channel or the media that has to be streamed by the RTMP server 120, for example the Stream Name may contain an indicator such as "chl ". As the RTMP client 200 is not provided with available media bit rate options, RTMP client 200 issues the request for the channel provided, for example, "chl". On receiving the play command, RTMP server 120 sends a protocol message to set the chunk size (step 7). Then RTMP server 120 sends protocol message (user control) specifying the event 'StreamBegin' (step 8), to indicate beginning of the streaming to the RTMP client 200. Then RTMP server 120 sends OnStatus command messages NetStream.Play.Start (step 9) if the play command sent by the RTMP client 200 is successful.

[00040] RTMP server 120, based on its own Adaptive channel Bandwidth detection heuristics, along with other factors and logic, requests the transcoder for the media content stream at the determined QoS. Audio and video metadata obtained on reading the MPEGTS stream is sent to RTMP client 200 as AMF encoded RTMP data with a notify "onMetadata" object (step 10). When the media becomes available, then RTMP server 120 sends the relevant AAC audio and AVC H.264 video configuration details (step 1 1) and the AAC audio and Advanced Video Coding (AVC) H.264 video data to the RTMP client 200 (steps 12 and 13). The RTMP server 120 typically supports H.264/MPEG-4 Part 10 or AVC and AAC standards.

[00041] The RTMP server 120 then continuously monitors the network condition and when its unbiased estimated network bandwidth calculation warrants a change in the video bit rate for the channel, RTMP server 120 triggers the media transcoder to change the video bit rate beginning with an appropriate Group of Pictures ("GOP") of the video. Once the transcoded new bit rate stream is ready, RTMP server 120 sends the relevant audio and video config messages (steps 14 and 15) and then appropriately switches to the new bit rate stream for the video messages (the audio bit rate may remain fixed or may follow the new video bit stream rate) at the optimal point with minimal playback impact (steps 16 and 17). Then the RTMP server 120 continues to send the audio and video in chunks to RTMP client 200. Of note, step N of Figure 2, the Command (play2) message, is no longer needed, as the RTMP server 120 selects the video bit rate.

[00042] Figure 4 is a block diagram of an embodiment of a dynamic adaptive multi bitrate RTMP server 120 according to the invention. System TC/IP stack 103 is an integral part of the server's underlying operating system's TCP/IP network stack software that implements Transmission Control Protocol (TCP) and the Internet Protocol (IP) Internet protocol suite. [00043] System Resource Monitor 102 is a software program (that may be incorporated in whole or part in hardware) that is a component of Media Server 1 10, and which monitors the system CPU and memory resources and advises on system available resources to carry out the requested advanced H.264 video compression. Tunable Media Transcoder 101 is a software program (or that may be incorporated in whole or part in hardware) that acts as a highly tunable stream media transcoder to achieve the desired speed, compression ratio, stream quality as well as the desired bit rate stream.

[00044] Channel TCP KPI and Network Monitor 104 is a software program (that may be incorporated in whole or part in hardware) that interacts with System TC/IP stack 103 periodically or upon request or at other appropriate times, to get various channel related details such as TCP metadata, and network Key Performance Indicators (KPI), related to a TCP session between RTMP Server 120 and the RTMP Client 200.

[00045] Stream Bandwidth Estimator and Dynamic Tuner 107 is a software program (that may be in whole or part incorporated in hardware) that carries out various heuristics to predict the desired Bandwidth for switching the media content stream and also determines the optimal H.264 transcoding tuning parameters suitable for next stream switching.

[00046] Stream Channel Manager 106 is a software program (that may be in whole or part incorporated in hardware) that manages the RTMP stream channel. RTMP Protocol handler 108 handles all the RTMP protocol related activities. Channel Aware Stream Transport Scheduler 105 is a dynamic stream transport scheduler that sends media packets at channel bandwidth through TCP stack 103 to the RTMP client 200.

[00047] Optimally switching the bit rate of compressed video in synchronization with the dynamic network capacity is an important and non-trivial process in delivering an adaptive bit rate stream completely transparent to RTMP client 200. Adaptive Dynamic Streaming is used to maximize Quality of Service by switching from one bitrate to another depending on the changing network channel conditions. This is achieved by continuously monitoring CPU and memory resource availability, dynamic network conditions, RTMP client 200 capabilities, and then using this information as feedback to tune the transcoder to get desired stream quality, compression efficiency and bandwidth.

[00048] Since RTMP uses TCP as the underlying transport protocol, exploiting the information provided by the TCP stack 103 on the connection is useful for predicting the optimal stream bandwidth time. TCP, which includes flow control, can already provide implicit feedback to the RTMP server 120 and thus no modification to the RTMP client 200 is necessary.

[00049] An example of logic flow for adaptive bandwidth switching follows. Deciding on the initial stream bandwidth among the range of supported bandwidth by the transcoder 101 begins with a heuristic. Although the amount of data transferred back and forth to the RTMP client 200 before the play command of the media content is not significant, initial RTMP protocol message exchanges prior to actual play command from RTMP client 200, such as the RTMP handshake, Netconnection::connect, CreateStream, and others, are useful as a starting point of bandwidth estimation using the low pass filtered Round Trip Time (rtt) obtained by probing TCP stack 103 for the channel. For example, in a LAN these initial rtts may be less than 5 milliseconds and in a heavily loaded Wi-Fi, may be in the range of few 10s of milliseconds, and in a 3G network, may be in the order of 100s of milliseconds. Based on the rtt and filtered smoothed rtt (srtt) obtained, initial bandwidth for starting the stream may be determined, for example, by using a memory-less, continuous-time Markov process model. This assists in determining a stream bandwidth suitable for the current network condition.

[00050] Using the initial bandwidth as determined, the starting media stream bandwidth is requested from transcoder 101. The transcoded stream is provided by transcoder 101 as a MPEGTS stream with AAC audio and AVC (H.264) video. This MPEGTS media stream in opened and parsed by RTMP server 120. First audio and video metadata are obtained from reading the MPEGTS stream. This information is sent to RTMP client 200 as an AMF encoded RTMP data notify "onMetadata" object. Then the RTMP/FLASH encoded AAC Audio Config and AVC Video Config is sent to the RTMP client 200. Next, the the actual time synchronized audio and video data are sent to the RTMP client 200.

[00051] After the initial phase of sending some media data to the RTMP client 200, TCP stack 130 is continuously probed, using the connected socket descriptor of the established client server TCP session, to get in-depth values of TCP congestion control specific information and TCP state variables. These variables may include: rtt; srtt; slow start threshold (ssthresh); congestion window (cwnd); receiver window (rwnd); maximum segment size (mss); amount of retransmission; TCP send buffer size; amount of unsent data in the socket send queue; packets that are in flight; lost packets; flight size (the amount of data has been sent but has not been acknowledged); and others.

[00052] A number of heuristics, using the channel connection state variables obtained from TCP along with application of standard deviation and exponential smoothing techniques to time series data, are employed to obtain the optimal bandwidth estimation for switching the bit rate of next GOP of images (e.g. all frames starting from an I frame until the next I frame).

[00053] In an embodiment of a method to determine bandwidth estimation, a conservative approach may be taken by using the data acknowledged bandwidth instead of the transmit bandwidth, for example:

Let T_x[z] be the total amount of data sent by RTMP server 120 to RTMP client 200 at a given instance i. Let T_a[z] be the actual amount of data acknowledged to TCP stack 103 by RTMP client 200 at a given instance . Let At be the time elapsed to get T_a[z], then the acknowledgement based bandwidth estimate (AckBWe) is calculated as follows:

AckBWe[ ] = Tat j/Δί

[00054] In an alternative embodiment, the bandwidth estimate at a given instance / may be computed as a function of the TCP congestion window (cwnd), maximum segment size (mss) and smoothed round trip time (srtt) as follows: BWe[z] = (cwnd[/] * mss[ ])/srtt[ ]

[00055] The following time-varying discrete time filter can be used to smooth the oscillation in the bandwidth estimate: fBWe[ ] = (1 - exp(-5t/T₀)) * BWe[ ] + exp(-5t/T₀) * fBWe[ ] wherein fBWe[ ] is the infinite impulse response filtered bandwidth estimate at a given instance z; fBWe[z-l] is the infinite impulse response filtered bandwidth estimate at a given instance prior to i; BWe[i] is the instantaneous bandwidth estimate obtained as described above; T₀ is a time constant, for example, 1 (one) second; δΐ is the time interval between the last two estimates; and the exp(x) function returns the value of e (the base of natural logarithms) raised to the power of x.

[00056] Such impulse response filtering reduces the frequency of switching up and down bandwidth levels due to network bandwidth oscillations, for example from 768 Kbps to 1024 Kbps and then quickly returning back to 768 Kbps from 1024Kbps. In such cases it may be prudent to stay at 768 Kbps to avoid the switching artifacts from going back and forth between 768 kbps and 1024 kbps. Also TCP stack socket 103 sends the buffer occupancy, which when estimated properly, reflects the RTMP client 200 buffer occupancy. Such embodiment of bandwidth estimation closely following the underlying TCP transport protocol, leads to an optimal and intelligent dynamic adaptive rate switching to provide a good QoS and smooth streaming to the RTMP client 200, completely transparent to client 200.

[00057] If there is a change in streaming bandwidth, the optimal video transcoding options are determined and fed to the tunable transcoder 101 for obtaining the desired streaming bit rate. When the new rate stream becomes available, then at the beginning of the video key frame, time synched properly with audio will be sent to the RTMP client 200. This process of dynamically adapting to the changing network condition continues until RTMP client 200 terminates the connection or when the media content is finished streaming. [00058] Server 120 initiated bandwidth adaption of the stream is thus achieved for live RTMP stream, without RTMP client 200 actually asking for change of bit rate, by switching to the new bandwidth stream at the next nearest matching H.264 keyframe video and also appropriately synchronizing audio and video.

[00059] During the initial connection to the Media Server 110, the RTMP client 200 is not provided with any option to select the bit rate and is just given the live channel name along with the RTMP connection URL that is used to request the media.

[00060] The adaptive bit rate switching is made possible directly by the RTMP server 120 by changing the transmitted media bit rate when appropriate. The RTMP client 200, when using the system according to the invention, need not issue a play2 switch command to RTMP server 120.

[00061] For the sake of convenience, the embodiments above are described as various interconnected functional blocks or distinct software modules. This is not necessary, however, and there may be cases where these functional blocks or modules are equivalently aggregated into a single logic device, program or operation with unclear boundaries. In any event, the functional blocks and software modules or features of the flexible interface can be implemented by themselves, or in combination with other operations in either hardware or software.

[00062] While particular embodiments have been described in the foregoing, it is to be understood that other embodiments are possible and are intended to be included herein. It will be clear to any person skilled in the art that modifications of and adjustments to the foregoing embodiments, not shown, are possible.

Claims

CLAIMS WHAT IS CLAIMED:

1. A media server, comprising a. a RTMP server configured to communicate streaming media content to a RTMP client; b. a stream channel monitor configured to evaluate the quality of the data stream between the RTMP server and the RTMP client; c. a bandwidth estimator for providing an estimate of the available bandwidth available for streaming the media content; d. a tunable transcoder for streaming the media content to the RTMP server at the estimated available bandwidth.

2. A method of providing dynamic adaptive multi bit rate live streaming for a RTMP client from an RTMP server, comprising: a. said RTMP server determining a first video bit rate for said live streaming in consideration of network conditions; b. said RTMP server sending said video stream to said RTMP client; c. said RTMP server determining a second bit rate in consideration of changes in network conditions; d. said RTMP server sending said video stream to said RTMP client at said second bit rate.

3. The method of claim 2 wherein said RTMP client is not provided with a choice of bit rates from said RTMP server.

4. The method of one of claims 2 or 3 wherein an initial bit rate is determined using RTMP protocol message exchanges.

5. The method of one of claims 2 through 4 wherein impulse response filtering is used to determine the second bit rate.

6. The method of one of claims 2 through 5 wherein the second bit rate is an estimated bit rate.

7. The method of one of claims 2 through 6 wherein the second bit rate determination uses information from a TCP stack.

8. The method of one of claims 2 through 7 wherein the RTMP server triggers a transcoder to change to the second bit rate.