CA2804741A1 - Media server - Google Patents
Media server Download PDFInfo
- Publication number
- CA2804741A1 CA2804741A1 CA 2804741 CA2804741A CA2804741A1 CA 2804741 A1 CA2804741 A1 CA 2804741A1 CA 2804741 CA2804741 CA 2804741 CA 2804741 A CA2804741 A CA 2804741A CA 2804741 A1 CA2804741 A1 CA 2804741A1
- Authority
- CA
- Canada
- Prior art keywords
- rtmp
- server
- client
- bit rate
- media
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/30—Definitions, standards or architectural aspects of layered protocol stacks
- H04L69/32—Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
- H04L69/322—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions
- H04L69/329—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions in the application layer [OSI layer 7]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/65—Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/762—Media network packet handling at the source
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/80—Responding to QoS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0882—Utilisation of link capacity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0894—Packet rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/163—In-band adaptation of TCP data exchange; In-band control procedures
Abstract
A RTMP Server initiated bit rate switching in RTMP live media streaming is provided, directed to fully RTMP server originated Adaptive bit rate switching of live media stream containing H.264 video and AAC audio codecs served to RTMP based client players..
Description
MEDIA SERVER
TECHNICAL FIELD
[0001] The present disclosure is directed at a system and method for managing streaming media content delivery, and more particularly for managing the bit rate of such media content delivery between a server and a client.
BACKGROUND
TECHNICAL FIELD
[0001] The present disclosure is directed at a system and method for managing streaming media content delivery, and more particularly for managing the bit rate of such media content delivery between a server and a client.
BACKGROUND
[0002] The Internet allows the transmission and play of media content, and typically, a media server is responsible for obtaining the media content from a content source, and then transmitting the content to a client. Media servers may use HTTP, TCP, RTSP, or more commonly, RTMP or UDP, to deliver streaming content.
[0003] Media content is transmitted to mobile playback devices such as smart phones, tablets, laptop computers, and the like. Transmission of media content to such mobile devices offers additional challenges, as the bandwidth of the communication channel is typically reduced, and the processing power of the device itself is typically less than that of an ordinary computer or special purpose device.
[0004] In live streaming, the throughput of the communications channel used to transmit media content is highly variable and difficult to predict. Accordingly, media servers typically store several different versions of each media content, with each different version optimized for different communications channel throughput, and the appropriate version is selected for transmission based upon the communications channel throughput.
[0005] However, while the bit rate of most streamed media content is temporally constant, this is not the case where variable bit rate encoding is employed. With variable bit rate encoding, the media content is encoded using a bit rate that varies with the program material, and hence, varies with time as well. These temporal variations of the bit rate give rise to situations wherein the bandwidth of the communications channel is sufficient or more than is required for portions of the media program, but insufficient for other portions. Since decisions are made with respect to current throughput and the average bit rate of the media content stream, such variations cause {E6323648.DOC;1}
two problems: (1) the buffer becoming empty during the bitrate peak, resulting in choppy playback, and (2) the media program player may not switch up to a higher bitrate during a valley, which results in lower quality video displayed to the user who is capable of viewing a higher quality stream.
two problems: (1) the buffer becoming empty during the bitrate peak, resulting in choppy playback, and (2) the media program player may not switch up to a higher bitrate during a valley, which results in lower quality video displayed to the user who is capable of viewing a higher quality stream.
[0006] RTMP (Real Time Messaging Protocol) is a system for delivering on-demand and live media content to Adobe Flash applications or to any RTMP complaint players.
RTMP uses TCP
as the underlying protocol and maintains persistent connections between the server and a client.
To deliver streams smoothly and transmit as much information as possible, the protocol divides streams into "chunks" with a size negotiated dynamically between the client and server.
Fragments from different streams may be interleaved and multiplexed over a single connection.
RTMP uses TCP
as the underlying protocol and maintains persistent connections between the server and a client.
To deliver streams smoothly and transmit as much information as possible, the protocol divides streams into "chunks" with a size negotiated dynamically between the client and server.
Fragments from different streams may be interleaved and multiplexed over a single connection.
[0007] RTMP defines several virtual channels on which packets may be sent and received, and which operate independently of each other. For example, there is a channel for handling RPC
requests and responses, a channel for a video stream, a channel for an audio stream and a channel for out-of-band control messages. During a typical RTMP session, several channels may be active simultaneously at any given time. When RTMP data is encoded, a packet header is generated. The packet header includes the ID of the channel on which the packet is to be sent, a timestamp of when the packet was generated (if necessary), and the size of the packet's payload.
This header is then followed by the actual payload content of the packet, which is fragmented according to the currently agreed-upon fragment size before it is sent over the connection. The packet header itself is never fragmented, and its size does not count towards the data in the packet's first fragment. In other words, only the actual packet payload (the media content data) is subject to fragmentation.
requests and responses, a channel for a video stream, a channel for an audio stream and a channel for out-of-band control messages. During a typical RTMP session, several channels may be active simultaneously at any given time. When RTMP data is encoded, a packet header is generated. The packet header includes the ID of the channel on which the packet is to be sent, a timestamp of when the packet was generated (if necessary), and the size of the packet's payload.
This header is then followed by the actual payload content of the packet, which is fragmented according to the currently agreed-upon fragment size before it is sent over the connection. The packet header itself is never fragmented, and its size does not count towards the data in the packet's first fragment. In other words, only the actual packet payload (the media content data) is subject to fragmentation.
[0008] At a higher level, the RTMP encapsulates AAC audio and H.264 video multimedia streams with appropriate timing information. The RTMP includes remote procedure calls (RPCs) made using the Action Message Format.
[0009] Adaptive multi bit rate media streaming is known in the art and is the process of delivering streaming video to users by dynamically switching among different streams of varying quality and size without disrupting the continuous playback. Adaptive streaming may be the {E6323648.DOC; I }
single most important capability for delivering the optimum quality of service to a diverse range of viewers and devices.
single most important capability for delivering the optimum quality of service to a diverse range of viewers and devices.
[00010] In a typical prior art embodiment for RTMP based dynamic adaptive multi bit rate streaming for an RTMP based server and client, as shown in Figure 1, a streaming media player 210 is installed on a client device 240, typically as a web browser plugin.
The RTMP client 200 requests streaming media from a media server 110, which may be part of a gateway 150. The media server 110 provides the streaming media player 210 operating on the client 240, an initial request for the appropriate configuration file, such as SMIL (Synchronized Multimedia Integration Language) or for information including the RTMP connection URL and the available video bit-rates for the client.
The RTMP client 200 requests streaming media from a media server 110, which may be part of a gateway 150. The media server 110 provides the streaming media player 210 operating on the client 240, an initial request for the appropriate configuration file, such as SMIL (Synchronized Multimedia Integration Language) or for information including the RTMP connection URL and the available video bit-rates for the client.
[00011] An example of a small section from a SMIL sample showing the RTMP:URL and available bit rates follows:
<smil>
<head>
<meta base="RTMP://medisaserver.com/live" />
</head>
<body>
<switch>
<video src="chl -256" system-bitrate="1208" />
<video src="ch1-256" system-bitrate="256" />
<video src="ch1-5120" system-bitrate="5120" />
<video src="ch1-1024" system-bitrate="1024"/>
</switch>
</body>
</smil>
<smil>
<head>
<meta base="RTMP://medisaserver.com/live" />
</head>
<body>
<switch>
<video src="chl -256" system-bitrate="1208" />
<video src="ch1-256" system-bitrate="256" />
<video src="ch1-5120" system-bitrate="5120" />
<video src="ch1-1024" system-bitrate="1024"/>
</switch>
</body>
</smil>
[00012] As shown in Figure 2, the RTMP client 200 first directly establishes a TCP
session with the RTMP server 120. After establishing the TCP connection, the RTMP client 200 initiates the RTMP Handshake process. Following a successful RTMP specific handshake process, the client and server exchanges a series of complex commands and Remote Procedure (E6323648.DOC;1) Calls, which are mostly Action Message Format (AMF) encoded. AMF is a compact binary format that is used to serialize ActionScript object graphs.
session with the RTMP server 120. After establishing the TCP connection, the RTMP client 200 initiates the RTMP Handshake process. Following a successful RTMP specific handshake process, the client and server exchanges a series of complex commands and Remote Procedure (E6323648.DOC;1) Calls, which are mostly Action Message Format (AMF) encoded. AMF is a compact binary format that is used to serialize ActionScript object graphs.
[00013] The RTMP client 200 then sends a RTMP Netconnection::connect() Command Message to the server requesting connection to a server application instance (step 1). The RTMP
server 120 sends the _result command message (step 2) informing the RTMP
client 200 of the connection status (success/fail) (step 3).
server 120 sends the _result command message (step 2) informing the RTMP
client 200 of the connection status (success/fail) (step 3).
[00014] On a successful RTMP connect response, the RTMP client 200 then sends a Netconnection::createStream command (step 4) to the RTMP server 120 to create a logical channel for message communication. Audio, video, and metadata flows over this steam channel created using the createStream command.
[00015] Then, on a successful createStream response from the RTMP server 120 (step 5), the RTMP client 200 sends the NetStream::play command (step 6). The NetStream defines the channel through which the streaming audio, video, and data messages can flow over the NetConnection that connects the RTMP client 200 to the RTMP server 120. The "play"
command object contains a Stream Name field, which has the live channel or the media that has to be streamed by the RTMP server 120. On receiving the play command, the RTMP
server 120 sends a protocol message to set the chunk size (step 7).
command object contains a Stream Name field, which has the live channel or the media that has to be streamed by the RTMP server 120. On receiving the play command, the RTMP
server 120 sends a protocol message to set the chunk size (step 7).
[00016] Then RTMP server 120 sends a protocol message (user control) specifying the event 'StreamBegin', to indicate beginning of the streaming to the RTMP client 200 (step 8).
Then RTMP server 120 sends OnStatus command messages NetStream.Play.Start if the play command sent by the RTMP client 200 is successful (step 9). Then RTMP server 120 sends the relevant audio and video config (step 10) and then starts streaming the media content by continuously sending video and audio messages in chunks to RTMP client 200 (steps 12 and 13).
Then RTMP server 120 sends OnStatus command messages NetStream.Play.Start if the play command sent by the RTMP client 200 is successful (step 9). Then RTMP server 120 sends the relevant audio and video config (step 10) and then starts streaming the media content by continuously sending video and audio messages in chunks to RTMP client 200 (steps 12 and 13).
[00017] Streaming Media Player 210 and RTMP client 200 use heuristics based on factors such as device capabilities, estimated buffer occupancy and the estimated network bandwidth available, when deciding to switch to a different bit rate stream. RTMP client also chooses the appropriate bandwidth channel as provided in the initial configuration.
{E6323648.DOC; 1) [00018] RTMP client 200 then sends NetStream::play2 command requests (step 14) to RTMP server 120. Unlike the play command, play2 can switch to a different bit rate stream without changing the timeline of the content played. The command structure of play2 from RTMP client 200 contains "streamName" that has the name of new bit rate stream.
{E6323648.DOC; 1) [00018] RTMP client 200 then sends NetStream::play2 command requests (step 14) to RTMP server 120. Unlike the play command, play2 can switch to a different bit rate stream without changing the timeline of the content played. The command structure of play2 from RTMP client 200 contains "streamName" that has the name of new bit rate stream.
[00019] The RTMP server 120 receives the RTMP client 200 originated play2 bit rate change request. Based on the client rate change request, the RTMP server 120 handles the play2 request (steps 15 and 16) and then appropriately switches to the new bit rate stream for the video messages (the audio bit rate remains fixed) at the optimal point with minimal playback impact (steps 17 and 18). Then the RTMP server 120 continues to send the audio and video in chunks to RTMP client 200.
[00020] What is needed is a method and apparatus allowing selection, by a RTMP server, of media content bit-streams according to time-varying bit rate requirements.
The present invention satisfies this need.
SUMMARY OF THE INVENTION
The present invention satisfies this need.
SUMMARY OF THE INVENTION
[00021] A method of streaming multi bit rate media to a RTMP client is provided. In the system according to invention, a RTMP server implements a fully Adaptive Dynamic Multi Bitrate by not using the client initiated RTMP "NetStream::play2" invocation bit rate switching request.
[00022] In the system according to the invention, the RTMP streaming server completely initiates the bitrate switching of live streaming media, unlike the usual method wherein the RTMP client player originates a bit rate switch request for RTMP.
[00023] In the system according to the invention, the RTMP client is not provided with a multibit rate option to choose a bit rate and the RTMP client has no influence on the bandwidth requested.
[00024] As an example, suppose the RTMP client is provided with one simple connection URL such as RTMP://mediaserver_ip/live/ch[Chanrielid]. The client or RTMP
player is completely unaware of the available video quality or bit rates of the media before the request.
{E6323648.DOC; 1}
player is completely unaware of the available video quality or bit rates of the media before the request.
{E6323648.DOC; 1}
[00025] The use of fully server initiated adaptive multi bit rate switching for serving RTMP clients has some benefits for the RTMP media server. The advantages of this approach for the server include:
[00026] 1. Usually the configuration files involving adaptive multi bit rate support are very elaborate or complicated, and vary for different media player's plug-ins.
By using a server imitated, multi bit rate switching technique, media servers only need to provide a simple RTMP
stream access URL to the client.
By using a server imitated, multi bit rate switching technique, media servers only need to provide a simple RTMP
stream access URL to the client.
[00027] 2. The system works with any RTMP players or flash based players (clients).
[00028] 3. The system provides improved Quality of Service (QoS) to the clients and a fair share of network bandwidth utilization by servers distributed in a carrier network.
[00029] 4. The RTMP streaming server need not have to rely on the client to choose the right bit rate stream and instead can decide the band width based on its own heuristics.
BRIEF DESCRIPTION OF THE FIGURES
BRIEF DESCRIPTION OF THE FIGURES
[00030] Figure 1 is a block diagram showing an environment in which the RTMP media server and RTMP media player operate.
[00031] Figure 2 is a state diagram showing the prior art communication process between a prior art RTMP server and a RTMP client.
[00032] Figure 3 is a state diagram showing the communication between a RTMP server and RTMP client according to the invention.
[00033] Figure 4 is a block diagram of a media server and RTMP server according to the invention.
DETAILED DESCRIPTION
DETAILED DESCRIPTION
[00034] The following details the work flow of the system according to the invention to allow media server 110 initiated adaptive dynamic multi bitrate switching of a live stream that is transparent to the RTMP client 200.
(E6323648.DOC; I) [00035] As shown in Figures 1 and 3, streaming media player 210, installed mostly as a web browser plug in with a RTMP Client 200, requests media from Media Server 110. The RTMP Client 200 refers to an Adobe Flash Player or any other equivalent player that supports playback of audio and video streamed from RTMP servers. Media Server 110, just provides the Streaming Media Player 210 a simple RTMP media request URL, such as RTMP://medisaserver.com/live/ch 1 . Media server 110 may be within an Internet based server, or may be within a gateway device 150 connecting a user device to a local network, which in turn is connected to the Internet.
(E6323648.DOC; I) [00035] As shown in Figures 1 and 3, streaming media player 210, installed mostly as a web browser plug in with a RTMP Client 200, requests media from Media Server 110. The RTMP Client 200 refers to an Adobe Flash Player or any other equivalent player that supports playback of audio and video streamed from RTMP servers. Media Server 110, just provides the Streaming Media Player 210 a simple RTMP media request URL, such as RTMP://medisaserver.com/live/ch 1 . Media server 110 may be within an Internet based server, or may be within a gateway device 150 connecting a user device to a local network, which in turn is connected to the Internet.
[00036] The steps taken in the communications between RTMP server 120 and RTMP
client 200 are similar to those in the prior art until the initial play command. RTMP client 200 first directly establishes a successful TCP session with RTMP Server 120 and initiates the RTMP
Handshake process. Upon a successful handshake process, RTMP client 200 and RTMP server 120 exchanges a series of complex commands and Remote Procedure Calls, which are mostly Action Message Format(AMF) encoded.
client 200 are similar to those in the prior art until the initial play command. RTMP client 200 first directly establishes a successful TCP session with RTMP Server 120 and initiates the RTMP
Handshake process. Upon a successful handshake process, RTMP client 200 and RTMP server 120 exchanges a series of complex commands and Remote Procedure Calls, which are mostly Action Message Format(AMF) encoded.
[00037] RTMP client 200 then sends the RTMP Netconnection::connect() Command Message (step 1) to RTMP server 120 requesting connection to a server application instance.
RTMP server sends the _result command message (step 2) informing the RTMP
client 200 of the connection status (success/fail) (step 3).
RTMP server sends the _result command message (step 2) informing the RTMP
client 200 of the connection status (success/fail) (step 3).
[00038] On a successful RTMP connect response, the RTMP client 200 then sends Netconnection::createStream command (step 4) to the RTMP server 120 to create a logical channel for message communication. Audio, video, and metadata flows over this stream channel created using the createStream command. On a successful createStream response (step 5) from RTMP server 120, the RTMP client 200 sends the NetStream::play command (step 6). The NetStream defines the channel through which the streaming audio, video, and data messages can flow over the NetConnection that connects the RTMP client 200 to the RTMP
server 120.
server 120.
[00039] The "play" command object contains a Stream Name field, that has the live channel or the media that has to be streamed by the RTMP server 120, for example the Stream Name will contain something like "chi ". As the RTMP client 200 is not provided with available media bit rate options, RTMP client just issues the request for the channel provided, for {E6323648.DOC;1}
example, "chi". On receiving the play command, RTMP server 120 sends a protocol message to set the chunk size (step 7). Then RTMP server 120 sends protocol message (user control) specifying the event 'StreamBegin' (step 8), to indicate beginning of the streaming to the RTMP
client 200. Then RTMP server 120 sends OnStatus command messages NetStream.Play.Start (step 9) if the play command sent by the RTMP client 200 is successful.
example, "chi". On receiving the play command, RTMP server 120 sends a protocol message to set the chunk size (step 7). Then RTMP server 120 sends protocol message (user control) specifying the event 'StreamBegin' (step 8), to indicate beginning of the streaming to the RTMP
client 200. Then RTMP server 120 sends OnStatus command messages NetStream.Play.Start (step 9) if the play command sent by the RTMP client 200 is successful.
[00040] RTMP server 120, based on its own Adaptive channel Bandwidth detection heuristics, along with other factors and logic, requests the transcoder for the media stream at the determined QoS. . Audio and video metadata obtained on reading the MPEGTS
stream is sent to RTMP client 200 as AMF encoded RTMP data with a notify "onMetadata" object (step 10).
When the media becomes available, then RTMP server 120 sends the relevant AAC
audio and AVC H.264 video configuration details (step 11) and the AAC audio and AVC
H.264 video data to the RTMP client 200 (steps 12 and 13). The RTMP server 120 typically supports H.264/MPEG-4 Part 10 or AVC(Advanced Video Coding) and Advanced Audio Coding (AAC) standards.
stream is sent to RTMP client 200 as AMF encoded RTMP data with a notify "onMetadata" object (step 10).
When the media becomes available, then RTMP server 120 sends the relevant AAC
audio and AVC H.264 video configuration details (step 11) and the AAC audio and AVC
H.264 video data to the RTMP client 200 (steps 12 and 13). The RTMP server 120 typically supports H.264/MPEG-4 Part 10 or AVC(Advanced Video Coding) and Advanced Audio Coding (AAC) standards.
[00041] The RTMP server 120 then continuously monitors the network condition and when its unbiased estimated network bandwidth warrants a change in the video bit rate for the channel, RTMP server triggers the media transcoder to change the video bit rate beginning with an appropriate Group of Pictures ("GOP") of the video. Once the transcoded new bit rate stream is ready, RTMP server 120 sends the relevant audio and video config messages (steps 15 and 16) and then appropriately switches to the new bit rate stream for the video messages (the audio bit rate remains fixed) at the optimal point with minimal playback impact (steps 17 and 18). Then the RTMP server 120 continues to send the audio and video in chunks to RTMP
client 200. Of note, step 14, the Command (play2) message, is no longer needed, as the RTMP
server 120 selects the video bit rate.
client 200. Of note, step 14, the Command (play2) message, is no longer needed, as the RTMP
server 120 selects the video bit rate.
[00042] Figure 4 is a block diagram of a system describing the views and embodiments of a dynamic adaptive multi bitrate RTMP server 120. System TC/IP stack 103 is an integral part of the underlying operating system's TCP/IP network stack software that implements Transmission Control Protocol (TCP) and the Internet Protocol (IP) Internet protocol suite.
{E6323648.DOC; 11 [00043] System Resource Monitor 102 is a software program that is a component of Media Server 110, and which monitors the system CPU and memory resources and advises on system available resources to carry out the requested advanced H.264 video compression.
Tunable Media Transcoder 101 is a software program that is a highly tunable stream media transcoder to achieve the desired speed, compression ratio, stream quality and desired bit rate stream.
{E6323648.DOC; 11 [00043] System Resource Monitor 102 is a software program that is a component of Media Server 110, and which monitors the system CPU and memory resources and advises on system available resources to carry out the requested advanced H.264 video compression.
Tunable Media Transcoder 101 is a software program that is a highly tunable stream media transcoder to achieve the desired speed, compression ratio, stream quality and desired bit rate stream.
[00044] Channel TCP KPI and Network Monitor 104 is a software program that interacts with System TC/IP stack 103 periodically or upon request or at other appropriate times, to get channel related various details such as TCP metadata, and network Key Performance Indicator (KPI), related to a TCP session between RTMP Server 120 and the RTMP Client 200.
[00045] Stream Bandwidth Estimator and Dynamic Tuner 107 is a software program that carries out various heuristics to predict the desired Bandwidth for switching stream and also decides on the optimal H.264 transcoding tuning parameters suitable for next stream switching.
[00046] Stream Channel Manager 106 is a software program that manages the RTMP
stream channel. RTMP Protocol handler 108 handles all the RTMP protocol related activities.
Channel Aware Stream Transport Scheduler 105 is a dynamic stream transport scheduler that actually sends media packets at channel Bandwidth through TCP stack to the RTMP client 200.
stream channel. RTMP Protocol handler 108 handles all the RTMP protocol related activities.
Channel Aware Stream Transport Scheduler 105 is a dynamic stream transport scheduler that actually sends media packets at channel Bandwidth through TCP stack to the RTMP client 200.
[00047] Optimally switching the bit rate of compressed video in synchronization with the dynamic network capacity is an important and non-trivial process in delivering an adaptive bit rate stream completely transparent to RTMP client 200. Adaptive Dynamic Streaming is used to maximize Quality of Service by switching from one bitrate to another depending on the changing network channel conditions. This is achieved by continuously monitoring CPU
and memory resource availability, dynamic network conditions, RTMP client 200 capabilities, and then using this information as feedback to tune the transcoder to get desired stream quality, compression efficiency and bandwidth.
and memory resource availability, dynamic network conditions, RTMP client 200 capabilities, and then using this information as feedback to tune the transcoder to get desired stream quality, compression efficiency and bandwidth.
[00048] Since RTMP uses TCP as the underlying transport protocol, exploiting the information provided by the TCP stack 103 on the connection is useful for predicting the optimal {E6323648.DOC; 1) stream Bandwidth time. TCP, with its built-in flow control, can already provide implicit feedback to the RTMP server 120 and thus no modification to the RTMP client 200 is necessary.
[00049] A simple flow of logic involved with adaptive bandwidth switching follows:
deciding on the initial stream bandwidth among the range of supported bandwidth by the transcoder 101 starts with a simple heuristic. Although the amount of data transferred back and forth to the RTMP client 200 before play is not significant, initial RTMP
protocol message exchanges prior to actual play command from RTMP client 200, such as the RTMP
handshake, Netconnection::connect, CreateStream, etc. are useful as a starting point of bandwidth estimation using the low pass filtered Round Trip Time (rtt) obtained by probing TCP
stack 103 for the channel. For example, in a LAN these initial rtts were found to be less than 5 milliseconds and in a heavily loaded Wi-Fi, in the range of few lOs of milliseconds, and in a 3G network, in the order of 100s of milliseconds. Based on the rtt and filtered smoothed rtt (srtt) obtained thus far, initial bandwidth for starting the stream may be decided, for example, by using a memory-less, continuous-time Markov process model. This assists in determining a stream bandwidth suitable for the current network condition.
deciding on the initial stream bandwidth among the range of supported bandwidth by the transcoder 101 starts with a simple heuristic. Although the amount of data transferred back and forth to the RTMP client 200 before play is not significant, initial RTMP
protocol message exchanges prior to actual play command from RTMP client 200, such as the RTMP
handshake, Netconnection::connect, CreateStream, etc. are useful as a starting point of bandwidth estimation using the low pass filtered Round Trip Time (rtt) obtained by probing TCP
stack 103 for the channel. For example, in a LAN these initial rtts were found to be less than 5 milliseconds and in a heavily loaded Wi-Fi, in the range of few lOs of milliseconds, and in a 3G network, in the order of 100s of milliseconds. Based on the rtt and filtered smoothed rtt (srtt) obtained thus far, initial bandwidth for starting the stream may be decided, for example, by using a memory-less, continuous-time Markov process model. This assists in determining a stream bandwidth suitable for the current network condition.
[00050] Using the transcoding options, the starting media stream bandwidth is requested from transcoder 101. The transcoded stream is provided by transcoder 101 as a MPEGTS stream with AAC audio and AVC(H.264) video. This MPEGTS media stream in opened and parsed by RTMP server 120. First audio and video metadata are obtained from reading the MPEGTS
stream. This information is sent to RTMP client 200 as an AMF encoded RTMP
data notify "onMetadata" object. Then the RTMP/FLASH encoded AAC Audio Config and AVC
Video Config is sent to the RTMP client 200. Next, the the actual time synchronized audio and video data are sent to the RTMP client 200.
stream. This information is sent to RTMP client 200 as an AMF encoded RTMP
data notify "onMetadata" object. Then the RTMP/FLASH encoded AAC Audio Config and AVC
Video Config is sent to the RTMP client 200. Next, the the actual time synchronized audio and video data are sent to the RTMP client 200.
[00051] After the initial phase of sending some media to the RTMP client 200, TCP stack 130 is continuously probed, using the connected socket descriptor of the established client server TCP session, to get in-depth values of TCP congestion control specific information and TCP
state variables. These variables may include: rtt; smoothed low pass filtered rtt(srtt), slow start threshold (ssthresh); congestion window (cwnd); receiver window (rwnd);
maximum segment size (mss); amount of retransmission; TCP send buffer size; amount of unsent data in the socket (E6323648.DOC; 1) send queue; packets that are in flight; lost packets; flight size (the amount of data has been sent but has not been acknowledged); and others.
state variables. These variables may include: rtt; smoothed low pass filtered rtt(srtt), slow start threshold (ssthresh); congestion window (cwnd); receiver window (rwnd);
maximum segment size (mss); amount of retransmission; TCP send buffer size; amount of unsent data in the socket (E6323648.DOC; 1) send queue; packets that are in flight; lost packets; flight size (the amount of data has been sent but has not been acknowledged); and others.
[00052] A number of heuristics, using the channel connection state variables obtained from TCP along with application of standard deviation and exponential smoothing techniques to time series data, are employed to obtain the optimal bandwidth estimation for switching the bit rate of next GOP of pictures (i.e. all frames starting from an I frame until the next I frame).
[00053] In one embodiment of determining bandwidth estimation, a simple conservative approach may be taken, to use the data acknowledged bandwidth instead of the transmit bandwidth, such as for example, [00054] Let Tx[i] be the total data sent by RTMP server 120 to RTMP client 200 at a given instance i. Let Ta[i] be the actual amount of data acknowledged to TCP stack 103 by RTMP
client 200 at a given instance i. Let At be the time elapsed to get Ta[i], then the acknowledgement based bandwidth estimate (AckBWe) is calculated as follows:
AckBWe[i] = Ta[i]/At [00055] In an alternative embodiment, the bandwidth estimate at a given instance i is computed as a function of the TCP congestion window (cwnd), maximum segment size (mss) and smoothed round trip time (srtt) as follows:
BWe[i] = (cwnd[i] * mss[i])/srtt[i]
client 200 at a given instance i. Let At be the time elapsed to get Ta[i], then the acknowledgement based bandwidth estimate (AckBWe) is calculated as follows:
AckBWe[i] = Ta[i]/At [00055] In an alternative embodiment, the bandwidth estimate at a given instance i is computed as a function of the TCP congestion window (cwnd), maximum segment size (mss) and smoothed round trip time (srtt) as follows:
BWe[i] = (cwnd[i] * mss[i])/srtt[i]
[00056] The following time-varying discrete time filter can be used to smooth the oscillation in the bandwidth estimate:
ffiWe[i] = (1 - exp(-8t/T0)) * BWe[i] + exp(-St/T0) * fBWe[i-1 ]
where fBWe[i] is the infinite impulse response filtered bandwidth estimate at a given instance i;
fBWe[i-1] is the infinite impulse response filtered bandwidth estimate at a given instance prior to i; BWe[i] is the instantaneous bandwidth estimate obtained as described above;
To is a time constant, for example, 1 (one) second; St is the time interval between the last two estimates; and {E6323648.DOC; I}
the exp(x) function returns the value of e (the base of natural logarithms) raised to the power of x.
ffiWe[i] = (1 - exp(-8t/T0)) * BWe[i] + exp(-St/T0) * fBWe[i-1 ]
where fBWe[i] is the infinite impulse response filtered bandwidth estimate at a given instance i;
fBWe[i-1] is the infinite impulse response filtered bandwidth estimate at a given instance prior to i; BWe[i] is the instantaneous bandwidth estimate obtained as described above;
To is a time constant, for example, 1 (one) second; St is the time interval between the last two estimates; and {E6323648.DOC; I}
the exp(x) function returns the value of e (the base of natural logarithms) raised to the power of x.
[00057] Such impulse response filtering reduces the frequency in switching up and down due to network bandwidth oscillations, for example from 768 Kbps to 1024 Kbps and back to 768 Kbps from 1024Kbps. In such cases it is prudent to stay at 768 Kbps to avoid the switching artifacts from going back and forth between 768 kbps and 1024 kbps. Also TCP
stack socket 103 sends the buffer occupancy, which when estimated properly, can reflect the RTMP client 200 buffer occupancy. Such embodiment of bandwidth estimation closely following the underlying TCP transport protocol, leads to an optimal and intelligent dynamic adaptive rate switching to provide a good QoS and smooth streaming to the RTMP client 200, completely transparent to client 200.
stack socket 103 sends the buffer occupancy, which when estimated properly, can reflect the RTMP client 200 buffer occupancy. Such embodiment of bandwidth estimation closely following the underlying TCP transport protocol, leads to an optimal and intelligent dynamic adaptive rate switching to provide a good QoS and smooth streaming to the RTMP client 200, completely transparent to client 200.
[00058] Based on the new change in stream bandwidth, optimal video transcoding options are determined and fed to the tunable transcoder 101 for getting the desired streaming bit rate.
When the new rate stream becomes available, then at the beginning of the video key frame, time synched properly with audio will be sent to the RTMP client 200. This process of dynamically adapting to the changing network condition continues until RTMP client 200 terminates the connection or when the media is finished.
When the new rate stream becomes available, then at the beginning of the video key frame, time synched properly with audio will be sent to the RTMP client 200. This process of dynamically adapting to the changing network condition continues until RTMP client 200 terminates the connection or when the media is finished.
[00059] Server 120 initiated bandwidth adaption of the stream is achieved for live RTMP
stream, without RTMP client 200 actually asking for change of bit rate, by switching to the new bandwidth stream at the next nearest matching H.264 keyframe video and also appropriately synchronizing audio and video.
stream, without RTMP client 200 actually asking for change of bit rate, by switching to the new bandwidth stream at the next nearest matching H.264 keyframe video and also appropriately synchronizing audio and video.
[00060] During the initial connection to the Media Server 110, the RTMP
client 200 is not provided with any option to select the bit rate and is just given the live channel name along with the RTMP connection URL that is used to request the media.
client 200 is not provided with any option to select the bit rate and is just given the live channel name along with the RTMP connection URL that is used to request the media.
[00061] The adaptive bit rate switching is made possible directly by the RTMP server 120 by changing the transmitted media bit rate when appropriate. The RTMP client 200, when using the system according to the invention, need not issue a play2 switch command to RTMP server 120.
(E6323648.DOC; 1}
(E6323648.DOC; 1}
[00062] For the sake of convenience, the embodiments above are described as various interconnected functional blocks or distinct software modules. This is not necessary, however, and there may be cases where these functional blocks or modules are equivalently aggregated into a single logic device, program or operation with unclear boundaries. In any event, the functional blocks and software modules or features of the flexible interface can be implemented by themselves, or in combination with other operations in either hardware or software.
[00063] While particular embodiments have been described in the foregoing, it is to be understood that other embodiments are possible and are intended to be included herein. It will be clear to any person skilled in the art that modifications of and adjustments to the foregoing embodiments, not shown, are possible.
{E6323648.DOC; 1)
{E6323648.DOC; 1)
Claims (3)
1. A media server, comprising a. a RTMP server configured to communicate streaming media content to a RTMP
client;
b. a stream channel monitor configured to evaluate the quality of the data stream between the RTMP server and the RTMP client;
c. a bandwidth estimator for providing an estimate of the available bandwidth available for streaming the media content;
d. a tunable transcoder for streaming the media content to the RTMP server at the estimated available bandwidth.
client;
b. a stream channel monitor configured to evaluate the quality of the data stream between the RTMP server and the RTMP client;
c. a bandwidth estimator for providing an estimate of the available bandwidth available for streaming the media content;
d. a tunable transcoder for streaming the media content to the RTMP server at the estimated available bandwidth.
2. A method of providing dynamic adaptive multi bit rate live streaming for a RTMP client from an RTMP server, comprising:
a. said RTMP server determining a first video bit rate for said live streaming in consideration of network conditions;
b. said RTMP server sending said video stream to said RTMP client;
c. said RTMP server determining a second bit rate in consideration of changes in network conditions;
d. said RTMP server sending said video stream to said RTMP client at said second bit rate.
a. said RTMP server determining a first video bit rate for said live streaming in consideration of network conditions;
b. said RTMP server sending said video stream to said RTMP client;
c. said RTMP server determining a second bit rate in consideration of changes in network conditions;
d. said RTMP server sending said video stream to said RTMP client at said second bit rate.
3. The method of claim 2 wherein said RTMP client is not provided with a choice of bit rates from said RTMP server.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA 2804741 CA2804741A1 (en) | 2013-01-21 | 2013-01-21 | Media server |
| PCT/CA2014/000038 WO2014110670A1 (en) | 2013-01-21 | 2014-01-21 | Media server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA 2804741 CA2804741A1 (en) | 2013-01-21 | 2013-01-21 | Media server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2804741A1 true CA2804741A1 (en) | 2014-07-21 |
Family
ID=51208893
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA 2804741 Abandoned CA2804741A1 (en) | 2013-01-21 | 2013-01-21 | Media server |
Country Status (2)
| Country | Link |
|---|---|
| CA (1) | CA2804741A1 (en) |
| WO (1) | WO2014110670A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967884A (en) * | 2015-04-17 | 2015-10-07 | 北京奇艺世纪科技有限公司 | Code stream switching method and code stream switching device |
| CN109756488A (en) * | 2018-12-25 | 2019-05-14 | 深圳市网心科技有限公司 | A kind of data flow acquisition methods, device, equipment and medium |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104602044B (en) * | 2015-02-05 | 2019-02-15 | 秦永红 | A kind of RTMP Streaming Media public network live broadcast system and its design method |
| CN107659601B (en) * | 2016-07-26 | 2019-12-17 | 中国科学院声学研究所 | code rate self-adaption method based on HTTP self-adaption flow |
| CN111629277A (en) * | 2020-04-15 | 2020-09-04 | 视联动力信息技术股份有限公司 | Video data transmission method, device and computer readable storage medium |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3699910B2 (en) * | 2000-10-31 | 2005-09-28 | 株式会社東芝 | Data transmission apparatus, data transmission method and program |
| US8218657B2 (en) * | 2005-09-02 | 2012-07-10 | Netgear, Inc. | System and method for automatic adjustment of streaming video bit rate |
| US20080040757A1 (en) * | 2006-07-31 | 2008-02-14 | David Romano | Video content streaming through a wireless access point |
| US7962637B2 (en) * | 2006-11-03 | 2011-06-14 | Apple Computer, Inc. | Dynamic adjustments of video streams |
| US7987285B2 (en) * | 2007-07-10 | 2011-07-26 | Bytemobile, Inc. | Adaptive bitrate management for streaming media over packet networks |
| US8001260B2 (en) * | 2008-07-28 | 2011-08-16 | Vantrix Corporation | Flow-rate adaptation for a connection of time-varying capacity |
| US20100121974A1 (en) * | 2008-11-11 | 2010-05-13 | Einarsson Torbjoem | Stepwise probing for adaptive streaming in a packet communication network |
| JP2011009904A (en) * | 2009-06-24 | 2011-01-13 | Hitachi Ltd | Wireless video distribution system, content bit rate control method, and computer readable recording medium having content bit rate control program stored therein |
| CN102131084A (en) * | 2010-01-19 | 2011-07-20 | 深圳市在线通网络科技开发有限公司 | RTMP (Real Time Messaging Protocol) pushing device and method for audio/video streaming media |
| US8904027B2 (en) * | 2010-06-30 | 2014-12-02 | Cable Television Laboratories, Inc. | Adaptive bit rate for data transmission |
| US20120324122A1 (en) * | 2011-06-20 | 2012-12-20 | David Miles | Method and apparatus for server-side adaptive streaming |
-
2013
- 2013-01-21 CA CA 2804741 patent/CA2804741A1/en not_active Abandoned
-
2014
- 2014-01-21 WO PCT/CA2014/000038 patent/WO2014110670A1/en active Application Filing
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967884A (en) * | 2015-04-17 | 2015-10-07 | 北京奇艺世纪科技有限公司 | Code stream switching method and code stream switching device |
| CN104967884B (en) * | 2015-04-17 | 2018-01-26 | 北京奇艺世纪科技有限公司 | A kind of bitstreams switching method and apparatus |
| CN109756488A (en) * | 2018-12-25 | 2019-05-14 | 深圳市网心科技有限公司 | A kind of data flow acquisition methods, device, equipment and medium |
| CN109756488B (en) * | 2018-12-25 | 2021-09-24 | 深圳市网心科技有限公司 | Data stream acquisition method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2014110670A1 (en) | 2014-07-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9191664B2 (en) | Adaptive bitrate management for streaming media over packet networks | |
| KR101942208B1 (en) | Server-side Adaptive Bitrate Control for DLNA HTTP Streaming Clients | |
| EP2415234B1 (en) | Adaptive bitrate management for streaming media over packet networks | |
| EP2962435B1 (en) | Link-aware streaming adaptation | |
| CN106375783B (en) | Method for quality-aware adaptive streaming over hypertext transfer protocol | |
| CA2953422C (en) | Server side adaptive bit rate control for http streaming clients | |
| EP3382992B1 (en) | Cross-layer optimized adaptive http streaming | |
| CN104967872B (en) | Live broadcasting method and server based on dynamic self-adapting code rate transport protocol HLS Streaming Media | |
| EP2364017B1 (en) | Method, system and user device for obtaining key frame in streaming media service | |
| KR20170101192A (en) | Link-aware streaming adaptation | |
| WO2014110670A1 (en) | Media server | |
| Luthra et al. | Server-Based Smart Adaptive Bit Rate (SABR) Streaming With Statistical Multiplexing | |
| Gunnam | Investigation of Different DASH Players: Retrieval Strategy & Quality of Experience of DASH | |
| EP3228035A1 (en) | Server-side adaptive bit rate control for dlna http streaming clients |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FZDE | Discontinued |
Effective date: 20160121 |