US7953880B2 - Content-aware adaptive packet transmission - Google Patents
Content-aware adaptive packet transmission Download PDFInfo
- Publication number
- US7953880B2 US7953880B2 US11/560,457 US56045706A US7953880B2 US 7953880 B2 US7953880 B2 US 7953880B2 US 56045706 A US56045706 A US 56045706A US 7953880 B2 US7953880 B2 US 7953880B2
- Authority
- US
- United States
- Prior art keywords
- content elements
- content
- receiver
- transmission
- rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/26—Flow control; Congestion control using explicit feedback to the source, e.g. choke packets
- H04L47/263—Rate modification at the source after receiving feedback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/32—Flow control; Congestion control by discarding or delaying data units, e.g. packets or frames
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
- H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/80—Responding to QoS
Definitions
- the embodiments of the present invention relate to streaming data, particularly to packet transmission scheduling and source pruning.
- a multimedia stream may be transmitted, for example, over network segments with varying link capacities. Ways of dynamically adapting a multimedia stream to varying network conditions are thus highly desirable for efficient transmission of the multimedia stream.
- a method of streaming a source content from a sender to a receiver via a network includes the steps of determining, by the sender, a transmission rate; determining, in priority index order, a first set of content elements based on a rate constraint for the receiver, wherein said first set of content elements is associated with content elements of said source content within a transmission buffer, and wherein said rate constraint is based on said determined transmission rate; and determining, in transmission order, whether each of said content elements in said determined first set of content elements meets its associated delay constraint, and, if said content element does not meet its said associated delay constraint, identifying said content element not meeting its said associated delay constraint as a drop content element.
- a device adapted to be operably coupled to a network includes a transmission buffer, a rate-determination module, a selective dropping module, and a buffer transmitter module.
- the transmission buffer is adapted to store content elements associated with a source content that includes a plurality of content elements.
- the rate-determination module is adapted to determine a transmission rate.
- the selective dropping module is adapted to determine, in priority index order, a first set of content elements based on a rate constraint for the receiver, wherein said first set of content elements is associated with said content elements of said transmission buffer, and wherein said rate constraint is based on said determined transmission rate determined by said rate-determination module; and determine, in transmission order, whether each of said content elements in said determined first set of content elements meets its associated delay constraint, and, if said content element does not meet its said associated delay constraint, identify said content element not meeting its said associated delay constraint as a drop content element.
- the buffer transmitter module is adapted to transmit, in transmission order, a filtered set of content elements, wherein said filtered set comprising content elements of said transmission buffer without said content elements associated with said identified drop content elements.
- a system in another aspect, includes a sender coupled to a receiver via one or more network segments and a receiver.
- the sender includes a transmission buffer, a rate-determination module, a selective dropping module, and a buffer transmitter module.
- the transmission buffer is adapted to store content elements associated with a source content, which includes a plurality of content elements.
- the rate-determination module is adapted to determine a transmission rate.
- the selective dropping module is adapted to determine, in priority index order, a first set of content elements based on a rate constraint for said receiver, wherein said first set of content elements is associated with said content elements of said transmission buffer, and wherein said rate constraint is based on said determined transmission rate determined by said rate-determination module; and determine, in transmission order, whether each of said content elements in said determined first set of content elements meets its associated delay constraint, and, if said content element does not meet its said associated delay constraint, identify said content element not meeting its said associated delay constraint as a drop content element.
- the buffer transmitter module is adapted to transmit to said receiver, in transmission order, a filtered set of content elements, wherein said filtered set comprising content elements of said transmission buffer without said content elements associated with said identified drop content elements.
- the receiver is adapted to receive and present said filtered set of content elements.
- FIG. 1 is a high-level block diagram of an exemplary system according to an embodiment of the invention.
- FIG. 2 is a high-level block diagram of an adaptive and selective streaming source content distribution or transmission system, according to an embodiment of the invention
- FIG. 3 is a block diagram of exemplary content elements of an exemplary source content, according to an embodiment of the invention.
- FIG. 4 is a high-level flowchart showing an exemplary adaptive and selective transmission process, according to an embodiment of the invention.
- FIG. 5 is a high-level block diagram illustrating exemplary information that may be contained in an exemplary feedback report and may be maintained by a sender, according to an embodiment of the invention
- FIG. 6 is a high-level flowchart illustrating an exemplary transmission rate determination process of an exemplary adaptive and selective transmission system, according to an embodiment of the invention
- FIG. 7 is a high-level flowchart illustrating an exemplary selective dropping process, which determines content elements to be dropped or filtered, according to an embodiment of the invention
- FIGS. 8A , 8 B, 8 C, 8 D, 8 E, 8 F, 8 G, and 8 H illustrate exemplary tables representing exemplary content elements of exemplary transmission buffers, according to embodiments of the invention
- FIGS. 9A and 9B together illustrate another exemplary selective dropping process of an exemplary adaptive and selective transmission system, according to an embodiment of the invention.
- FIG. 10 is a high-level block diagram of an exemplary sender or sending entity, according to an embodiment of the invention.
- FIG. 11 is an exemplary graph showing frame priority index versus frame loss group of picture (GOP) peak signal-to-noise ratio (PSNR), according to an embodiment of the invention
- FIG. 12 is an exemplary correlation coefficients graph, according to an embodiment of the invention.
- FIGS. 13 and 14 are exemplary graphs showing rate-distortion results, according to embodiments of the invention.
- reference numerals within the one hundred series are initially introduced in FIG. 1
- reference numerals in the two hundred series are initially introduced in FIG. 2
- reference numerals in the eight hundred series e.g., 826 and 830
- FIG. 8 reference numerals in the eight hundred series
- the embodiments of the present invention generally relate to streaming source content or media, such as video—both visual and audio, streaming visual data, streaming audio data, and streaming control data.
- Streaming media in general is the transfer of source content so that this content may be received as a continuous real-time stream.
- Streamed source content elements are typically transmitted by a sender, e.g., a server/server application or sender entity, and received by a receiver, e.g., client/client application or receiver entity.
- the receiver or client typically may start presenting or playing back the source content as soon as the receiving client application has sufficient data or content elements stored in its receiving buffer.
- the playback or presentation typically continues until the end of the presentation of the source content.
- FIG. 1 is an exemplary diagram of a system 100 wherein digital source content, such as audio and/or visual/image, data, are transmitted or streamed according to some embodiments of the invention.
- a local network 150 includes a number of consumer electronics, including a set-top box 134 , a digital television (DTV) 138 , a wireless personal computer (PC) 142 , a digital video or versatile disc (DVD) player 136 , a computer laptop 114 , a gateway/router 102 , and a consumer appliance/device 122 , connected via various network links or segments.
- DTV digital television
- PC personal computer
- DVD digital video or versatile disc
- the local network 150 comprises various networks—e.g., power line communication (PLC) networks, 802.11a wireless networks, 802.11g wireless networks, and 802.11b wireless networks. Future network specifications such as 802.11n may also be incorporated in such networks.
- the local network 150 may be operably coupled to one or more source content providers 192 , 198 , for example, via satellite, cable, and/or terrestrial broadcast 190 or via an external wide area network, such as the Internet 194 .
- a source content provider 192 , 198 may provide pre-encoded and stored source content and/or live real-time or substantially real-time encoded source content to be received by a receiver/client and accordingly be presented in a user interface. For example, a movie may be requested from a source provider 198 that provides on-demand pre-encoded and stored data. The encoded source content is then transmitted and streamed over network segments, which may include wide, local, and/or metropolitan area network segments. This source content is then received by a set-top box 134 , for example, via a home wireless network and presented by a digital television 138 .
- a source provider or an intermediate network node also has one or more proxy servers 196 that are operably connected to the source provider 198 .
- a proxy server 196 thus may be a node in the system, for example, where source contents may directly or indirectly be requested.
- the receivers or clients of such streaming media content are depicted to be within a local area network, the embodiments of the invention may also apply to other types of receivers, for example, mobile devices adapted to receive and/or present wireless streaming source content. These wireless devices may also be incorporated in vehicles.
- FIG. 2 is a high-level block diagram 200 showing an exemplary content-aware adaptive and selective streaming system for the delivery of streaming media or source content, which may be employed over variable bit-rate channels or links.
- the content-aware adaptive and selective system 200 of the present invention includes monitoring network conditions by utilizing feedbacks 260 from receivers, and dropping or filtering of encoded content elements, typically of a transmission buffer, in order to match a dynamically changing target bandwidth.
- the exemplary system 200 is also typically adapted to adjust transmission rates so as to match or adjust to receiver capacity and/or network congestion.
- the adaptive and selective transmission system in some embodiments, prunes or filters the source content transmitted to the receiver.
- components of the exemplary system may perform transrating and/or transcoding.
- the adaptive system includes an iterative process that determines which content elements are to be dropped based on rate constraints and delay constraints of the network channel.
- a media source provider 192 , 198 is providing streaming source content to a consumer 270 , which is presented 268 by a receiver 250 .
- the original source content may have been previously captured or captured in real-time.
- the original source content is captured 204 and then encoded 206 by an encoder module 206 .
- the step of encoding 206 typically includes dividing the original source content 204 into one or more components, e.g., content elements, and compressing such components into one or more encoded source content elements.
- the structure, format, and/or data contained in the source content and the content elements may depend on the compression technology, e.g., codec or standard being supported.
- the encoded source content elements of the source content are typically received and processed by a sender 210 .
- the encoder module 206 and the sender module 210 may be embodied in separate or the same entities, such as devices or applications.
- pre-encoded data 220 may be encoded and transmitted by a media provider to the sender 210 , which may be regarded as a proxy server.
- the sender 210 functioning as the proxy server, filters the received encoded data for transmission 230 to the receiver 250 .
- the sender functions both as the encoder 206 and the sender 210 .
- a sender 210 herein is also referred to as a server, sender module, or a sender entity.
- the sender performs the adaptive and selective transmission (AST) process described herein, which includes determining which content elements are not to be transmitted or dropped/filtered.
- AST adaptive and selective transmission
- the sender performs the AST process on encoded source contents, including their encoded source content elements.
- encoded source content and content elements are herein also referred to as source content and content elements, respectively.
- a sender for example, may be embodied as a media server or a proxy server.
- the encoded source content, including content elements, 220 is typically stored, e.g., in a transmission (TX) buffer 212 prior to transmission to the receiver 250 by the buffer transmitter module 208 .
- TX transmission
- the buffer transmitter module 208 in general interfaces with the TX buffer 212 and handles or performs the transmission or streaming of the content elements in the transmission buffer 212 to the receiver 250 .
- a module within the sender 210 or separate but interfacing with the sender 210 may also be included in the system, not shown, to perform transrating and/or transcoding functions. Typically, the transrating and/or transcoding functions are performed prior to applying the AST process.
- the feedback and rate module 218 typically receives and processes the feedbacks 260 from receivers 250 and also determines transmission rates based on the received feedbacks 260 .
- the determined transmission rate may then be applied by the selective dropping module (SDM) module 214 in determining or identifying the content elements within the transmission buffer that are to be dropped/filtered, i.e., not sent to the receiver.
- SDM selective dropping module
- the SDM 214 modifies the content of TX buffer 212 , for example, by deleting content elements in the TX buffer or by setting the appropriate flags associated with the content elements.
- the SDM creates a drop set identifying the content elements to be dropped, which may then be used by the buffer transmitter module 208 to identify which content elements in the TX buffer 212 are to be dropped and not transmitted.
- the encoded data or content elements 220 received by the sender 210 from the encoder 206 are filtered by the sender 210 , i.e., some content elements are intentionally not transmitted by the sender 210 to the receiver 250 .
- the filtered encoded data or content elements 230 are then delivered or transmitted 240 , for example, via one or more network segments 240 , wired or wireless, using a transport protocol, which may include user datagram protocol (UDP), transmission control protocol (TCP), real-time transport protocol (RTP), RTP control protocol (RTCP), and the like.
- UDP user datagram protocol
- TCP transmission control protocol
- RTP real-time transport protocol
- RTCP RTP control protocol
- the filtered set of content elements 230 is then received by the receiver 250 , which typically includes a decoder module 254 , which then decodes the filtered encoded data for presentation 268 to a user/consumer 270 .
- the decoder 254 to appropriately decode the received filtered content elements 230 , typically supports the decompression and codec scheme performed by the encoder, i.e., adapted to support a common interpretation scheme such that the decoder is able to reconstruct the bit stream(s) into a format that may be used for presentation.
- a receiver is herein also referred to as a receiving entity or a client.
- the receiver for example, may be embodied as a media player.
- the receiver 250 also transmits feedbacks to the sender 210 .
- These feedbacks 260 may be transmitted via the same network via which the filtered encoded data 230 have been sent.
- these feedbacks 260 may be embodied as RTCP receiver reports (RR), for example, as described in the Request for Comments (RFC) 3550 of the Network Working Group.
- RFC 3550 document is herein referred to as RFC3550.
- RFC3550 also describes RTP.
- RFC3550 is available, as of this time of writing, from http://www.ietf.org/rfc/rfc3550.txt.
- the frequency and timing of when feedbacks are transmitted to the sender 210 may be based on conditions described in RFC3550. Other variations on timing and frequency of feedbacks may also be implemented within the system.
- the embodiments of the present invention may also apply to situations wherein a sender transmits streaming source content to more than one receiver (not shown), for example, in a multi-cast environment.
- the sender is adapted to dynamically adjust to the various receivers' bandwidth.
- the sender 210 may adjust to two varying receiver network conditions.
- a different set of filtered source content 230 may be sent to the first receiver and another different set of filtered source content 230 may be sent to the second receiver (not shown).
- a sender may perform adaptive and selective transmission per receiver.
- a sender may perform adaptive and selective transmission based on a condition, such as the average network condition of all receivers or the worst restrictive network condition among participants of a session.
- the adaptive and selective transmission may also be performed for only a subset of receivers within a session, applied randomly on random receivers, and in other varying means.
- a condition such as the average network condition of all receivers or the worst restrictive network condition among participants of a session.
- the adaptive and selective transmission may also be performed for only a subset of receivers within a session, applied randomly on random receivers, and in other varying means.
- the adaptive transmission feature of the present invention may be applied in various manners.
- RTP and RTCP are employed to transmit the filtered encoded source content 230 , as well as to provide feedbacks 260 .
- RTP and RTCP other protocols may also be employed.
- the use of RTP and RTCP, including RTP header formats and structures, is for illustrative purpose and is not intended to limit the scope of the invention.
- the embodiments of the invention are described in terms of frames, other content element structures, e.g., fields, may also be employed.
- the filtered encoded source content elements of the present invention may be transmitted as RTP packets.
- RTP generally relates to a real-time transport protocol that typically provides end-to-end delivery service for data typically with real-time properties. In some applications, RTP may be used with UDP or other suitable underlying network or transport protocols. RTP may also support data transfer to multiple destinations or receivers, e.g., multicast distribution.
- Content elements, particularly encoded content elements, embodied in RTP are usually transmitted or delivered as RTP packets.
- An RTP packet typically includes an RTP header and payload data.
- An RTP payload typically refers to the data transported by RTP in the packet, e.g., source content elements, such as audio samples or compressed video data.
- the payload data may be a portion of a frame or an entire frame.
- RTP packets are generally delivered in sequence—identified by a sequence number, thereby enabling a receiver to reconstruct the sender's packet sequence, if appropriate.
- Table I below shows portions of an exemplary RTP packet, which may contain the source content elements payload.
- RTP Packet Structure/Format FIELD DESCRIPTION: Sender or Source Source or Transmitter of a stream of RTP packets, also called Identifier Synchronization Source (SSRC) (referred in RFC3550 as SSRC) Sequence
- SSRC Identifier Synchronization Source
- Sequence The sequence number is typically incremented by one for each RTP Number data packet sent. This sequence number enables a receiver to detect packet loss and to restore packet sequence. Timestamp Sampling Instant. Timestamps may be used to place received video or content element packets in the correct timing order. Typically, the timestamp is incremented monotonically and linearly in time to enable synchronization and jitter calculation. In some embodiments, several consecutive RTP packets may have the same timestamp value, if they are logically generated at once, for example, the data packets belong to the same video frame. The sequence numbers of the packets transmitted, however, are still monotonic.
- RTCP generally relates to RTP and is typically used to monitor the quality of service and to convey information about the participants in an on-going RTP or RTCP session.
- RTCP is typically based on the periodic transmission of control packets to all participants in the session.
- RTCP receiver reports are employed to transmit feedbacks from the receiver(s) 250 to the sender 210 .
- This feedback report may include or indicate the current network condition, including, for example, an indication of the amount of data that may be transmitted to the receiver or receiver capacity.
- the feedback reports 260 may be sent on a periodic basis or based on other conditions, e.g., such as those defined or supported within RTCP (see RFC3550) or based on bandwidth constraints.
- a minimum interval between two feedback reports is provided or supported within the system, for example, as described in RFC3550. Table II below shows portions of an exemplary RTCP packet of an RR.
- RTCP Packet Structure/Format of a Receiver Report Field Name Description: Sender/Source (e.g., Source or Transmitter identifier (typically the receiver 250) Synchronization Source for the originator of this RR packet. (SSRC) of sender) Timestamp Time when the RR was sent Fraction Lost The fraction of RTP data packets from source SSRC (e.g., the sender 210) lost since the previous RR packet was sent. In some embodiments, this fraction may be defined to be the number of packets lost divided by the number of packets expected. An exemplary implementation is disclosed in Appendix A.3 of RFC3550. In some embodiments, if the loss is negative due to duplicates, the fraction lost is set to zero.
- Sender/Source e.g., Source or Transmitter identifier (typically the receiver 250) Synchronization Source for the originator of this RR packet. (SSRC) of sender) Timestamp Time when the RR was sent
- SSRC Sender/Source
- a receiver is unable to determine whether any packets are lost after the last one is received, and there may be no reception report block issued for a source if all packets from that source sent during the last reporting interval have been lost.
- SSRC Packets Lost transmitter
- the number of packets expected is typically defined as the extended last sequence number received, less the initial sequence number received.
- An exemplary calculation is shown in Appendix A.3 of the RFC3550.
- Extended Highest The extended highest sequence number received in an RTP Sequence Number data packet from source SSRC. Received
- the fields “Fraction Lost,” “Cumulative Number of Packets Lost,” and “Extended Highest Sequence Number Received” are transmitted by the receiver typically within the RR or a report block for a sender typically identified by its SSRC.
- SRRC in a report block corresponds to and helps identify the sender whose information and/or statistical data are reported in the report block.
- Other information may be included in the exemplary RR packets, e.g. packet count number, inter-arrival jitter, and user-defined fields. Variations of fields included in the feedback 260 are expected and still be in the scope of the invention.
- the sender 210 may modify the source content to be transmitted based on feedbacks, e.g., RRs, received from the receiver(s) 250 .
- feedbacks e.g., RRs
- cumulative counts may be used in RRs so that differences may be calculated between any two RRs to determine measurements over both short and long time periods, thereby providing additional resilience against the loss or non-receipt of an RR.
- the difference between the last two RRs received may be used to estimate the recent quality of the distribution.
- the timestamp e.g., the network time protocol (NTP)
- NTP network time protocol
- An exemplary calculation that may be performed is the packet loss rate over the interval between two RRs.
- the difference in the cumulative number of packets lost may provide the number of packets lost during that interval.
- the difference in the last sequence numbers received may provide the number of packets expected during the interval.
- the ratio of the difference in the cumulative number of packets lost and the difference in the last sequence number received may be used to define the packet loss fraction over the interval. This ratio typically equals the fraction lost field (e.g., see Table II) transmitted in the RR, if the two RRs are consecutive, but otherwise it may not.
- the loss rate per second may also be determined by dividing the loss fraction by the difference in timestamps expressed in seconds.
- the number of packets received may be counted.
- the number of packets expected may also be used to determine or weigh the statistical validity of any loss estimates.
- fraction lost field may also provide a short-term measurement from a single report. This may become more important as the size of a session scales up enough that reception state information might not be kept for all receivers or the interval between reports becomes long enough that only one report might have been received from a particular receiver.
- FIG. 3 is a high-level diagram 300 depicting an exemplary encoded source content, for example, an encoded video consisting of a number of content elements, e.g., frames 302 , 304 , 306 , 308 , 310 , 312 , 314 , 316 , 318 , 324 .
- a source content typically consists of a set of content elements. These content elements, typically depending on implementation, may be frames, packets, groups of pictures (GOPs), slices, pictures, groups of slices, fields, layers, macroblocks, and other data units. The content elements may be subdivided into further content elements, e.g., a frame may be further defined by fields or macroblocks.
- the original source content is thus typically encoded to appropriately generate properly encoded content elements.
- the encoding algorithm and the decoding algorithm used by an encoder and decoder, respectively, may depend on the standard being supported.
- the source content is divided into frames 302 - 324 .
- the AST process described herein selectively drops/filters content elements at the frame level.
- the AST process may operate or process at sub-frame or packet level.
- Other content element data units may also be applied.
- a frame may be packetized in a sequence of packets.
- an encoded frame may be packetized and transmitted as multiple RTP packets.
- one frame is transmitted as one RTP packet.
- Other variations in the manner of packetizing the encoded content element are expected and within the scope of the present invention.
- the sender typically performs the adaptive and selective transmission (AST) process on an encoded source content or portions thereof.
- the entire video sequence or source content may have been previously encoded 206 and pre-stored or may be encoded 206 live or in real time.
- the exemplary frames 302 - 324 are shown in transmission order, where the first frame 302 , fr 1 , is the first encoded frame to be transmitted, fr 2 304 is the next frame to be transmitted, and frn 324 is the last encoded frame to be transmitted.
- fr 1 302 through fr 8 316 are in the TX buffer 212 .
- Other frames, e.g., fr 9 318 to frn 324 are still being encoded.
- the AST process is applied to fr 1 302 -fr 8 316 .
- the AST process is performed only on content elements in the TX buffer, or portions thereof.
- the contents of a TX buffer 212 may change, for example, because of incoming new streaming content elements that are to be transmitted.
- the TX buffer 212 may be regarded as a series of windows 330 , 350 .
- the AST process for example, may be applied in a time-window-based approach in which all content elements or packets with timestamps within a certain period of time are placed in a window and processed by applying AST before transmission to the receiver.
- the window approach applies to the frames already in the TX buffer.
- An exemplary window 330 is shown, bounded by timestamp values T a 332 and T b 338 , which includes several frames fr 1 -fr 8 302 - 316 .
- Another window 350 may be defined for another set of frames, assuming that such set of frames are in the TX buffer 212 .
- the content elements of the transmission buffer that are processed by the AST process are based on when the next transmission opportunity may occur. This transmission opportunity may be controlled by an application layer, e.g., by the buffer transmitter module 208 .
- the AST process is applied to only the content elements in the TX buffer, which were originally scheduled to be transmitted in the next transmission opportunity if the AST process were not applied.
- content elements are assigned to windows based on their delivery deadlines.
- the AST process, particularly the selective dropping process is applied to all the contents within the TX buffer.
- the sender may apply the AST process over these frames 302 - 316 .
- a drop set 360 is typically determined and maintained at the sender.
- the drop set 360 includes fr 6 312 and fr 4 308 .
- the drop set 360 maintains the content elements that are to be dropped and not transmitted by the sender.
- the sender 210 of the present invention may intentionally delete content elements in the transmission buffer, identify content elements for deletion, or based on a look-up of the drop set determine whether or not to transmit content elements in the transmission buffer.
- content elements not transmitted in the transmission buffer are eventually eliminated from the transmission buffer.
- the dropping is only applied to the next unit scheduled for transmission from the transmission buffer.
- the sender typically transmits a filtered set of source contents, e.g., only fr 1 302 , fr 2 304 , fr 3 306 , fr 5 310 , fr 7 314 , and fr 8 316 —without fr 4 308 and fr 6 312 .
- a receiver may perform some processing to the received filtered set of content elements, e.g., processes adapted to handle dropped packets or perform error correction. In other embodiments, receiver-side processing is not performed.
- the AST process drops entire frames.
- the sender may decide to drop only a few packets from a frame instead of the entire frame.
- the sender 210 may, instead of dropping a frame, replace an original frame with an alternate frame, such as a smaller size frame or a frame with lesser bits, with data representing a similar frame as the previous or next frame.
- the sender 210 may also decide the timing of when to transmit an encoded frame and the amount of data or content element to send at each transmission opportunity.
- FIG. 4 is a high-level flowchart of an exemplary adaptive and selective transmission (AST) process 400 according to some embodiments of the invention.
- the sender 210 first determines the transmission rate based on feedbacks, e.g., RRs, received (step 408 ). Other factors, however, may also be considered such as current backlog of data in the transmission buffer, amount of outstanding/unacknowledged data/packets on the network, past RRs, etc.
- the sender typically then applies the received feedback information to estimate the amount of data that may be transmitted for the probable current network condition.
- This determination process is further exemplified in FIG. 6 and may be performed by the feedback and rate module 218 .
- the sender also typically determines the content elements in the transmission buffer to apply the AST process (step 412 ). In the next operation, the sender determines the frames, packets, or content elements to be dropped or filtered (step 416 ). This operation (step 416 ) may be performed by the SDM 214 and is further exemplified and described in FIGS. 7-9B , including the accompanying text.
- the sender then transmits the filtered set of frames in stream order or transmission order, typically using the determined transmission rate (step 420 ).
- the filtered set of content elements is based on the determined drop set. In some embodiments, the transmission order is based on the timestamps encoded within or associated with the content elements. In this embodiment, no out-of-order packets are sent. In other embodiments, packets may be sent out-of-order in order of their importance/priority.
- the AST process 400 of the present invention typically does not require information from the Media Access Control (MAC) layer.
- MAC Media Access Control
- This adaptive transmission process 400 is typically repeated, for example, based on a cycle, periodically, or based on other conditions as defined within the system (step 424 ).
- the AST process may be repeated until the entire source content is fully transmitted, decoded, and presented to a user, thereby enabling the sender to dynamically adjust to the network conditions of the receivers.
- the AST process is repeated based on the cycle of when RTCP RRs are received by the sender.
- Other conditions of when the AST process is performed or the number of iterations may also be defined, for example, periodically, repeated a defined number of times, based on transmission opportunities, and/or repeated based on a defined condition.
- FIG. 5 is a high-level block diagram of data that may be contained in an exemplary RR 260 .
- RR(m) denote an exemplary RTCP receiver report m containing or indicating:
- the sender 210 keeps track of the cumulative number of bytes of data payload sent 520 , C(s), since the beginning of transmission after sending an RTP packet with sequence number s.
- FIG. 6 illustrates a flowchart of an exemplary rate-determination process 408 that determines the transmission rate for a particular receiver based on that receiver's RRs.
- this transmission rate is applied by the SDM.
- the transmission rate is based on when bits, for example, may leave or be transmitted from the transmission buffer and be received by the decoder or receiver after a constant delay.
- the transmission rate determines the size of content elements over time, size of content elements/time, the receiver may be able to support.
- variable R(0) is set to the default stream rate and the variable D is set to zero (step 604 ).
- the rate-determination process 408 of the exemplary feedback and rate module 218 typically starts by reading or retrieving feedback information, particularly of BR(m) and RR(m-1) (step 606 ). By using these RRs, the rate-determination process may calculate rates based on current or recent network conditions.
- O T may be defined based on an estimate or knowledge about the maximum outstanding bytes that may be held in various buffers (e.g. sender and receiver network stack buffers). If the condition (step 612 ) is not met, a new transmission rate is calculated, for example, by an exemplary equation (step 616 ):
- R ⁇ ( t m ) ⁇ * ( C ⁇ ( hs ⁇ ( m ) ) - C ⁇ ( hs ⁇ ( m - 1 ) ) ) * ( 1 - fr ⁇ ( m ) ) ( t m ′ - t m - 1 ′ )
- ⁇ is defined, for example, to be 0 ⁇ 1.
- an exemplary manner in determining a transmission rate is based on considering the difference between the number of packets received between the two reports, the fraction of packets lost by the receiver, and the time interval between the two reports, which may indicate or estimate the current or recent capacity of the receiver or the network congestion. Other calculations or manners of determining transmission rates, such as:
- R ⁇ ( t m ) ⁇ * ( C ⁇ ( hs ⁇ ( m ) ) - C ⁇ ( hs ⁇ ( m - 1 ) ) ) * ( 1 - fr ⁇ ( m ) ) ( t m - t m - 1 ) ⁇ ⁇ or b )
- R ⁇ ( t m ) ( C ⁇ ( hs ⁇ ( m ) ) - C ⁇ ( hs ⁇ ( m - 1 ) ) ( t m - t m - 1 ) ⁇ ⁇ or c )
- R ⁇ ( t m ) ( C ⁇ ( hs ⁇ ( m ) ) - C ⁇ ( hs ⁇ ( m - 1 ) ) ( t m ′ - t m - 1 ′ )
- the new transmission rate R(t m ) is assigned the minimum of the previous calculated transmission rate plus D, R(t m-1 )+D, and the default stream rate R(0) (step 624 ). This calculation is made so as to account for possible decreased network congestion or interference, for example.
- R(t m ) may be calculated as a filtered valued using the previous transmission rate. For example:
- R ⁇ ( t m ) ⁇ * ( C ⁇ ( hs ⁇ ( m ) ) - C ⁇ ( hs ⁇ ( m - 1 ) ) ) * ( 1 - fr ⁇ ( m ) ) ( t m - t m - 1 ) + ( 1 - ⁇ ) * R ⁇ ( t m - 1 )
- FIG. 7 is a more detailed flowchart 416 of an exemplary selective-dropping process of the present invention, which may be performed by the SDM 214 .
- FIGS. 8A-8H show exemplary tables representing content elements in exemplary transmission buffers. FIG. 7 and FIGS. 8A-8H are generally discussed together.
- a ⁇ work set associated with the determined set of content elements, may be created or copied for processing and manipulation (step 702 ).
- the drop set is also typically initialized to the empty set (step 702 ).
- the ⁇ work set contains several content elements, for example, frames.
- MPEG-2 generally specifies a generic coding of moving pictures and associated audio and specifies a video stream format, which may be constructed of three types of frame type—intra frames (I-frames), forward predicted frames (P-frames), and bidirectionally predicted frames (B-frames). These frames may be arranged in a specified order sometimes referred to as a group of pictures (GOP) structure.
- MPEG-2 encoding is known to those of ordinary skill in the art. Although the discussion herein relates to the MPEG-2 video coding standard, the embodiments of the present invention may also apply to other video coding standards.
- An exemplary ⁇ work set may be represented by the exemplary table 820 in FIG. 8A .
- the ⁇ work set contains seventeen frames 802 —fr 1 822 , fr 2 824 , fr 3 826 , fr 4 828 , fr 5 830 , fr 6 832 , fr 7 834 , fr 8 836 , fr 9 838 , fr 10 840 , fr 11 842 , fr 12 844 , fr 13 846 , fr 14 848 , fr 15 850 , ft 16 852 , and fr 17 854 .
- these frames may be packetized in various manners, e.g., a frame may be packetized into two or more RTP packets (not shown).
- each frame is packetized into one RTP packet and is associated with a corresponding sequence number 812 .
- the transmission order is typically based on the timestamp, decoding timestamp, 804 of the frame.
- the order of the sequence number 812 is also associated with the timestamp order 804 .
- the content elements are typically transmitted based on sequence number 812 .
- the timestamp 804 typically indicates a time deadline when the frame has to be decoded by a decoder, typically at the receiver, for presentation.
- Each frame is also associated or identified with a frame type 810 , e.g., “I” for I-frame, “B” for B-frame, and “P” for P-frame.
- Each frame is also associated with a priority or importance index 806 , which may be based on the frame type.
- the payload size of each content element, in this case frame is shown, for example, as the number of bits 808 .
- frame 7 , fr 7 834 is a “P-frame” with a sequence number of “7,” a timestamp of “233,” a priority index of “4,” and a payload size represented by “B7.”
- the priority index 806 of a content element may be based on the video compression technology or encoding scheme employed, in particular, the coding dependencies between content elements. For example, contents elements with lesser importance are those that typically minimize the coding dependency chain and/or encoding error propagation.
- the priority index may be based on the amount by which the distortion or peak signal-to-noise ratio (PSNR) at the receiver may be decreased, e.g., by calculation or estimation, if the content element is decoded (on time) at the receiver.
- PSNR distortion or peak signal-to-noise ratio
- the priority index may also be based on the frame type and its position in the group of pictures (GOP).
- the source content is encoded with an IBBPBBPBBPBBPBB.
- the first frame which is an I-frame 822
- fr 1 822 is assigned a priority or importance value of six (“6”).
- the next I-frame, fr 16 852 in the next GOP is also assigned a priority index of “6.”
- the P-frames are then ranked in descending priority value.
- a P-frame depends on the closest I— or P-frame preceding such P-frame.
- P-frame fr 10 840 depends of P-frame fr 7 834 .
- the P-frames fr 4 828 , fr 7 834 , fr 10 840 , and fr 13 846 are assigned priority values “5,” “4,” “3,” and “2,” 806 respectively.
- All the B-frames 824 , 826 , 830 , 832 , 836 , 838 , 842 , 844 , 848 , 850 , 854 are assigned the priority value of one (“1”), considering they have the least dependence and generally may be dropped without impacting other frames.
- the exemplary ⁇ work set 820 shows that there are seventeen frames 820 in the transmission buffer—fr 1 822 to fr 17 854 , to be transmitted by the sender, e.g., by the buffer transmitter module 208 .
- the drop set at this point is initialized to an empty set (step 702 ).
- a first or rate set based on the rate constraint of the receiver is determined, based on a T work set ordered in priority order (step 704 ).
- This first/rate set is typically determined based on the payload size of the content element 808 and generally indicates the set of content elements, which if transmitted may potentially be received by the receiver based on its rate constraint.
- This operation may involve sorting the content elements in the ⁇ work set 820 in priority order, as represented by the next table 860 ( FIG. 8B ), and adding each of the payload size 808 of the frame, in priority order, until the accumulated payload size is typically of the maximum size that may be accommodated by the sender-determined transmission rate/receiver rate constraint.
- the first/rate set 862 is determined to consist of fr 1 822 , fr 16 852 , fr 4 828 , fr 7 834 , fr 10 840 , fr 13 846 , fr 2 824 , fr 3 826 , fr 5 830 , fr 6 832 , fr 8 836 , and fr 9 838 .
- This means that the accumulated payloads of B 1 +B 16 +B 4 + . . . +B 6 +B 8 +B 9 may be supported by the receiver.
- Adding the additional payload of fr 11 842 , B 11 808 , for transmission, however, is calculated to exceed the receiver's rate constraint.
- the selective dropping process typically stops adding frames, e.g., their bits sizes 808 , when the selective dropping process calculated that fr 11 842 may potentially exceed the receiver's rate constraint.
- fr 11 842 and the rest of the exemplary frames, fr 12 844 , fr 14 848 , fr 15 850 , and fr 17 854 are typically not part of this first/rate subset.
- the selective dropping process determines a second set or drop set, indicating frames that even if received at the receiver may not be decoded and/or presented at its specified presentation deadline (step 708 ). For example, after determining the first set 862 ( FIG. 8B ), each content element in this first set, now shown as exemplary table 864 ( FIG.
- fr 6 832 , 866 is determined or calculated so that even if transmitted, that frame 832 may not arrive, not be potentially decoded and/or presented at the receiver at the specified deadline, e.g., based on the decoding timestamp 804 .
- the content element fr 6 832 , 866 is thus added to the drop set (step 712 ).
- the selective dropping process 416 performed, e.g., by the SDM 214 may be repeated, e.g., based on defined conditions, as shown (step 716 ).
- the AST process in general is performed and/or repeated before each transmission opportunity, as defined or implemented in the system.
- the sender e.g., the buffer transmitter module 208
- the filtered set of content elements is typically transmitted in timestamp order or transmission order, which in some embodiments may also be the sequence number, e.g., as represented in the exemplary table 866 ( FIG. 8D ).
- the selective dropping process has calculated that transmitting fr 11 842 , and potentially fr 12 844 , fr 14 848 , fr 15 850 , and fr 17 854 may potentially exceed receiver's rate constraint.
- These frames are not added to the drop set.
- These frames 842 , 844 , 848 , 850 , 854 are left as part of the T work set so they may be part of the SDM process in the next cycle. In other embodiments of the invention, however, these frames fr 11 842 , fr 12 844 , fr 14 848 , fr 15 850 , and fr 17 854 are included in the drop set or not transmitted.
- the SDM module or another module may delete the identified drop set content elements, e.g., fr 6 832 , in the transmission buffer.
- a flag associated with the drop set content element is set so as to indicate that the particular frame is not be transmitted to the client.
- the buffer transmitter module 208 may prior to transmitting the content element, check the drop set to determine if the frame to be transmitted is included in the drop set. If such frame is indicated in the drop set, the buffer transmitter 208 accordingly drops that frame, i.e., not send, and retrieves the next content element in the transmission buffer for transmission, if appropriate.
- the selective dropping process may be repeated based on one or more conditions, e.g., repeated once, repeated a defined number of times, and repeated periodically. In some embodiments, the process is repeated until there is no change to the drop set content elements and/or in the ⁇ work set. Assuming that the selective dropping feature (step 716 ) is to be repeated, the SDM process continues from the table 864 in FIG. 8C to the table 868 of FIG. 8E .
- the drop set content element e.g., fr 6 832 , is ignored in the calculation or eliminated from the next T work set as represented in the table 868 , considering that the process assumes that fr 6 832 is not going to be transmitted.
- the rate set 870 which for this example contains fr 1 822 , fr 16 852 , fr 4 828 , fr 7 834 , fr 10 840 , fr 13 846 , fr 2 824 , fr 3 826 , fr 5 830 , fr 8 836 , fr 9 838 , fr 11 842 , and fr 12 844 , is then again determined based on the updated ⁇ work set 868 (step 704 ). Transmitting fr 14 848 is determined to exceed the receiver's rate constraint. Transmitting fr 15 850 and fr 17 854 may also potentially exceed the receiver's rate constraint. Based from this rate set 870 , represented also in the set 892 but in timestamp-order, in the table 872 of FIG.
- each content element is then checked to determine if the defined deadline may be met.
- fr 13 846 , 890 has been determined, even if transmitted, being unable to meet its deadline.
- This frame fr 13 846 is thus added in the drop set.
- the process if repeated, may then determine another rate set 876 , e.g., represented in the table 874 of FIG. 8G .
- all content elements in the ⁇ work set, without the two drop set content elements fr 6 and fr 13 may be transmitted and still potentially meet the receiver's rate constraint.
- Each content element is then checked to determine if the deadline may be met. In this example, all content elements have been calculated to meet their deadlines.
- the filtered set of content elements that may be transmitted by the sender is represented in the table 880 of FIG. 8H .
- the filtered set 880 is transmitted in transmission order or timestamp order 804 , which in this example, also relates to the sequence number order 812 . Considering that there is no more change to the drop set and/or all the ⁇ work set, the selective dropping process may then be terminated for this exemplary transmission buffer. Frames fr 6 and fr 13 are not transmitted.
- additional dropping strategies may be incorporated as part of the selective dropping procedure.
- fr 13 846 , 890 is a P-frame, thus B-frames that depend on fr 13 846 may also be included in the drop set.
- the content elements in the transmission buffer may have changed, for example, due to new content elements being added in the transmission buffer.
- the selective dropping process may consider such content element in the next cycle of calculation, drop such content element, or send that content element.
- conditions may be defined to handle additional content elements being added to the transmission buffer while the selective dropping process is being performed.
- the filtered set of content elements or a portion thereof is transmitted, e.g., one or more content elements—within the set of filtered content elements but without content elements identified in the drop set—are transmitted.
- fr 6 832 is part of the drop set
- only the first content element in the filtered set of content element in timestamp order see e.g., Table 866 in FIG. 8D , i.e., fr 1 822 , is transmitted.
- the process then repeats itself, for example, by determining another first or rate set, but now without fr 1 822 , and from such first/rate set determine a next drop set. Once the next drop set is identified, only the first content element in timestamp order in that filtered set of content elements is then again transmitted. The process then continues, for example, until there is no more content element in the transmission buffer.
- the adaptive and selective process described herein is applied, e.g., content element by content element, so as to determine if the next content element adapted to be transmitted, e.g., added, in the transmission buffer is to be transmitted or not, based on the process described herein.
- Other variations of process repetition and/or the number of content elements to be transmitted in the determined filtered set of content elements may be also be adapted and still be in the scope of the present invention.
- FIG. 9A and 9B illustrate another exemplary embodiment of the selective dropping module process 416 according to another embodiment of the invention.
- the selective dropping process is typically performed after a rate determination process 408 (see FIG. 4 ).
- a rate determination process 408 see FIG. 4 .
- the transmission buffer typically consists of a number of frames or content elements, for example, as represented in the table 820 ( FIG. 8A ).
- the embodiments of the invention provide for two order sequences for the set of frames in the transmission buffer.
- the exemplary orders are:
- the AST process in general finds the subset of frames in the transmission buffer in timestamp order—SS T (t).
- SS T ⁇ ( t ) ⁇ ⁇ fr ⁇ ( t j T , b j T , p j T ) ⁇ ⁇ ⁇ ⁇ j ⁇ : ⁇ ( t + es j T ⁇ d j T , R ⁇ ( SS T ⁇ ( t ) ) ⁇ R ⁇ ( t m ) ⁇ ⁇ max ( ⁇ SS T ⁇ ( t ) ⁇ p j ⁇ ⁇ ⁇ ( t j ) ) may be applied, for example.
- the subset SS T (t) to be transmitted may iteratively be determined.
- an initialization process is performed, which may include ordering the content elements in the transmission buffer in priority index order.
- this may be handled, for example, by a set of program instructions that orders or sorts data in an array, e.g., based on priority index, for example.
- the ⁇ work set may contain only a subset of all the frames currently in the transmission buffer or may be defined based on the next transmission opportunity.
- the S P (n) or S P (0) thus may be the set of all content elements in the transmission buffer.
- an N n value is determined such that
- ⁇ may be defined as 0 ⁇ 1 and B c is the receiver buffer size in time units or end-to-end delay, e.g., from transmitting from sender to receiving by receiver, and where RR(m) is the last RR received.
- the receiver buffer size may have been previously exchanged between the sender and the receiver.
- N n 11 and is pointing to fr 9 838 , such that the size of each candidate frame in the rate set 862 is added resulting in (B 1 +B 16 +B 4 +B 7 +B 10 +B 13 +B 2 +B 3 +B 5 +B 6 +B 8 +B 9 ) ⁇ *B c *R(t m ).
- step 912 after determining the candidate frames (rate set 862 ), the frames being processed are reordered in transmission order, e.g., based on timestamp order.
- ( t+es y T )> d y T ⁇ and where S T (n,N n ) S P (n,N n ) and t is the current time frame.
- an exemplary stopping condition is verified, wherein the selective dropping process determines whether there has been a changed between the previous drop set and the current drop set.
- the set ⁇ contains all the content elements in the transmission buffer, without the content elements identified in the drop set.
- only the next immediate frame to be sent is dropped, if such frame or content element belongs to the drop set ⁇ (n).
- the selective dropping function is performed before each transmission opportunity, e.g., before transmitting each frame, which may be defined, for example, by another application or entity, e.g., by the buffer transmitter module.
- FIG. 10 is an exemplary sender device 1000 adapted to perform the adaptive and selective transmission process described herein, according to an embodiment of the invention.
- the exemplary sender 1000 typically includes an input/output I/O interface card 1010 adapted to enable the sender 1000 to communicate with the network.
- the sender 1000 also includes a data store 1026 , which may be volatile or non-volatile memory. Such data store may contain the transmission buffer 1024 , which typically temporarily stores content elements ready for transmission to the receiver.
- the data store may also contain cumulative counts 1030 and other information maintained by the sender, e.g., drop set information.
- the buffer transmitter module 1008 of the sender 1000 is adapted to transmit the content elements in the transmission buffer, but typically excluding content elements identified by the selective dropping module 1020 .
- the feedback and rate module 1016 of the sender 1000 receives and processes the feedback reports received from the receivers, as well as perform the rate determination process described herein.
- the selective dropping module 1020 typically using the transmission rate determined by the feedback and rate module 1016 , determines which content elements in the transmission buffer are not to be transmitted by the buffer transmitter module 1008 .
- the device controller 1004 of the sender 1000 is typically adapted to control the overall functions of the sender device 1000 .
- the sender may also include an encoder or codec module 1012 adapted to encode source content, particularly, if the sender also performs the encoding process.
- modules 10 may communicate and interface with each other via a bus, dedicated signal paths or one or more channels 1002 .
- other modules including functions and capabilities, may be added or removed.
- the modules described herein may be further subdivided and combined with other functions so long as the function and processes described herein may be performed.
- the various modules may also be implemented in hardware, software, or both, i.e., firmware.
- FIGS. 11 and 12 illustrate exemplary results of an experiment.
- the GOP format is IBBPBBPBBPBBPBB with each frame respectively assigned priority index 6, 1, 1, 5, 1, 1, 4, 1, 1, 3, 1, 1, 2, 1, and 1.
- FIG. 11 shows a scatter plot 1100 showing the frame priority index versus frame loss GOP PSNR for an exemplary video sequence or source content, with varying number of GOPs.
- the source content contains moving picture sequences or mobile sequence.
- FIG. 12 shows an exemplary graph 1200 of correlation coefficients between frame priority index and frame loss GOP PSNR for intra GOPs and inter GOPs for an exemplary video sequence.
- intra GOP means within one GOP
- inter GOP means between two or three GOPs, as shown, for example, by the curve.
- FIGS. 11 and 12 thus show that the priority index exemplified above may work well considering that the priority index is correlated, may even be construed as highly correlated, to the decrease in PSNR achieved by transmitting the particular frame.
- the distortion caused by not sending a frame may be computed and considered in determining such frame's priority index.
- a client/receiver executes the following steps:
- the AST feature described herein was implemented using Network Simulator (NS 2 ), with some modifications.
- the NS 2 network simulator may be obtained from http://www.isi.edu/nsnam/ns/.
- a number of simulations were performed to evaluate the performance of the adaptive and selective transmission process described herein.
- the performance of the AST process with that of a transrater and also with the combination of transrater and AST (Transrater+APT) were compared.
- FIG. 13 shows the rate-distortion (R-D) performance comparison 1300 for the following scenarios:
- the exemplary source content/video sequence is “mobile” with common intermediate format (CIF) resolution, and 30 frames per second (fps) original sequence. Each rate point on the plot corresponds to a network bandwidth variation scenario.
- the results in FIG. 13 were averaged over different receiver/client buffer sizes and feedback frequency.
- FIG. 14 shows another R-D performance 1400 for different transrater control lag values, e.g., 5 and 9 frames, for:
- the exemplary source content is mobile, with CIF resolution, and 30 fps original sequence.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
TABLE I |
Exemplary RTP Packet Structure/Format |
FIELD: | DESCRIPTION: |
Sender or Source | Source or Transmitter of a stream of RTP packets, also called |
Identifier | Synchronization Source (SSRC) |
(referred in | |
RFC3550 as | |
SSRC) | |
Sequence | The sequence number is typically incremented by one for each RTP |
Number | data packet sent. This sequence number enables a receiver to detect |
packet loss and to restore packet sequence. | |
Timestamp | Sampling Instant. Timestamps may be used to place received video |
or content element packets in the correct timing order. Typically, the | |
timestamp is incremented monotonically and linearly in time to | |
enable synchronization and jitter calculation. In some embodiments, | |
several consecutive RTP packets may have the same timestamp | |
value, if they are logically generated at once, for example, the data | |
packets belong to the same video frame. The sequence numbers of | |
the packets transmitted, however, are still monotonic. | |
TABLE II |
Exemplary RTCP Packet Structure/Format of a Receiver Report |
Field Name: | Description: |
Sender/Source (e.g., | Source or Transmitter identifier (typically the receiver 250) |
Synchronization Source | for the originator of this RR packet. |
(SSRC) of sender) | |
Timestamp | Time when the RR was sent |
Fraction Lost | The fraction of RTP data packets from source SSRC (e.g., the |
sender 210) lost since the previous RR packet was sent. In | |
some embodiments, this fraction may be defined to be the | |
number of packets lost divided by the number of packets | |
expected. An exemplary implementation is disclosed in | |
Appendix A.3 of RFC3550. In some embodiments, if the loss | |
is negative due to duplicates, the fraction lost is set to zero. | |
In some embodiments, a receiver is unable to determine | |
whether any packets are lost after the last one is received, and | |
there may be no reception report block issued for a source if | |
all packets from that source sent during the last reporting | |
interval have been lost. | |
Cumulative Number of | The total number of RTP data packets from the source or |
Packets Lost | transmitter (SSRC) (e.g., the sender) that have been lost since |
the beginning of reception. This number is typically defined | |
to be the number of packets expected less the number of | |
packets actually received, where the number of packets | |
received may include late or duplicate packets. In some | |
embodiments, packets that arrive late are not counted as lost, | |
and the loss value may be negative if there are duplicates. | |
The number of packets expected is typically defined as the | |
extended last sequence number received, less the initial | |
sequence number received. An exemplary calculation is | |
shown in Appendix A.3 of the RFC3550. | |
Extended Highest | The extended highest sequence number received in an RTP |
Sequence Number | data packet from source SSRC. |
Received | |
-
- a) the extended highest sequence number received by the receiver when report m was sent: hs(m) 512;
- a) the fraction or percentage of packets loss as seen by the receiver since the previous report m-1: fr(m) 514; and
- b) the cumulative number of packets loss as seen by the receiver starting from the first feedback report: pl(m) 518.
-
- a) RR(m) indicates the current or most recent RR m;
- b) RR(m-1) indicates the report previous to the RR(m) received by the sender for that receiver;
- c) t′m indicates when RR(m) was sent by the receiver
- d) t′m-1 indicates when associated RR(m-1) was sent by receiver
- e) tm indicates when RR(m) was received by the sender
- f) tm-1 indicates when RR(m-1) was received by the sender
- g) Stm indicates the extended highest sequence number of the RTP packet sent by the sender at time Stm;
- h) C(Stm) indicates the cumulative number of bytes of data payload sent by the sender since the beginning of transmission after sending the RTP packet with sequence number Stm;
- i) R(0) is the default stream rate for that receiver typically determined by being aware or knowing the nominal rate supported by the channel and/or default stream encoding rate; and;
- j) M is a constant which is typically a design parameter. In general, a larger M value means a slower reaction to dynamic network conditions, while a smaller M means a faster reaction to dynamic network conditions;
- k) OT indicates a threshold value, which may be defined, for example, based on an estimate or knowledge about the maximum outstanding/unacknowledged bytes that may be held in various buffers (e.g. sender and receiver network stack buffers).
where φ is defined, for example, to be 0<φ≦1. In general, an exemplary manner in determining a transmission rate is based on considering the difference between the number of packets received between the two reports, the fraction of packets lost by the receiver, and the time interval between the two reports, which may indicate or estimate the current or recent capacity of the receiver or the network congestion. Other calculations or manners of determining transmission rates, such as:
may also be performed. In the next operation, a variable D is calculated based on the difference between the default stream rate and the calculated new transmission rate. D, for example, may be calculated (step 620), with an exemplary equation:
-
- a) the time-stamp of the frame: tj;
- b) the size the of the frame: bj;
- c) the priority or importance index: pj.
-
- a) by Priority or Importance Index order:
S P ={fr(t 1 P ,b 1 P ,p 1 P),fr(t 2 P ,b 2 P ,p 2 P), . . . ,fr(t N P ,b N P ,p N P)|p 1 P ≧p 2 P ≧ . . . ≧p N P} - where SP is typically ordered in a monotonic non-increasing order based on priority or importance index; and
- b) by Timestamp Order:
S T ={fr(t 1 T ,b 1 T ,p 1 T),fr(t 2 T ,b 2 T ,p 2 T), . . . ,fr(t N T ,b N T ,p N T)|t 1 T ≦t 2 T ≦ . . . ≦t N T} - where ST is typically ordered in a monotonic non-decreasing order based on timestamp.
- a) by Priority or Importance Index order:
-
- dj T: delivery deadline for fr(tj, bj, pj);
- Oc=O(m)−[(t-tm)*R(tm)], where R(tm), for example, is one determined in the rate determination process (e.g., See
FIG. 6 ) and t is the current time; and
when RR (m) is the last receiver report received.
may be applied, for example. The set SST(t) thus contains the set of frames in the transmission buffer ordered in timestamp order, where such frames have been determined or calculated to arrive before or at the delivery deadline, (t+esT j)<=dT j, and where R(SST(t))<=R(tm). Furthermore, δ(tj)=1 if all the frames on which fr(tj, bj, pj) depends are included in SST(t).
where α may be defined as 0<α≦1 and Bc is the receiver buffer size in time units or end-to-end delay, e.g., from transmitting from sender to receiving by receiver, and where RR(m) is the last RR received. The receiver buffer size may have been previously exchanged between the sender and the receiver. Referring to
S T(n,N n)={fr(t 1 T ,b 1 T ,p 1 T),fr(t 2 T ,b 2 T ,p 2 T), . . . ,fr(t N
may be applied. See, for example, exemplary table 864 in
Ω(n)=Ω(n−1)∪{∀fr(t y T ,b y T ,p y T)εS T(n,N n)|(t+es y T)>d y T}
and where ST(n,Nn)=SP(n,Nn) and t is the current time frame.
-
- a) The client/receiver receives the media packets/content elements sent by the server/sender. The media packets are typically sent from the server in the stream order, e.g., sequence number order, thus the only possible out-of-order delivery may happen due to the network elements. The client looks to the sequence number of each RTP packet to determine if any reordering is required. If the packet is received before its deadline, the client decodes and presents the packets.
- b) The client periodically sends feedback reports, e.g., RTCP RRs, to the server.
-
- a) Applying AST only,
- b) Applying Transrating only and
- c) Applying Transrating+AST.
-
- a) Transrating only and
- b) Transrating+AST.
Claims (25)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/560,457 US7953880B2 (en) | 2006-11-16 | 2006-11-16 | Content-aware adaptive packet transmission |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/560,457 US7953880B2 (en) | 2006-11-16 | 2006-11-16 | Content-aware adaptive packet transmission |
Publications (2)
Publication Number | Publication Date |
---|---|
US20080120424A1 US20080120424A1 (en) | 2008-05-22 |
US7953880B2 true US7953880B2 (en) | 2011-05-31 |
Family
ID=39471640
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/560,457 Expired - Fee Related US7953880B2 (en) | 2006-11-16 | 2006-11-16 | Content-aware adaptive packet transmission |
Country Status (1)
Country | Link |
---|---|
US (1) | US7953880B2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070263818A1 (en) * | 2006-03-31 | 2007-11-15 | Fujitsu Limited | Relay apparatus, relay method, relay program, and communication system |
US20080181302A1 (en) * | 2007-01-25 | 2008-07-31 | Mehmet Umut Demircin | Methods and Systems for Rate-Adaptive Transmission of Video |
US20100268836A1 (en) * | 2009-03-16 | 2010-10-21 | Dilithium Holdings, Inc. | Method and apparatus for delivery of adapted media |
US20130340023A1 (en) * | 2012-06-15 | 2013-12-19 | Samsung Electronics Co., Ltd. | Reception device and program for reception device |
US9769281B2 (en) | 2011-12-19 | 2017-09-19 | Google Technology Holdings LLC | Method and apparatus for determining a multimedia representation for a multimedia asset delivered to a client device |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050114474A1 (en) * | 2003-11-20 | 2005-05-26 | International Business Machines Corporation | Automatic configuration of the network devices via connection to specific switch ports |
US7724750B2 (en) * | 2004-04-01 | 2010-05-25 | Nokia Corporation | Expedited data transmission in packet based network |
US7953880B2 (en) * | 2006-11-16 | 2011-05-31 | Sharp Laboratories Of America, Inc. | Content-aware adaptive packet transmission |
FR2916600B1 (en) * | 2007-05-24 | 2013-11-22 | Canon Kk | METHOD AND DEVICE FOR DATA TRANSMISSION |
US7991904B2 (en) * | 2007-07-10 | 2011-08-02 | Bytemobile, Inc. | Adaptive bitrate management for streaming media over packet networks |
US7987285B2 (en) | 2007-07-10 | 2011-07-26 | Bytemobile, Inc. | Adaptive bitrate management for streaming media over packet networks |
US20090204812A1 (en) * | 2008-02-13 | 2009-08-13 | Baker Todd M | Media processing |
US8775665B2 (en) * | 2009-02-09 | 2014-07-08 | Citrix Systems, Inc. | Method for controlling download rate of real-time streaming as needed by media player |
US9197677B2 (en) * | 2009-03-09 | 2015-11-24 | Arris Canada, Inc. | Multi-tiered scalable media streaming systems and methods |
US9485299B2 (en) * | 2009-03-09 | 2016-11-01 | Arris Canada, Inc. | Progressive download gateway |
CN101924603B (en) * | 2009-06-09 | 2014-08-20 | 华为技术有限公司 | Self-adaption adjusting method, device and system of data transmission rate |
US8566393B2 (en) * | 2009-08-10 | 2013-10-22 | Seawell Networks Inc. | Methods and systems for scalable video chunking |
GB0921831D0 (en) | 2009-12-14 | 2010-01-27 | British Telecomm | Graphical data delivery |
GB201000738D0 (en) | 2010-01-18 | 2010-03-03 | British Telecomm | Graphical data processing |
KR20110090596A (en) * | 2010-02-04 | 2011-08-10 | 삼성전자주식회사 | Method and apparatus for correcting interarrival jitter |
US8190677B2 (en) * | 2010-07-23 | 2012-05-29 | Seawell Networks Inc. | Methods and systems for scalable video delivery |
CN102469072A (en) | 2010-11-08 | 2012-05-23 | 华为技术有限公司 | Streaming media service method and system, and client |
US9253105B2 (en) | 2010-12-30 | 2016-02-02 | Nokia Technologies Oy | Methods and apparatuses for facilitating determination of a state of a receiver buffer |
CN103828324B (en) | 2011-06-10 | 2017-11-28 | 茨特里克斯系统公司 | Method, apparatus and system for adaptive bitrate management |
US9288251B2 (en) | 2011-06-10 | 2016-03-15 | Citrix Systems, Inc. | Adaptive bitrate management on progressive download with indexed media files |
US8909764B2 (en) * | 2011-07-28 | 2014-12-09 | Xyratex Technology Limited | Data communication method and apparatus |
KR101804516B1 (en) * | 2011-08-31 | 2017-12-07 | 삼성전자주식회사 | Broadcast receiving device and method |
US9712887B2 (en) | 2012-04-12 | 2017-07-18 | Arris Canada, Inc. | Methods and systems for real-time transmuxing of streaming media content |
US20130282868A1 (en) * | 2012-04-23 | 2013-10-24 | Jae Chung | Integral Controller Based Pacing for HTTP Pseudo-streaming |
KR101917766B1 (en) * | 2012-05-25 | 2018-11-13 | 삼성전자주식회사 | Method and apparatus for controlling congestion in a wireless communication system |
US9106887B1 (en) * | 2014-03-13 | 2015-08-11 | Wowza Media Systems, LLC | Adjusting encoding parameters at a mobile device based on a change in available network bandwidth |
US20150373075A1 (en) * | 2014-06-23 | 2015-12-24 | Radia Perlman | Multiple network transport sessions to provide context adaptive video streaming |
KR102196492B1 (en) * | 2014-07-04 | 2020-12-30 | 삼성전자주식회사 | Method and apparatus for transmitting and receiving data in a communication system |
US10433004B2 (en) * | 2015-04-13 | 2019-10-01 | Sony Semiconductor Solutions Corporation | Reception apparatus, receiving method, and program |
CN107835133B (en) * | 2017-06-16 | 2021-10-22 | 南京航空航天大学 | Stream priority control method based on multi-attribute decision |
US10523979B2 (en) * | 2017-12-21 | 2019-12-31 | Vyu Labs, Inc. | Streaming live video |
US10349059B1 (en) | 2018-07-17 | 2019-07-09 | Wowza Media Systems, LLC | Adjusting encoding frame size based on available network bandwidth |
NO346434B1 (en) * | 2021-11-18 | 2022-08-08 | Pexip AS | Method, system and computer program product for initiating downspeeding in a videoconferencing session |
Citations (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5392280A (en) | 1994-04-07 | 1995-02-21 | Mitsubishi Electric Research Laboratories, Inc. | Data transmission system and scheduling protocol for connection-oriented packet or cell switching networks |
JPH08256340A (en) | 1989-09-27 | 1996-10-01 | Sony Corp | Video signal transmitting method and video signal transmitter |
JPH09312656A (en) | 1996-03-21 | 1997-12-02 | Sony Corp | Transmitter and method therefor |
US5802311A (en) | 1995-06-15 | 1998-09-01 | David Hall | Using adaptive redundant packet retrieval to improve reliability in a unidirectional data broadcasting system |
US6031584A (en) | 1997-09-26 | 2000-02-29 | Intel Corporation | Method for reducing digital video frame frequency while maintaining temporal smoothness |
US6157674A (en) | 1996-03-21 | 2000-12-05 | Sony Corporation | Audio and video data transmitting apparatus, system, and method thereof |
US6351471B1 (en) | 1998-01-14 | 2002-02-26 | Skystream Networks Inc. | Brandwidth optimization of video program bearing transport streams |
JP2002094533A (en) | 2000-07-06 | 2002-03-29 | Matsushita Electric Ind Co Ltd | System for scheduling transfer of variable bit rate data in network and its relating method |
US20020093930A1 (en) | 2001-01-16 | 2002-07-18 | Dertz Gregory A. | Method and apparatus for organizing and scheduling multimedia data transfers over a wireless channel |
US6498865B1 (en) | 1999-02-11 | 2002-12-24 | Packetvideo Corp,. | Method and device for control and compatible delivery of digitally compressed visual data in a heterogeneous communication network |
WO2003041055A1 (en) | 2001-11-02 | 2003-05-15 | The Regents Of The University Of California | End-to-end distortion estimation for streaming of pre-compressed video |
US6609149B1 (en) | 1999-04-12 | 2003-08-19 | International Business Machines Corporation | Method and apparatus for prioritizing video frame retrieval in a shared disk cluster |
US20030233464A1 (en) | 2002-06-10 | 2003-12-18 | Jonathan Walpole | Priority progress streaming for quality-adaptive transmission of data |
US20030236904A1 (en) | 2002-06-19 | 2003-12-25 | Jonathan Walpole | Priority progress multicast streaming for quality-adaptive transmission of data |
WO2004010250A2 (en) | 2002-07-18 | 2004-01-29 | Koninklijke Philips Electronics N.V. | Adaptive dropping of prioritized transmission packets |
US6747991B1 (en) | 2000-04-26 | 2004-06-08 | Carnegie Mellon University | Filter and method for adaptively modifying the bit rate of synchronized video and audio streams to meet packet-switched network bandwidth constraints |
EP1432183A2 (en) | 2002-12-16 | 2004-06-23 | Electronics and Telecommunications Research Institute | System and method for downlink packet scheduling in a mobile communication system, and recording medium storing a program thereof |
WO2004056057A2 (en) | 2002-12-13 | 2004-07-01 | General Instrument Corporation | Method and apparatus for providing a buffer architecture to improve presentation quality of images |
US20040136409A1 (en) | 1998-01-14 | 2004-07-15 | Robert Robinett | Video remultiplexer for dynamic remultiplexing, multi-mode operation and jitter reduced asynchronous communication |
US20040194142A1 (en) | 1999-12-22 | 2004-09-30 | Zhimei Jiang | Method and system for adaptive transmission of smoothed data over wireless channels |
WO2004088858A2 (en) | 2003-03-29 | 2004-10-14 | Regents Of University Of California | Method and apparatus for improved data transmission |
US20050013303A1 (en) * | 2003-07-16 | 2005-01-20 | Nandu Gopalakrishnan | Method of transmitting or retransmitting packets in a communication system |
WO2005029867A1 (en) | 2003-09-23 | 2005-03-31 | Koninklijke Philips Electronics, N.V. | System and method for providing video content and concealment dependent error protection and scheduling algorithms |
US20050071876A1 (en) | 2003-09-30 | 2005-03-31 | Van Beek Petrus J. L. | Wireless video transmission system |
WO2005048606A1 (en) | 2003-11-13 | 2005-05-26 | Koninklijke Philips Electronics, N.V. | Method and apparatus for smoothing overall quality of video transported over a wireless medium |
US20050169312A1 (en) | 2004-01-30 | 2005-08-04 | Jakov Cakareski | Methods and systems that use information about a frame of video data to make a decision about sending the frame |
WO2005101755A1 (en) | 2004-04-13 | 2005-10-27 | Orange Sa | Priority based multiplexing of data packet transport |
WO2005112449A1 (en) | 2004-04-30 | 2005-11-24 | Worldgate Service, Inc. | Adaptive video telephone system |
EP1619839A1 (en) | 2004-07-21 | 2006-01-25 | Siemens Mobile Communications S.p.A. | Method of and apparatus for scheduling transmission of multimedia streaming services over the radio channel of wireless communication systems |
US20060026294A1 (en) | 2004-07-29 | 2006-02-02 | Microsoft Corporation | Media transrating over a bandwidth-limited network |
US6996061B2 (en) * | 2000-08-11 | 2006-02-07 | Industrial Technology Research Institute | Dynamic scheduling for packet data network |
US20060095944A1 (en) | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Sender-side bandwidth estimation for video transmission with receiver packet buffer |
US20060095943A1 (en) | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Packet scheduling for video transmission with sender queue control |
WO2006061801A1 (en) | 2004-12-10 | 2006-06-15 | Koninklijke Philips Electronics, N.V. | Wireless video streaming using single layer coding and prioritized streaming |
WO2006064454A1 (en) | 2004-12-15 | 2006-06-22 | Koninklijke Philips Electronics N.V. | Method and system for the transmission of digital video over a wireless network |
WO2006067374A1 (en) | 2004-12-22 | 2006-06-29 | British Telecommunications Public Limited Company | Rate control with buffer underflow prevention |
US20060153217A1 (en) | 2001-11-14 | 2006-07-13 | Microsoft Corporation | Intelligent buffering process for network conference video |
US20060165088A1 (en) | 2002-11-27 | 2006-07-27 | Rgb Networks, Inc. | Method and apparatus for time-multiplexed processing of multiple digital video programs |
US20070058557A1 (en) | 2005-09-15 | 2007-03-15 | Interdigital Technology Corporation | Method and apparatus for scheduling data transmissions based on a traffic data pattern model |
US7263064B2 (en) * | 2001-03-06 | 2007-08-28 | Ntt Docomo, Inc. | Packet communication system with packets classification, division and retransmission |
US20080120424A1 (en) * | 2006-11-16 | 2008-05-22 | Deshpande Sachin G | Content-aware adaptive packet transmission |
-
2006
- 2006-11-16 US US11/560,457 patent/US7953880B2/en not_active Expired - Fee Related
Patent Citations (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08256340A (en) | 1989-09-27 | 1996-10-01 | Sony Corp | Video signal transmitting method and video signal transmitter |
US5392280A (en) | 1994-04-07 | 1995-02-21 | Mitsubishi Electric Research Laboratories, Inc. | Data transmission system and scheduling protocol for connection-oriented packet or cell switching networks |
US5802311A (en) | 1995-06-15 | 1998-09-01 | David Hall | Using adaptive redundant packet retrieval to improve reliability in a unidirectional data broadcasting system |
JPH09312656A (en) | 1996-03-21 | 1997-12-02 | Sony Corp | Transmitter and method therefor |
US6157674A (en) | 1996-03-21 | 2000-12-05 | Sony Corporation | Audio and video data transmitting apparatus, system, and method thereof |
US6031584A (en) | 1997-09-26 | 2000-02-29 | Intel Corporation | Method for reducing digital video frame frequency while maintaining temporal smoothness |
US6351471B1 (en) | 1998-01-14 | 2002-02-26 | Skystream Networks Inc. | Brandwidth optimization of video program bearing transport streams |
US20050105486A1 (en) | 1998-01-14 | 2005-05-19 | Robert Robinett | Bandwidth optimization of video program bearing transport streams |
US20040136409A1 (en) | 1998-01-14 | 2004-07-15 | Robert Robinett | Video remultiplexer for dynamic remultiplexing, multi-mode operation and jitter reduced asynchronous communication |
US20020131443A1 (en) | 1998-01-14 | 2002-09-19 | Robert Robinett | Bandwidth optimization of video program bearing transport streams |
US6498865B1 (en) | 1999-02-11 | 2002-12-24 | Packetvideo Corp,. | Method and device for control and compatible delivery of digitally compressed visual data in a heterogeneous communication network |
US6609149B1 (en) | 1999-04-12 | 2003-08-19 | International Business Machines Corporation | Method and apparatus for prioritizing video frame retrieval in a shared disk cluster |
US20040194142A1 (en) | 1999-12-22 | 2004-09-30 | Zhimei Jiang | Method and system for adaptive transmission of smoothed data over wireless channels |
US6747991B1 (en) | 2000-04-26 | 2004-06-08 | Carnegie Mellon University | Filter and method for adaptively modifying the bit rate of synchronized video and audio streams to meet packet-switched network bandwidth constraints |
JP2002094533A (en) | 2000-07-06 | 2002-03-29 | Matsushita Electric Ind Co Ltd | System for scheduling transfer of variable bit rate data in network and its relating method |
US6996061B2 (en) * | 2000-08-11 | 2006-02-07 | Industrial Technology Research Institute | Dynamic scheduling for packet data network |
US20020093930A1 (en) | 2001-01-16 | 2002-07-18 | Dertz Gregory A. | Method and apparatus for organizing and scheduling multimedia data transfers over a wireless channel |
US7002985B2 (en) | 2001-01-16 | 2006-02-21 | Motorola, Inc. | Method and apparatus for organizing and scheduling multimedia data transfers over a wireless channel |
US7263064B2 (en) * | 2001-03-06 | 2007-08-28 | Ntt Docomo, Inc. | Packet communication system with packets classification, division and retransmission |
WO2003041055A1 (en) | 2001-11-02 | 2003-05-15 | The Regents Of The University Of California | End-to-end distortion estimation for streaming of pre-compressed video |
US20060153217A1 (en) | 2001-11-14 | 2006-07-13 | Microsoft Corporation | Intelligent buffering process for network conference video |
US20030233464A1 (en) | 2002-06-10 | 2003-12-18 | Jonathan Walpole | Priority progress streaming for quality-adaptive transmission of data |
US20030236904A1 (en) | 2002-06-19 | 2003-12-25 | Jonathan Walpole | Priority progress multicast streaming for quality-adaptive transmission of data |
US20060164987A1 (en) | 2002-07-18 | 2006-07-27 | Carles Ruiz Floriach | Adaptive dropping of prioritized transmission packets |
WO2004010250A2 (en) | 2002-07-18 | 2004-01-29 | Koninklijke Philips Electronics N.V. | Adaptive dropping of prioritized transmission packets |
US20060165088A1 (en) | 2002-11-27 | 2006-07-27 | Rgb Networks, Inc. | Method and apparatus for time-multiplexed processing of multiple digital video programs |
US20040125816A1 (en) | 2002-12-13 | 2004-07-01 | Haifeng Xu | Method and apparatus for providing a buffer architecture to improve presentation quality of images |
WO2004056057A2 (en) | 2002-12-13 | 2004-07-01 | General Instrument Corporation | Method and apparatus for providing a buffer architecture to improve presentation quality of images |
EP1432183A2 (en) | 2002-12-16 | 2004-06-23 | Electronics and Telecommunications Research Institute | System and method for downlink packet scheduling in a mobile communication system, and recording medium storing a program thereof |
WO2004088858A2 (en) | 2003-03-29 | 2004-10-14 | Regents Of University Of California | Method and apparatus for improved data transmission |
US20050013303A1 (en) * | 2003-07-16 | 2005-01-20 | Nandu Gopalakrishnan | Method of transmitting or retransmitting packets in a communication system |
US7385954B2 (en) * | 2003-07-16 | 2008-06-10 | Lucent Technologies Inc. | Method of transmitting or retransmitting packets in a communication system |
WO2005029867A1 (en) | 2003-09-23 | 2005-03-31 | Koninklijke Philips Electronics, N.V. | System and method for providing video content and concealment dependent error protection and scheduling algorithms |
US20050071876A1 (en) | 2003-09-30 | 2005-03-31 | Van Beek Petrus J. L. | Wireless video transmission system |
WO2005048606A1 (en) | 2003-11-13 | 2005-05-26 | Koninklijke Philips Electronics, N.V. | Method and apparatus for smoothing overall quality of video transported over a wireless medium |
US20050169312A1 (en) | 2004-01-30 | 2005-08-04 | Jakov Cakareski | Methods and systems that use information about a frame of video data to make a decision about sending the frame |
WO2005101755A1 (en) | 2004-04-13 | 2005-10-27 | Orange Sa | Priority based multiplexing of data packet transport |
WO2005112449A1 (en) | 2004-04-30 | 2005-11-24 | Worldgate Service, Inc. | Adaptive video telephone system |
EP1619839A1 (en) | 2004-07-21 | 2006-01-25 | Siemens Mobile Communications S.p.A. | Method of and apparatus for scheduling transmission of multimedia streaming services over the radio channel of wireless communication systems |
US20060026294A1 (en) | 2004-07-29 | 2006-02-02 | Microsoft Corporation | Media transrating over a bandwidth-limited network |
US20060095943A1 (en) | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Packet scheduling for video transmission with sender queue control |
US20060095944A1 (en) | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Sender-side bandwidth estimation for video transmission with receiver packet buffer |
WO2006061801A1 (en) | 2004-12-10 | 2006-06-15 | Koninklijke Philips Electronics, N.V. | Wireless video streaming using single layer coding and prioritized streaming |
WO2006064454A1 (en) | 2004-12-15 | 2006-06-22 | Koninklijke Philips Electronics N.V. | Method and system for the transmission of digital video over a wireless network |
WO2006067374A1 (en) | 2004-12-22 | 2006-06-29 | British Telecommunications Public Limited Company | Rate control with buffer underflow prevention |
US20070058557A1 (en) | 2005-09-15 | 2007-03-15 | Interdigital Technology Corporation | Method and apparatus for scheduling data transmissions based on a traffic data pattern model |
US20080120424A1 (en) * | 2006-11-16 | 2008-05-22 | Deshpande Sachin G | Content-aware adaptive packet transmission |
Non-Patent Citations (15)
Title |
---|
Chakareski, Jacob; Apostolopoulos, John; Girod, Bernd,"Low-Complexity Rate-Distortion Optimized Video Streaming,"ICIP 2004. |
Chou, Philip A., Miao, Zhouron,"Rate-Distortion Optimized Streaming of packetized Media," Technical Report MSR-TR-2001-35, Feb. 2001, Microsoft Research, Redding, CA. |
Huang, Jie;Krasic, Charles;Walpole, Jonathan;Feng, Wu-Chi, "Adaptive Live Video Streaming by Priority Drop," OGI School of Science and Engineering, Packet Video 2003. |
Kalman, M.,Steinbach,E.& Girod,B.,"Adaptive Media Playout for Low-Delay Video Streaming Over Error-Prone Channels,"IEEE Transactions, Jun. 2004,pp. 841-851,vol. 14,No. 6. |
Kalman, Mark and Girod, Bernd, "Rate-Distortion Optimized Video Streaming with Multiple Deadlines for Low Latency Applications,"Dec. 2004, Packet Video Workshop, Irvine,CA. |
Kalman,Mark;Girod,Bernd; Van Beek, Peter,"Optimized Transcoding Rate Selection and Packet scheduling for Transmitting Multiple Video Streams Over a Shared Channel,"ICIP 2005. |
Non-Final Office action for U.S. Appl. No. 11/738,313 dated Jun. 3, 2009. |
Notice of Allowance for U.S. Appl. No. 11/738,313 dated Dec. 10, 2009. |
Notice of Allowance for U.S. Appl. No. 11/743,547 dated Oct. 6, 2009. |
Schulzrinne,H. et al.,"RTP:A Transport Protocol for Real-Time Applications,"RFC3550, [online], [retrieved on Jun. 2, 2006].Retrieved from. |
Schulzrinne,H. et al.,"RTP:A Transport Protocol for Real-Time Applications,"RFC3550, [online], [retrieved on Jun. 2, 2006].Retrieved from<URL:http://www.ietf.org/rfc/rfc3550.txt>. |
Sehgal, Anshul,Verscheure,Oliver and Frossard,Pascal,"Distortion-Buffer Optimized TCP Video Streaming,"IEEE ICP, Oct. 2004,Singapore. |
Sehgal,Anshul,Jagmohan,Ashish,Verscheure,Oliver and Frossard,Pascal,"Fast Distortion-Buffer Optimized Streaming of Multimedia,"IEEE ICP, Sep. 2005, Genoa,Italy. |
The ns Manual (formerly ns Notes and Documentation), The VINT Project, [online], [retreived on Jun. 2, 2006]. Retrieved from the Internet. |
The ns Manual (formerly ns Notes and Documentation), The VINT Project, [online], [retreived on Jun. 2, 2006]. Retrieved from the Internet<URL:http://www.isi.edu/nsnam/ns/>. |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070263818A1 (en) * | 2006-03-31 | 2007-11-15 | Fujitsu Limited | Relay apparatus, relay method, relay program, and communication system |
US8229087B2 (en) * | 2006-03-31 | 2012-07-24 | Fujitsu Limited | Relay apparatus, relay method, relay program, and communication system |
US20080181302A1 (en) * | 2007-01-25 | 2008-07-31 | Mehmet Umut Demircin | Methods and Systems for Rate-Adaptive Transmission of Video |
US8243789B2 (en) * | 2007-01-25 | 2012-08-14 | Sharp Laboratories Of America, Inc. | Methods and systems for rate-adaptive transmission of video |
US20100268836A1 (en) * | 2009-03-16 | 2010-10-21 | Dilithium Holdings, Inc. | Method and apparatus for delivery of adapted media |
US8838824B2 (en) * | 2009-03-16 | 2014-09-16 | Onmobile Global Limited | Method and apparatus for delivery of adapted media |
US9769281B2 (en) | 2011-12-19 | 2017-09-19 | Google Technology Holdings LLC | Method and apparatus for determining a multimedia representation for a multimedia asset delivered to a client device |
US20170353578A1 (en) * | 2011-12-19 | 2017-12-07 | Google Technology Holdings LLC | Method and apparatus for determining a multimedia representation for a multimedia asset delivered to a client device |
US10547706B2 (en) * | 2011-12-19 | 2020-01-28 | Google Technology Holdings LLC | Method and apparatus for determining a multimedia representation for a multimedia asset delivered to a client device |
US20130340023A1 (en) * | 2012-06-15 | 2013-12-19 | Samsung Electronics Co., Ltd. | Reception device and program for reception device |
US9191696B2 (en) * | 2012-06-15 | 2015-11-17 | Samsung Electronics Co., Ltd. | Reception device and program for reception device |
Also Published As
Publication number | Publication date |
---|---|
US20080120424A1 (en) | 2008-05-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7953880B2 (en) | Content-aware adaptive packet transmission | |
US7706384B2 (en) | Packet scheduling with quality-aware frame dropping for video streaming | |
US7668170B2 (en) | Adaptive packet transmission with explicit deadline adjustment | |
US9544602B2 (en) | Wireless video transmission system | |
EP2061174B1 (en) | Data communication system, data transmitting device and method, using probe packets and having a transmission buffer control | |
US7797723B2 (en) | Packet scheduling for video transmission with sender queue control | |
US7784076B2 (en) | Sender-side bandwidth estimation for video transmission with receiver packet buffer | |
US7860005B2 (en) | Methods and systems that use information about a frame of video data to make a decision about sending the frame | |
US8356327B2 (en) | Wireless video transmission system | |
CN103828324B (en) | Method, apparatus and system for adaptive bitrate management | |
US8718094B2 (en) | Network optimized distribution | |
US7652993B2 (en) | Multi-stream pro-active rate adaptation for robust video transmission | |
US20090103635A1 (en) | System and method of unequal error protection with hybrid arq/fec for video streaming over wireless local area networks | |
US20150101003A1 (en) | Data transmission apparatus, system and method | |
US20130263201A1 (en) | Data transmission apparatus, system and method | |
CN102742245A (en) | A method and apparatus for parsing a network abstraction-layer for reliable data communication | |
Setton et al. | Peer-to-peer video streaming | |
US9258347B2 (en) | Encoding of a video frame for transmission to a plurality of clients | |
US20140115100A1 (en) | Video packet scheduling method for multimedia streaming | |
Freris et al. | Resource allocation for multihomed scalable video streaming to multiple clients | |
Van Beek et al. | Delay-constrained rate adaptation for robust video transmission over home networks | |
Argyriou | Real-time and rate–distortion optimized video streaming with TCP | |
Huszák et al. | Selective retransmission of MPEG video streams over IP networks | |
Kropfberger et al. | Evaluation of RTP immediate feedback and retransmission extensions [video streaming applications] | |
Futemma et al. | TFRC-based rate control scheme for real-time JPEG 2000 video transmission |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SHARP LABORATORIES OF AMERICA, INC., WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DESHPANDE, SACHIN GOVIND;REEL/FRAME:018526/0625 Effective date: 20061109 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SHARP KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHARP LABORATORIES OF AMERICA INC.;REEL/FRAME:027008/0149 Effective date: 20111003 |
|
AS | Assignment |
Owner name: INTELLECTUALS HIGH-TECH KFT., HUNGARY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHARP KABUSHIKI KAISHA;REEL/FRAME:030089/0732 Effective date: 20121216 |
|
AS | Assignment |
Owner name: PATOIS TELECOM LIMITED LIABILITY COMPANY, DELAWARE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTELLECTUALS HIGH-TECH KFT.;REEL/FRAME:030109/0004 Effective date: 20121217 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20230531 |