AU2012202123A1 - System and method for management of scalability information in scalable video and audio coding systems using control messages - Google Patents

System and method for management of scalability information in scalable video and audio coding systems using control messages Download PDF

Info

Publication number
AU2012202123A1
AU2012202123A1 AU2012202123A AU2012202123A AU2012202123A1 AU 2012202123 A1 AU2012202123 A1 AU 2012202123A1 AU 2012202123 A AU2012202123 A AU 2012202123A AU 2012202123 A AU2012202123 A AU 2012202123A AU 2012202123 A1 AU2012202123 A1 AU 2012202123A1
Authority
AU
Australia
Prior art keywords
information
layer
media
transmitted
initializing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
AU2012202123A
Other versions
AU2012202123B2 (en
Inventor
Alexandros Eleftheriadis
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vidyo Inc
Original Assignee
Vidyo Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US60/786,178 priority Critical
Priority to US60/786,997 priority
Priority to US60/789,550 priority
Priority to USPCT/US2006/0028367 priority
Priority to USPCT/US2006/0028365 priority
Priority to USPTC/US06/028368 priority
Priority to AUPCT/US06/028366 priority
Priority to USPCT/US2006/0061815 priority
Priority to USPCT/US2006/0062569 priority
Priority to US60/884,148 priority
Priority to USPCT/US2006/0062357 priority
Priority to AU2007230602A priority patent/AU2007230602B2/en
Priority to AU2012202123A priority patent/AU2012202123B2/en
Application filed by Vidyo Inc filed Critical Vidyo Inc
Publication of AU2012202123A1 publication Critical patent/AU2012202123A1/en
Application granted granted Critical
Publication of AU2012202123B2 publication Critical patent/AU2012202123B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234327Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/633Control signals issued by server directed to the network components or client
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/637Control signals issued by the client directed to the server or network components
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/643Communication protocols
    • H04N21/6437Real-time Transport Protocol [RTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845Structuring of content, e.g. decomposing content into time segments
    • H04N21/8451Structuring of content, e.g. decomposing content into time segments using Advanced Video Coding [AVC]

Abstract

C.NkPonblDCC\XM 42AW,24_1 DOC-12/04/2m12 Systems and methods are provided for communicating timely information related to the scalability layer structure of signals received by decoders and other components in a video 5 and/or audio communication system. For a communication system, which uses the Standard H.264 SVC coding format, standard SSEI messages are modified or supplemented to include the ability to signal scalability layer structure information and changes thereof. Recipients can use the signal scalability layer information to properly process or decode received signals. WO 2007/112384 PCTIUS2007/065003 SVC TRANSMISSION SYSTEM 100 MEDIA - SIGNALING SIGNAUNG VIDEO/AUDIO 150 VIDEO/AUDIO SERVER OR RECEIVER SUBSTITUTE SHEET (RULE 26)

Description

A ustralian Patents Act 1990 - Regulation 3.2 ORIGINAL COMPLETE SPECIFICATION STANDARD PATENT Invention Title "System and method for management of scalability information in scalable video and audio coding systems using control messages" The following statement is a full description of this invention, including the best method of performing it known to me/us: P1/00/01 I C\NRPonbl\DCCMKXM\42(624_ 1 DOC-12/0420 12 -2 Cross Reference to Related Application The disclosure of the complete specification of Australian Patent Application No. 2007230602, as originally filed and as accepted, is incorporated herein by reference. 5 Field of the Invention The present invention relates to video communication systems. In particular, the invention relates to techniques for communicating the structure of a scalable video coded bit stream, 10 and changes thereto, in a video communication system in a way that reduces both the bit rate and the computational complexity. Background of the Invention 15 New digital video and audio "scalable" coding techniques, which are directed to general improvements in coding efficiency, have a number of new structural characteristics. Specifically, an important new characteristic is scalability. In scalable coding, an original or source signal is represented using two or more hierarchically structured bitstreams. The hierarchical structure implies that decoding of a given bitstream depends on the availability 20 of some or all other bitstreams that are lower in hierarchy. Each bitstream, together with the bitstreams it depends on, offer a representation of the original signal at a particular temporal, fidelity (e.g., in terms of signal-to-noise ratio (SNR)), or spatial resolution (for video). 25 It is understood that term 'scalable' does not refer to magnitude or scale in terms of numbers, but rather to the ability of the encoding technique to offer a set of different bitstreams corresponding to efficient representations of' the original or source signal at different 'scales' of resolutions or other qualities in general. The forthcoming ITU-T H-.264 Annex F specification, which is referred to as Scalable Video Coding (SVC)), is an 30 example of a video coding standard that offers video coding scalability in all of' temporal, spatial, and fidelity dimensions. SVC is an extension of the 1-1.264 standard (also known as Advanced Video Coding (AVC)). An example of an earlier standard, which also offered all three types of scalability, WO 2007/112384 PCT/US2007/065003 is ISO MPEG-2 (also published as ITU-T H.262). ITU G.729.1 (also known as G.729EV) is an example of a standard offering scalable audio coding. 10005] Scalability was introduced in video and audio coding as a solution to distribution problems in streaming and broadcasting, and with a view to allow a given communication system to operate with varying access networks (e.g., clients connected with different bandwidths), network conditions (e.g., bandwidth fluctuation), and client devices (e.g., a personal computer that uses a large monitor vs. a handheld device with a much smaller screen). [00061 Scalable video coding techniques, which are specifically designed for interactive video communication applications such as videoconferencing, are described in commonly assigned International patent application PCT/US06/028365. Further, commonly assigned International patent application PCT/US06/028365 describes the design of a new type of server, called Scalable Video Communication Server (SVCS). SVCS can advantageously use scalable coded video for high-quality and low-delay video communication and has a complexity, which is significantly reduced compared to traditional switching or transcoding Multipoint Control Units (MCUs). Similarly, commonly assigned International patent application PCT/US06/62569 describes a Compositing Scalable Video Coding Server (CSVCS), which has the same benefits as an SVCS but produces a single coded output bit stream. The scalable video coding design and the SVCS/CSVCS architecture can be used in further advantageous ways, which are described, for example, in commonly assigned International patent applications PCT/US06/028367, PCT/US06/027368, PCT/US06/061815, PCTIUS07/62357, and PCT/US07/63335. These applications describe use of scalable coding techniques and SVCS/CVCS architecture for effective trunking 3 WO 2007/112384 PCT/US2007/065003 between servers, reduced jitter buffer delay, error resilience and random access, "thinning" of scalable video bitstreams to improve coding efficiency with reduced packet loss, and rate control, respectively. Further, commonly assigned U.S. Provisional Patent Application Serial No. 60/786,997 described techniques for transcoding between scalable video coding formats and other formats, whereas commonly assigned U.S. Provisional Patent Application Serial No. 60/884,148 describes further improvements in error resilience in video communication systems that use scalable video coding. 100071 Consideration is now being given to improved video and audio communication systems that use scalable video or audio coding. In particular, with a view of improving such systems, attention is directed toward managing the scalability information communicated from a source of a video or audio bit stream to a recipient, either directly or through one or more servers. The source may be a transmitting endpoint that encodes and transmits live video over a communication network, a streaming server that transmits pre-coded video, or a software module that provides access to a file stored in a mass storage or other access device. Similarly, the recipient may be a receiving endpoint that obtains the coded video or audio bit stream over a communication network, or directly from a mass storage or other access device. SUMMARY OF THE INVENTION 100081 Systems and methods are provided for improved management of scalability information in a video and audio communication system that uses scalable coding. [0009] The systems and methods communicate the structure of a scalable coded bit stream, as well as changes that may occur over time, to system components. 4 WO 2007/112384 PCT/US2007/065003 The communication techniques utilize differential encoding of the relevant information and advantageously achieve reduced bit rate overhead and reduced computational complexity. [00101 In a preferred embodiment, for a video communication system which uses the Standard H.264 SVC coding format, the standard scalability information SEI messages are modified and also supplemented to include the ability to signal scalability information changes. The expanded set of scalability information SEI messages can inform recipients (e.g., servers, decoder/endpoints) of the scalability layer structure of transmitted signals and changes thereto. The recipients can use the scalability layer information to properly process or decode received signals. BRIEF DESCRIPTION OF THE DRAWINGS 100111 Further features, the nature, and various advantages of the invention will be more apparent from the following detailed description of the preferred embodiments and the accompanying drawings in which: 100121 FIG. I is a block diagram illustrating an exemplary architecture of an SVC transmission system in accordance with the principles of the present invention; 100131 FIG. 2 is a flow diagram of the scalability information state management process, in accordance with the principles of the present invention; 100141 FIG. 3 is a flow diagram of the scalability information state management process for an alternative embodiment, in accordance with the principles of the present invention. [0015] Throughout the figures the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the present invention 5 WO 2007/112384 PCT/US2007/065003 will now be described in detail with reference to the Figures, it is done so in connection with the illustrative embodiments. DETAILED DESCRIPTION OF THE INVENTION [00161 Systems and methods are provided for improved management of scalability information in a video and audio communication system that uses scalable coding. [00171 FIG. 1 shows an exemplary architecture of a communication system 100 which uses scalable coding. Communication system 100 includes a media server or encoder 110 (e.g., a streaming server or a transmitting endpoint), which communicates video and/or audio signals with a client/receiver 120 over a network 130 through a media gateway 140. [00181 For brevity, the description herein is limited to the video portion of such a communication system. It will be understood, however, that systems and methods described herein for the video portion can be used for the scalable audio portions, with the understanding that no spatial scalability dimension can be provided to an audio signal, but multi-channel coding may additionally be used in audio signal coding. Further the systems and methods describe herein also can be used for other multimedia data (e.g., graphics) which is coded in a scalable fashion. [0019] In a preferred embodiment of communication system 100, the H.264 SVC coding format ('SVC') is used for video communication. (See, e.g., the SVC JD5 specification, T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz, M. Wien, eds., "Joint Draft 5: Scalable Video Coding," Joint Video Team, Doc. JVT-R201, Bangkok, Thailand, January 2005, which is incorporated by reference herein in its entirety). SVC is the scalable video coding extension (Annex F) of the H.264 AVC video coding standard. 6 WO 2007/112384 PCT/US2007/065003 [00201 SVC offers considerable flexibility in creating bitstream structures with scalability in several dimensions (e.g. spatial, temporal, fidelity or quality dimensions). An SVC coded bitstream can be structured into several components or layers. A base layer offers a representation of the source signal at some basic fidelity. Additional layers (enhancement layers) provide information for improved representation of the signal in the additional scalability dimensions above the basic fidelity dimension. It is noted that the layers on the coded bitstream are typically formed in a pyramidal structure, in which the decoding of a layer may require the presence of one or more lower layers. Typically, the availability of the base layer is required for decoding of any of the enhancement layers. The pyramidal structure of the layers is not necessarily always the case. For example, when scalability is provided through multiple description coding or simulcasting, independent decoding of some or all layers may be possible. The systems and methods described herein are applicable to all of these scalability formats. 100211 Scalability can address several system-level challenges, such as heterogenous networks and/or clients, time-varying network performance, best effort network delivery, etc. In order to be able to effectively use these features, however, it is imperative that they are made accessible to system components beyond the video encoder and decoder. 100221 The systems and methods of the present invention, which effectively use all scalability features, are understood with reference to FIG. 1. First, it is noted that use of a media gateway 140 in system 100 is optional. The systems and methods of the present invention are the same or similar, when a direct media server to client connection is used, or when the media server is replaced by a file 7 WO 2007/112384 PCT/US2007/065003 that is directly accessible to the client on a mass storage or other access device, either directly or indirectly (e.g., a file access through a communication network). [0023] Consider a simple operational scenario in which media server/encoder 1 10 (e.g., a streaming server or encoder a transmitting endpoint encoder) communicates scalable media with client/receiver 120 through media gateway 140. This simple scenario requires that a connection be made between the media server and the client for transmitting an agreed-upon set of layers, which may, for example, be RTP-encapsulated SVC NAL units. Furthermore, media gateway 140 has to be instructed (or infer on its own) how to best operationally utilize the incoming packets (e.g., the transmitted RTP-encapsulated SVC NAL units). In the case where media gateway 140 has the SVCS/CSVCS architecture, this operational decision corresponds to deciding which packets to drop and which to forward. Further, for proper decoder operation, client/receiver 120 must know or be able to deduce which set of layers it is supposed to receive. 100241 To enable these functionalities, system 100 must represent and communicate the scalability structure of the transmitted bit stream to the various system components. As an illustrative example, consider a video signal with two temporal resolutions, 15 and 30 fps, and two spatial resolutions, QCIF and CIF. This corresponds to a four-layer scalability structure: layer LO containing the QCIF signal at 15 fps; layer LI containing the QCIF signal enhancement for 30 fps; layer SO containing the CIF signal enhancement for 15 fps; and layer SI containing the CIF signal enhancement for 30 fps. The coding dependency in the four-layer scalability structure could be such that LO is the base layer, Li depends on LO, SO depends on LO, and SI depends on both LI and SO. System 100 must 8 WO 2007/112384 PCT/US2007/065003 describe this four-layer structure to the system components so that they can properly process the video signal. [00251 SVC offers a mechanism for describing the scalability structure of an SVC coded video bitstream through its Scalability Information SEI message (SSEI). Supplemental Enhancement Information (SEI) messages, are data structures contained in an SVC bitstream that provide ancillary information about the coded video signal but are not necessary for the operation of the decoding process. The SSEI, defined in Section F.10.1.1 of the SVC JD5 specification, includes descriptive information about each layer (e.g., frame rate, profile information), and significantly, coding dependency information, i.e., which other layers a given layer depends on for proper decoding. Each layer is identified, within the scope of the bitstream, by a unique 'layer id'. The coding dependency information for a particular layer is communicated by encoding the number of directly dependent layers (num_ directlydependent layers), and a series of difference values (directlydependent layer iddelta), which when added to the particular layer's layer id identify the layer id's of the layers that the particular layer depends on for decoding. 100261 The SSEI as defined in Section F.10.1.1 of the SVC JD5 specification was designed in order to enable capability negotiation (e.g., during a connection setup), stream adaptation (by video server or intermediate media gateways), and low-complexity processing (e.g., without inference based on detailed bitstream parsing). The SSEI JD5 design, however, has several limitations, which become apparent upon close examination of the way the SSEI message information has to be used in a communication system. 9 WO 2007/112384 PCT/US2007/065003 10027] In the operation of a communication system (e.g., system 100), there can be several instances where the structure of the transmitted stream is established or modified (either at the server/source or the media gateway): - Initial definition during capability negotiation. The media server offers a set of scalability layers, and the client accepts all or a subset of them, either directly, or via a negotiation process (e.g., using Session Description Protocol (SDP)). - Modification of the layering structure by the media server or media gateway to accommodate changing system conditions (e.g., increased packet losses between media gateway and client or change of desired display resolution at the client). - Modification of the layering structure due to encoder decisions. An example of such an encoder decision relates to Fine Granularity Scalability (FGS), a mechanism for providing fidelity scalability where a fraction of the FGS layer bitstream can be used to produce a signal at a reduced quality. As a result of the way FGS is designed in SVC JD5, it is possible for the layer dependency to change from one picture to the next (or in fact, from one slice to the next). Specifically, consider the case where a prediction is performed at a given layer using as a reference a lower layer that uses FGS, and the prediction is based on only a fraction of that layer. In this case, it is possible that different pictures of the same layer may depend on different lower layers, as some layers could be completely eliminated due to rate control. A possible solution, which would not change the layer dependency structure, is to use an empty NAL unit. However, this is not allowed in SVC JD5. This case may be viewed as being similar to the 10 WO 2007/112384 PCT/US2007/065003 immediately preceding one, with the difference that the time scale in which the change in layer dependency occurs is extremely high (per-slice). [00281 Capability negotiation is a communication system feature that is outside the scope of the SVC specification, and scalable coding in general. Related communication application domains, such as IP telephony, have developed extensive procedures for capability negotiation. For example, Session Initiation Protocol (SIP)-based communication systems use SDP-based offer/answer. It is noted that use of the SSEI for this capability negotiation functionality alone is superfluous. In accordance with the present invention, the SSEI should not merely indicate what the encoder is capable of producing, but preferably should indicate what is (or will be) actually contained in the transmitted bitstream. In this respect, the SSEI, according to the present invention, is modified to define the overall scalability information structure for the bitstream, which is assumed to be in effect unless communicated otherwise. 100291 Modification of the layering structure (i.e., stream adaptation) can occur in the operation of a communication system at either the media server/encoder or intermediate media gateways (more than one such media gateway may be present). An example of stream adaptation is the case where client/receiver 120 decides to switch or change resolution (e.g., from CIF to QCIF). The switch or change in resolutions is effected by dropping the corresponding packets either at media gateway 140 or at transmitting media server 110. Triggering of the change of resolution is accomplished using traditional signaling (shown as signaling paths 150 in FIG. 1). An important consideration for proper switching operation is when client/receiver 120 can be signaled that the requested change has taken place. In the absence of any such signaled information, client/receiver 120 will, at 11 WO 2007/112384 PCT/US2007/065003 some point, receive only a subset of the total packets or bits (those corresponding to the QCIF resolution). It cannot know, however, if the loss of the high resolution packets or bits is intentional (the result of its request) or due to packet or bit losses. Considerations related to single-loop decoding may exacerbate the problem, since a lower layer picture may not be available at client/receiver 120 to be displayed in the absence of target layer data. 100301 As previously noted such switching information can be provided by signaling messages from the media server or media gateway to client/receiver 120 in a traditional manner (e.g., via signaling paths 150). However, the present invention recognizes that there are considerable benefits (e.g., in terms of synchronization and response time to error conditions) if the layering structure information is carried inband with the coded data. The layering structure information (an changes thereto) may be carried inband in modified SSEI messages. 100311 Another example of stream adaption due to changing system conditions, relates to adaptation to network-induced packet losses. In such case, the media gateway can decide to change or reduce, for example, the bandwidth of the transmitted stream using quality scalability features. Again for proper operation, client/receiver 120 must be timely informed or instructed about the change in the to-be-received bitstream. The changing layering structure information can be timely carried in modified SSEI messages inband with the coded data. 100321 Similar to the cases of stream adaptation, are the cases of encoder-driven layer structure changes. In contrast to the cases of stream adaptation discussed above, the encoder-driven layer structure changes are not initiated by the client or the media server in response to system parameter changes, but instead are an 12 WO 2007/112384 PCT/US2007/065003 "artefact" of the encoding process used by a video encoder. Like the signal adaptation cases, the information about the changes to layering structure can be timely signaled via modified SSEI messages inband with the coded data for proper synchronization with the media gateway and decoder. [00331 Consideration of the operation of system 100 in foregoing cases shows that there is a need for providing scalability information messages at various points in the course of a communication session, and at time scales (or frequencies) that may be as high as the slice level. 10034] It is noted that providing scalability information messages (e.g., via modified SSEI messages) can be practical only if the overhead in using the messages is made extremely small. Further, since the number of clients supported by a given server or gateway is often maximized for economy in a communication system, it is obviously desirable to minimize the computational burden in the communication system in parsing as well as generating the relevant messages. [00351 It is noted that the SVC JD5 specification mandates the use of SSEI only at an instantaneous decoding refresh (IDR) access unit (Section F. 10.2.1). IDR access units are equivalent to intra pictures, which have considerable bit rate overhead, and are often used only once in interactive video communication systems (for the encoding of the very first picture). This limitation further prohibits use of the standard SSEI to signal scalability information changes. [00361 Since one of the primary uses of the SSEI is to assist servers and media gateways in the proper handling of SVC streams, it is advantageous if the processing complexity of such messages is minimized. This is typically accomplished using fixed-length fields and as few options as possible. This simplicity contradicts the need for efficient compression, in order to minimize the 13 WO 2007/112384 PCT/US2007/065003 bit rate overhead associated with SSEI messages. Coding efficiency is particularly important if the granularity of SSEI message transmission is very high (which as noted above, could be at the slice level). [00371 Proposals have been made for more efficient representation, of the SSEI information from a compression point of view. (See, e.g., I. Amonou, N. Cammas, S. Kervadec, and S. Pateux, "Improved Signaling of Scalability Information," Joint Video Team, Doc. JVT-R068, Nice, Bangkok, January 2006, which is incorporated by reference herein in its entirety). However, the proposed representations have considerable computational complexity. For example, the aforementioned Doc. JVT-R068 proposes a representation, which shows a reduction of about 50% in overhead, when using a large number of scalability layers. The proposed representation is based on a tree-structured approach in coding of the different layers, where each node level corresponds to one of the three scalability dimensions (spatial, temporal, and quality, in that order). Presentation points correspond to the leaves of the tree. In contrast, the SVC JD5 syntax is based on a sequential parsing of the presentation points, coded in arbitrary order. This is similar to parsing of the leaves only of the tree structure of mentioned in Doc. JVT-R068, with the noted drawback that some information is unnecessarily duplicated from one layer to the next. In the tree-based approach, it is not necessary for leaves with a common ancestor to duplicate the information represented by their common parent(s). The unnecessary duplication, however, can be avoided by an alternative coding approach that employs judicious use of option flags when creating the SSEI. For example, when only the temporallevel changes from one layer to the next, it is practical to directly infer that the frame 14 WO 2007/112384 PCT/US2007/065003 sizes in both layers are the same and thus the frame size needs to be provided only once (e.g., at the lowest layer). 100381 With both of the aforementioned approaches (i.e., Doc. JVT-R068 and SVC JD5 syntax with the alternate interpretation of coding of option flags) it is noted that the "computation-compression" tradeoff is not ideal. In contrast to these approaches, the present invention employs a technique based on differential coding of the layer dependency information, so that layers can be added, replaced, or removed at will. In contrast to the aforementioned approaches, the inventive technique simultaneously reduces the bit rate overhead and computational complexity to minimal levels. 100391 The inventive technique for conveying necessary SSEI information can be understood with reference to its application to an example in which it is assumed that an initial SSEI is obtained and maintained as state information at the server, gateway, or client in the communication system. The needed bits to remove a layer or a subset of layers from the SSEI state can be very small, as one needs to only transmit enough "differential" information to identify the layer at which the SSEI information structure will be truncated. In other words, the indicated layer as well as all layers dependent on it can be eliminated from the state information upon identification of the layer. Similarly, in order to add a new layer or a new set of layers to the SSEI state, it is sufficient to transmit only the differential information pertaining to the layer(s) to be added. By design, this approach eliminates all overhead related to duplicate coding of layering structure information from one SSEI instance to the next. [00401 The reduction of the total bit rate associated with coding of SSEI information due to such differential encoding allows the use of simple coding 15 WO 2007/112384 PCT/US2007/065003 structures for the required messages, so that their parsing by media unaware (at least at the video coding level or VLC in H.264 terminology) gateways is simplified. It is noted that the technique does make any assumptions about how these SSEI messages will be transported or - more importantly - if and how they are transported reliably. It is sufficient to merely ensure state consistency if the same message is received more than once. 100411 The SSEI messages may be supplemented with the layer structure information in any suitable manner. Two embodiments (embodiments A and B) are described herein. In embodiment A, the SSEI is supplemented by three additional scalability information control SEI messages, corresponding to the operations of adding, replacing, or removing one or more layers. These operations are applied cumulatively; in other words, the input state assumed upon applying the indicated operation is the one in effect at the time just before the operation. In the alternative embodiment (embodiment B), the SSEI information is supplemented by two additional scalability information control SEI messages indicating which layers are not present, and which layers have changed decoding dependency information, respectively. In contrast to the cumulative operations of embodiment A, the operations in embodiment B are not applied in a cumulative manner, but instead assume that the input state for applying the indicated operation is the one established by the most recent SSEI message. Embodiment B may be preferred in environments where the SEI messages may be subject to losses. In this case, the initial SSE] can be communicated to the receiver or media gateway during session setup using reliable transport mechanisms. Loss of any intervening scalability information control messages in embodiment B will not 16 WO 2007/112384 PCT/US2007/065003 affect the receiver or media gateway's state information embodiment B as it would in Embodiment A. [00421 Embodiment A of the present invention introduces three new SEI payload types in Annex D of SVC JD5 (Section D.l): scalability_info_add (payload type 26), which adds one or more layers to an existing SSEI information structure; scalabilityinforeplace (payload type 27), which replaces one or more layers of an existing SSEI information structure; and scalability_info_delete (payload type 28) which deletes one or more layers from an existing SSEI information structure. 100431 The syntax for the 'add' and 'replace' versions is identical to that of the basic scalabilityinfo (SSEI) structure. Only the semantics of the contained data are changed, and only in terms of how SSEI state management is performed (in other words, individual field interpretations remain the same). The syntax for the 'delete' version can be simple as it only needs to identify the layer id's of the layers to be removed. It is noted that the 'replace' version is mostly useful to handle changes in the layer dependency information. Since no other information needs to be coded, the size of such a message can be made extremely small (e.g. in the order of 30-40 bits, depending on the number of layers). 100441 In the following, the use of the term "SSEI information" refers to any of these four types of scalability information, unless explicitly indicated otherwise. [00451 As the current SVC JD5 specification mandates the use of SSEI only at an IDR access unit (Section F.10.2.1), SSEI information in the present invention is allowed to occur at the beginning of any access unit. 17 WO 2007/112384 PCT/US2007/065003 [00461 When the SPS (Sequence Parameter Set in SVC) parameter nalextensionflag is set to 0, the dependencyid, temporal_level, and quality_level values (referred to herein as DTQ values) for all layers are defined in the SPS (Section F.7.3.2). Association of a given NAL unit to these values is performed using the simplepriorityid, if the NAL syntax element extensionflag is 0. If extensionflag is 1, then simplepriorityid is ignored according to the current specification, and the NAL header values for the DTQ parameters are used. Note that when profileidc is not equal to 83, then all NAL units of type 20 and 21 have to have their extensionflag set (Section F.7.4.2). [00471 If we assume that at the SPS level nalextensionflag is set to 0, then all SVC NAL headers will only employ the simplepriorityid rather than the full DTQ set. In order to allow a server or gateway to process the NAL stream, both messages have to be parsed. The SSEI provides the detailed layering information, whereas the SPS maps particular layers to priority id values. There appear to be no compelling reasons to include this information at the SPS, and there may in fact be several reasons not to do so. Upon examination of the rest of the information contained in the SPS, it is noted that none relates to transport issues. Accordingly, under the present invention, the relevant portion of the SPS can be removed and its coding moved instead to the SSEL. [00481 The use of nalextensionflag should preferably be stream-wide, in other words the coexistence of NALs with, and without, the extended NAL header should not be allowed. Although in theory enabling such capability may appear to offer more flexibility, in practice such capability complicates the operation of servers and gateways without real tangible benefits. As a result, coding of the nalextensionflag under the present invention is not done per-layer, but per 18 WO 2007/112384 PCT/US2007/065003 SSEI. This is in agreement with the SVC JD5 where coding is done on a per-SPS basis. 100491 With the simpleprorityid encoding done at the SSEI level, it is no longer necessary to have the encoding of DTQ values be optional, and as a result the corresponding flag (decodingdependencyinfopresent flag) of the SSEI can be removed. 100501 It has been previously noted that SVC is the scalable video coding extension (Annex F) of the H.264 AVC video coding standard. An AVC compatible layer implies use of NAL unit headers without either the simple_priorityid or DTQ field extensions. According to the SVC JD5 specification, the values are assumed to be 0. In the present invention the values of the simplepriority id or DTQ field extensions are signaled at the SSEI level. [00511 TABLE I shows exemplary syntax changes that are made to Section F.7.3.2, Sequence Parameter set SVC extension syntax, of the SVC JD5 specification in accordance with the principles of the present invention. The changes consist of removal of the syntax elements related to priority id and DTQ. 19 WO 2007/112384 PCT/US2007/065003 TABLE I SPS SVC EXTENSION MODIFICATIONS seq_parametersetsyc_extension() {C Descriptor e U(4) number of simplecprior-ity Id - alues-miRUSI 0 Htem preorstyid0 u2) tempal_slecLlistl priffity 0i dipcndhn _idjitI picrity 4d 0 -3 cuhaitypeklxitpsiitid 0 u(2) -4 _ extended spatial salabilityy 0 u(2) If( extended spatial scalability > 0) if ( chroma-format idc > 0) chroma phase xplusl 0 u(2) chroma_phase_y_plusl 0 u(2) if( extendedspatialscalability == I ) { scaledbaseleft offset 0 se(v) scaledbasetopoffset 0 se(v) scaled base right offset 0 se(v) scaledbasebottom offset 0 se(v) [00521 TABLE II shows exemplary syntax changes (additions) that are made in Annex D, D. 1, SEI payload type syntax, in accordance with the principles of the present invention. Three new payload types are added (26-28) corresponding to the add, replace, and delete operations. 20 WO 2007/112384 PCT/US2007/065003 TABLE 11 SEI PAYLOAD TYPES seijpayload( payloadType, payloadSize) { C Descriptor if( payloadType = = 0 ) Buffering_period( payloadSize) 5 else if( payloadType =I= ) pic-timing( payloadSize) etc. else if( payloadType == 22 ) scalabiltyinfo( payloadSize ) /* specified in Annex F */ 5 etc. else if (payloadType = 26) scalabilityinfoadd(payloadSize) /* specified in Annex F */ 5 else if(payloadType == 27) scalabilityinforeplace(payloadSize) /* specified in Annex F / 5 else if (payloadType == 28) scalabilityinfodelete(payloadSize) /* specified in Annex F */ 5 else reserved sei message( payloadSize) 5 if( !byte_aligned( )) { bit equalto one /* equal to 1 5 f(l) while( !bytealigned( ) ) bit equalto zero /* equal to 0*/ 5 f(l) [00531 TABLE III shows exemplary syntax changes that are made in Section F. 10.1.1, Scalability information SEI message syntax, in accordance with the principles of the present invention. We note the introduction of the nalunitextensionflag (moved from the SPS), and the introduction of the avccompatibleflag. Further, we note that DTQ encoding is always present for non-AVC layers, and that priorityid to DTQ association is established if the nalunitextensionflag is not set (i.e., when NAL units do not carry explicit DTQ information). 21 WO 2007/112384 PCT/US2007/065003 TABLE III SCALABILITY INFORMATION SEI MESSAGE SYNTAX scalabilityinfo( payloadSize) { C Descriptor num_layersminusl 5 ue(v) nal-unit extension flag 5 u() for( i = 0; i <= numlayersminusI; i++) { layer id[ i ] 5 u(8) avccompatible flag[ i] 5 u(1) fgs layer-flag[ i 1 5 u(1) subpic layerflag[ i 5 u(l) subregionlayer flag[ i] 5 u(l) profilelevelinfo present flag[ i ] 5 u(1) deeedingdependeney nfope -flag[ -5 u( bitrate-infopresentflag[ i ] 5 u(l) frmrate infopresentflag[ i] 5 u(l) frm-size-info-present-flag[ i ] 5 u(l) if (! avccompatibleflag[ i ]) { layer dependencyinfopresent flag[ i 5 u(l) } initparametersets infopresent flag[ i] 5 u(l) if (profilelevelinfo-present-flag[ i ]) layer profile idc[ i ] 5 u(8) layer constraint setoflag[ i ] 5 u(1) layer-constraint set _flag[ i ] 5 u(l) layer constraint set2_flag[ i ] 5 u(1) layerconstraint set3_flag[ i ] 5 u(l) reserved zero_4bits /* equal to 0 */ 5 u(4) layer level idc[ i 5 u(8) i decoding dependence infoprczcnI Aag ++4 if(! avccompatibleflag[ i ]) { if(! nal_unit extension flag) priorityid[ i ] 5 u(6) temporallevel[ i 5 u(3) dependencyid[ i 5 u(3) qualitylevel[ i ] 5 u(2) if( bitrate info_present flag[ i]) { avgbitrate[ i 5 u(16) max-bitrate[ i 5 u(16) 22 WO 2007/112384 PCT/US2007/065003 if( frm rateinfo_present flag[ i ] ) { constantfrmrate idc[ i] 5 u(2) avgfrmrate[ i 5 u(16) if( frm sizeinfopresent-flag[ i ] ) { frm width in mbs-minus1[ i ] 5 ue(v) frmheight in mbs minus[ i] 5 ue(v) if( sub regionlayer flag[i]) { base regionjayer id[ i ] 5 u(8) dynamicrect flag[ i ] 5 u(1) if( dynamic rectflag[ i]) { horizontaloffset[ i 5 u(16) verticial offset[ i ] 5 u(16) region-width[ i] 5 u(16) regionheight[ i 5 u(16) } } if( subpiclayer flag[ i ) roiid [ i ] 5 u(3) if( layer depdendencyinfo_presentflag[ i ]) num directlydependent layers[ i ] 5 ue(v) for( j = 0; j < numdirectlydependentlayers[ i ]; j++) directlydependent layer id delta[ i ][j ] 5 ue(v) if( initparameter sets infopresentflag[ i]) num_ init seqparameter setminusl[ i ] 5 ue(v) for( j = 0; j <= num seqjarametersetminusI[ i ];jj++ ) init seqparameter set id delta[ i ][j ] 5 ue(v) num_ init_pic_parameter set minus[ i ] 5 ue(v) for(j = 0; j <= numpicparametersetminusl[ i ];j++ ) initpic parameter setid delta[ i ][j ] 5 ue(v) I [00541 Corresponding changes are also made to the semantic interpretation of the SSEI message syntax elements (Section F.10.2.1, Scalability information SEI message semantics, of the SVC JD5 specification). Specifically, the IDR access 5 unit placement restriction is removed. Furthermore, when present, the SSEI message will take effect for all NAL units that follow and replace any and all 23 WO 2007/112384 PCT/US2007/065003 previous scalability information effective for the current stream. The semantics of the message are valid until the next SEI message of the type scalability_info, scalabilityinfoadd, scalabilityinforeplace, or scalability_infodel. The semantics of the nalunitextension_flag[i] are that a value equal to 0 specifies that the mapping of dependencyid[ i ], temporallevel[ i ], and qualityid[ i ] to simplepriorityid[ i ] for the layer layer id[ i ] is included in this scalability information SEI message. nalunitextension_flag equal to 1 specifies that the simplepriorityid[ i ] parameter is not present. When nalunitextension flag is not present, it is inferred to be equal to 1. The NAL unit syntax element extensionflag of all NAL units with nal unit-type equal to 20 and 21 that follow this scalability information SEI message shall be equal to nalunitextensionflag. When profile ide is not equal to 83 (identifying SVC), the syntax element extensionflag of all NAL units with nalunittype equal to 20 and 21 that reference the current sequence parameter set shall be equal to 1. Furthermore, the value of nalunitextension flag is not allowed to be different in any subsequent scalabilityinfo add or scalabilityinforeplace messages, unless first modified by a scalabilityinfo message. The semantics of avc compatibleflag[i] are that a value equal to 1 specifies that the layer layer id[ i ] conforms to the AVC specification. When avccompatibleflag[ i ] is equal to 0, the current layer conforms to the SVC specification. When ave_compatibleflag[i] is equal to 1, the NAL unit header does not carry the extended header information. When avccompatibleflag is equal to 1, then the values of dependencyid[ i], temporallevel[ i ], and qualitylevel[ i ] for the current layer i are all assumed to be 0. The semantics of the priorityid[i] are that it specifies the simplepriorityid value to be used in the inferring process for the syntax 24 WO 2007/112384 PCT/US2007/065003 elements dependencyid, temporallevel, and qualitylevel (as specified in Section F.7.4.1 of the SVC JD5 specification). For all values of priority_id[ i], for which dependencyid[ i ], temporallevel [ i ], and qualitylevel list[ i ] are not present, dependencyid[ i ], temporallevel[ i ], and qualitylevel[ i ] shall be inferred to be equal to 0. Finally, temporallevel[ i ], dependencyid[ i ] and qualitylevel[ i ] are equal to temporal_level, dependencyid and quality-level, respectively, of the NAL units in the scalable layer with layer identifier equal to i. If temporal_level[ i ], dependencyid[ i ], and qualitylevel[ i ] are not present, they are assumed to be 0. [00551 The syntax of the new SEI 'scalabilityinfoadd' message is identical to the 'scalabilityinfo' SEI message. In terms of its semantics, when present, this SEI message will take effect for all NAL units that follow. It adds a number of layers to the scalability information currently effective. If the layer id of a layer described in this message is already present in the scalability information maintained at the receiver, then the entire information for that layer is deleted and the information contained in this message is used in its place. The scalability information state effective after the application of this message remains valid until the next SEI message of the type scalability info, scalabilityinfo_ add, scalabilityinfo replace, or scalabilityinfo del. The semantics of all fields are identical to scalabilityinfo. [00561 The syntax of the new SEI 'scalability_inforeplace' message is identical to the 'scalabilityinfo' SEI message. In terms of its semantics, when present, this SEI message will take effect for all NAL units that follow. It replaces the information for a number of layers, as specified by their layerid, in the scalability information currently effective. Only the information contained in the message is 25 WO 2007/112384 PCT/US2007/065003 replaced; any prior settings that are not modified by a given message remain in effect unmodified. The scalability information state effective after the application of this message remains valid until the next SEI message of the type scalabilityinfo, scalabilityinfoadd, scalabilityinfo replace, or scalabilityinfo del. The semantics of all fields are identical to scalabilityinfo. [00571 TABLE IV shows exemplary syntax for the new 'scalability infodel' SEI message, in accordance with the principles of the present invention. As shown in the table, the message consists of a list of the layer id's of the layers that are to be considered removed from the bitstream. When present, this SEI message will take effect for all NAL units that follow. It deletes the information for a number of layers, as specified by their layer id, in the scalability information currently effective. The scalability information state effective after the application of this message remains valid until the next SEI message of the type scalability_info, scalabilityinfoadd, salability inforeplace, or scalability_infodel. The semantics of all fields are identical to the corresponding ones in scalabilityinfo. 26 WO 2007/112384 PCT/US2007/065003 TABLE IV SCALABILITY INFORMATION DELETE SEI MESSAGE SYNTAX scalabilityinfo del( payloadSize) { C Descriptor num_layers minusl 5 ue(v) for( i = 0; i <= numlayers_minus 1; i++) layer id[ i] 5 u(8) } 100581 FIG. 2 shows a flow diagram of an exemplary state processing algorithm 200 in embodiment A. The initial layer structure state (State) of the receiver is empty. At step 22, the receiver obtains Scalability Information SEI messages, either from the bitstream or through alternate means such as signaling. An obtained scalabilityinfo SEI message can act as an initializing SSEI message and serves to initialize the State. Similarly obtained 'add', 'delete', and 'replace' scalability info SEI messages can add to, remove from, or modify, respectively, the current State for one or more layers. It is noted that multiple layers may be affected by each of these messages. However, for clarity the diagram in FIG. 2 shows the effect on a single layer (layer(i)). [00591 In some communication systems or communication scenarios, the communication of the SSEI information message may never be inband with video data, but always communicated during session establishment, or through other reliable communication means. As video data may be transported over channels with non-zero packet loss rates, it may be desirable to ensure that such critical information is not subject to the possibility of loss. 10060] For such communication systems or communication scenarios, in alternative embodiment B of the presentation invention, the Scalability 27 WO 2007/112384 PCT/US2007/065003 Information may be established by a scalabilityinfo SEI message, either offered during SDP negotiation or present in a in IDR access unit (e.g., in a file format). The information in such a scalabilityinfo SEI message establishes the Scalability Information until the end of the sequence, or until another scalabilityinfo SEI message is received, or re-negotiated in the case of SDP. The Scalability Information is not allowed to change during that time except from internal dependencies between layers. A further difference in embodiment B with respect to the embodiment A relates to the way state management is handled. In the embodiment A state changes are applied in a cumulative fashion, which has the drawback that the loss of a single message could corrupt the state information. In the alternative embodiment B, state changes are always applied with respect to the most recent scalabilityinfo SEI message, and loss effects are not compounded. [00611 In embodiment B, the 'add' SSEI message is not used, whereas the 'delete' SSEI message is renamed 'layers_ notpresent.' A 'layersnotpresent' SSEI message indicates that a particular layer, and all dependent layers, will not be present in the bitstream starting from the access unit in which the SEI message is contained. The indication of which layers are not present is always performed with respect to the currently active scalabilityinfo SEI message. It is noted that a convenient mechanism to indicate that a particular layer has been added after it has been removed is to transmit a 'layers notpresent' message that removes all currently absent layers except the particular layer that is added. As an illustrative example, assume a bit stream that has 3 layers, numbered 0, 1, and 2. Further assume that after initially transmitting all 3 layers, a media server transmits a 'layersnotpresent' message indicating that layers 1 and 2 are not present. In this case, layer I can be added at a later time by transmitting a 'layersnotpresent' 28 WO 2007/112384 PCT/US2007/065003 message which indicates that only layer 2 is not present. This, in effect, indicates that layer 1 has been added. The media server (or media gateway) can start transmitting the layer 1 information immediately after transmitting this 'layers notpresent' SEI message. [00621 In embodiment B, the 'replace' SSEI message of the first embodiment is changed to 'dependencychange' message. A scalability information dependency change SEI message indicates that the layer dependency for a particular layer-id has changed, and provides the new layer dependency information. The syntax for the dependency change SSEI message allows no other change in the information associated with the scalability information. [00631 TABLE V shows exemplary syntax for the alternate new SEI payload types scalabilityinfolayers notpresent (nr. 26) and scalabilityinfodependencychange (nr. 27), replacing the syntax in Section D. 1 of the SVC JD5 specification. In contrast to embodiment A, in embodiment B the Scalability Information SEI message is allowed to be present only at an IDR picture. The semantics of the message, with the exception of the layer dependency information, are valid until the next SEI message of the same type. 29 WO 2007/112384 PCT/US2007/065003 TABLE V ALTERNATE SEI PAYLOAD TYPES sei-payload( payloadType, payloadSize) { C Descriptor if( payloadType == 0 ) Bufferingperiod( payloadSize) 5 else if( payloadType = I ) etc. 5 else if( payloadType = = 22 ) scalabilty_info( payloadSize ) /* specified in Annex F */ 5 etc. else if (payloadType == 26) scalabilityinfolayersnotpresent(payloadSize) /* specified in Annex F */ 5 else if (payloadType == 27) scalabilityinfodependency changc(payloadSize) /* specified in Annex F 5 */ else reserved sei message( payloadSize) 5 if( !bytealigned() ){ bit_equalto one /* equal to 1 */ 5 f(l) while( !bytealigned( ) ) bit equal to zero /* equal toO */ 5 f(l) 5 [00641 Specific layers can be indicated as inactive (no longer present in the bitstream) with the scalabilityinfo layers notpresent SEI message. Exemplary syntax for this message is shown in TABLE VI. TABLE VI 10 SCALABILITY INFORMATION LAYERS-NOT-PRESENT SEI MESSAGE scalabilityinfo layersnotpresent( payloadSize) { C Descriptor num_layers 5 ue(v) for( I = 0; i < numlayers; i++ ) { layer id[ i] 5 u(8) 100651 When present, this SEI message will take effect for all NAL units that follow. The message indicates that the specified layers, as identified by their 30 WO 2007/112384 PCT/US2007/065003 layer id, will not be present in the bitstream. The scalability information state effective after the application of this message remains valid until the next SEI message of the type scalabilityinfo or scalabilityinfolayersnotpresent. The numlayers indicates the number of scalable layers or presentation points that will not be present in the bitstream after decoding of this SEI message. The value of numlayers is in the scope of 0 to 255, inclusive. When numlayers has the value 0 it indicates that all layers are present. The layer id[ i ] indicates the identifier of the scalable layer that is indicated as not present in the bitstream. The message simply contains a list of the layers that are identified as not present. 100661 TABLE VII shows exemplary syntax for the scalabilityinfodependencychange SEI message. When present, this SEI message will take effect for all NAL units that follow. Layer dependency information for a particular layer consists of the information resulting from the decoding of the syntax elements numdirectlydependent layer, directlydependent layer iddeltaminus 1, or alternatively the new syntax element layer dependency_info_src_layer id deltaminusl. This new element allows the system to identify he layer id of another layer that has identical layer dependency information to the present layer (by encoding the layer id difference minus 1). 31 WO 2007/112384 PCT/US2007/065003 TABLE VII SCALABILITY INFORMATION DEPENDENCY CHANGE SEI MESSAGE scalability info dependencychange ( payloadSize) { C Descriptor num_layers minus 5 ue(v) for( i = 0; i <= num_layersminus 1; i++) { layer id[ i ] 5 u(8) layer dependencyinfopresentjlag[i] 5 u(1) if (layer-depdendencyinfopresent-flag[ i]) { num directlydependent layers[ i ] 5 ue(v) for(j =0;j <numdirectlydependent_ layers[ i ];j++) directlydependent layer id delta minusl[ i ][j ] 5 ue(v) else { layer dependencyinfosrc_l ayer-id-delta minus[ i 5 ue(v) } } 5 100671 The scalabilityinfo dependencychange SEI message indicates that the layer dependency information for the specified layers, as identified by their layer id, will change as specified by the contents of this message. The scalability information state effective after the application of this message remains valid until the next SEI message of the type scalabilityinfo or 10 scalabilityinfo dependencychange. 100681 num layersminus I plus 1 indicates the number of scalable layers or presentation points for which the layer dependency information will change after decoding of this SEL message. The value of num_layers minus is in the scope of 0 to 255, inclusive. 15 [00691 The layer id[ i ] indicates the identifier of the scalable layer for which the layer dependency information will change. [00701 The layer dependencyinfopresent-flag[ i ] equal to 1 indicates the presence of the layer dependency information for the scalable layer with layer 32 WO 2007/112384 PCT/US2007/065003 identifier equal to layer id[ i ] in the SEI message. A value 0 indicates that the layer dependency information for the scalable layer with layer identifier equal to i is not present in the SEI message, and is the same as another layer indicated by layer-dependency_info_sre-layer-id-delta-minusI[ i ]. 100711 numdirectlydependentlayers[ i ] indicates the number of scalable layers that the scalable layer with layer identifier equal to layer id[ i ] is directly dependent on. For example, layer M is directly dependent on layer Q if there is at least one coded picture in layer M that uses inter-layer prediction from layer Q. The value of num directlydependent layers is in the scope of 0 to 255, inclusive. [00721 directlydependent layerid-delta_minus I( i ][ j ] indicates the difference between layer id[ i ] and the layer identifier of the jth scalable layer that the scalable layer with layer identifier equal to layer id[ i ] is directly dependent on and i. The layer identifier of the directly dependent scalable layer is equal to (layer id[ i ] - directlydependent layeriddeltaminus! - 1). [00731 layer dependencyinfo_src_layer id delta minus![ i I specifies that the layer with layer identifier equal to layer id[ i ] has the same layer dependency information as the layer with layer identifier equal to ( layer id[ i ] layer dependencyinfosrc_layerjiddeltaminus I[ i ] - I ). [00741 FIG. 3 shows a flow diagram of an exemplary state processing algorithm 300 in embodiment B. The initial layer structure state (State) of the receiver is empty. At step 33, the receiver obtains Scalability Information SEI messages, either from the bitstream or through alternate means such as signaling. A scalabilityinfo SEI message acts as an initializing SSEI message and it serves to initialize the state. The 'layers notpresent' and 'dependencychange' messages mark one or more layer as not present, or modify their decoding dependency 33 WO 2007/112384 PCT/US2007/065003 information, respectively. It is noted that in this alternative embodiment B, layers are neither added nor removed by the non-initializing messages. This ensures state consistency (with the possible exception of decoding dependency), as long as the initializing messages are reliably received. It is noted that multiple layers may be affected by each of these messages. However, for clarity the diagram in FIG. 3 like the diagram in FIG. 2 shows the effect on a single layer (layer(i)). [0075] Although the preferred embodiments described herein use the H.264 SVC standard, it will be obvious to persons skilled in the art that the techniques described herein can be directly applied or extended to any coding structure that allows multiple spatial/quality, and temporal levels, either for video or for audio, or any other multimedia data. 10076] While there have been described what are believed to be the preferred embodiments of the present invention, those skilled in the art will recognize that other and further changes and modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications as fall within the true scope of the invention. 100771 It also will be understood that the systems and methods of the present invention can be implemented using any suitable combination of hardware and software. The software (i.e., instructions) for implementing and operating the aforementioned systems and methods can be provided on computer-readable media, which can include without limitation, firmware, memory, storage devices, microcontrollers, microprocessors, integrated circuits, ASICS, on-line downloadable media, and other available media. 34

Claims (13)

1. A digital media communication system comprising: a media server that transmits media encoded in a scalable media coding format; 5 at least one receiver; and an electronic communication network linking the media server and the at least one receiver, wherein the media server is configured to transmit to the at least one receiver information describing structure of the transmitted media selected from the group 10 consisting of initializing information and update information, and wherein the update information. if any, is transmitted when the structure of the transmitted media changes and is encoded in a differential form.
2. The system of claim I wherein the differential form further includes using as a 15 reference at least one of: the most recently transmitted updated information and the most recently transmitted initializing information.
3. The system of claim 1 wherein the differential form further includes using as a reference the most recently transmitted initializing information. 20
4. A digital media decoder which is configured to process coded media data comprising: scalably coded media data; initializing structure information data; and 25 changes to initializing structure information data, wherein the changes to initializing structure information data are encoded in a differential form.
5. The decoder of claim 4 wherein the differential form further includes using as a 30 reference at least one of: the most recently transmitted updated information and the most recently transmitted initializing information. C \NRPonhI\DCC\KXM\426(A24_1 DOC- 12,4/2112 -36
6. The decoder of claim 4 wherein the differential form further includes sing as reference the most recently transmitted initializing information. 5
7. A method for digital media communication over an electronic communication network linking at least one receiver to a transmitting media server which transmits media encoded in a scalable media coding format, the method comprising: transmitting from the media server to the at least one receiver information describing structure of the transmitted media selected from the group consisting of 10 intializing information and update information., wherein the update information, if' any, is transmitted when the structure of the transmitted media changes and is encoded in a differential form.
8. The method of claim 7 wherein the differential further includes using as a reference 15 at least one of: the most recently transmitted updated information and the most recently transmitted initializing information.
9. The method of claim 7 wherein the differential form further includes using as reference the most recently transmitted initializing information. 20
10. A method for decoding digital media at a decoder, wherein the digital media includes scalably coded media data and information describing the structure of the transmitted media data as initializing information and update information, with said update information encoded in a differential form,, the method comprising: 25 applying the differentially encoded update information to previously received update information or initializing information to reconstruct the current structure of the transmitted media.
11. The method of claim 10 wherein the differential form of encoding encodes updates 30 using as a reference at least one of: the most recently transmitted updated information and the most recently transmitted initializing information. C:\NRPorbI\DCC\KXM\426M24_ I DOC-120412-112 - 37
12. The method of claim 10 wherein the differential form of encoding encodes updates using as reference the most recently transmitted initializing information. 5
13. A non-transitory computer-readable medium comprising instructions to perform the methods of one of claims 7 to 12.
AU2012202123A 2006-03-27 2012-04-12 System and method for management of scalability information in scalable video and audio coding systems using control messages Active AU2012202123B2 (en)

Priority Applications (13)

Application Number Priority Date Filing Date Title
US60/786,178 2006-03-27
US60/786,997 2006-03-29
US60/789,550 2006-04-05
USPTC/US06/028368 2006-07-20
USPCT/US2006/0028367 2006-07-20
USPCT/US2006/0028365 2006-07-20
AUPCT/US06/028366 2006-07-21
USPCT/US2006/0061815 2006-12-08
USPCT/US2006/0062569 2006-12-22
US60/884,148 2007-01-09
USPCT/US2006/0062357 2007-02-16
AU2007230602A AU2007230602B2 (en) 2006-03-27 2007-03-27 System and method for management of scalability information in scalable video and audio coding systems using control messages
AU2012202123A AU2012202123B2 (en) 2006-03-27 2012-04-12 System and method for management of scalability information in scalable video and audio coding systems using control messages

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
AU2012202123A AU2012202123B2 (en) 2006-03-27 2012-04-12 System and method for management of scalability information in scalable video and audio coding systems using control messages

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
AU2007230602A Division AU2007230602B2 (en) 2005-09-07 2007-03-27 System and method for management of scalability information in scalable video and audio coding systems using control messages

Publications (2)

Publication Number Publication Date
AU2012202123A1 true AU2012202123A1 (en) 2012-05-03
AU2012202123B2 AU2012202123B2 (en) 2014-01-30

Family

ID=46640347

Family Applications (1)

Application Number Title Priority Date Filing Date
AU2012202123A Active AU2012202123B2 (en) 2006-03-27 2012-04-12 System and method for management of scalability information in scalable video and audio coding systems using control messages

Country Status (1)

Country Link
AU (1) AU2012202123B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104270684A (en) * 2014-09-24 2015-01-07 北京中科大洋科技发展股份有限公司 Video and audio data network transmission system and method oriented to real-time application

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3263807B2 (en) * 1996-09-09 2002-03-11 ソニー株式会社 Image encoding apparatus and image encoding method
US20050254575A1 (en) * 2004-05-12 2005-11-17 Nokia Corporation Multiple interoperability points for scalable media coding and transmission

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104270684A (en) * 2014-09-24 2015-01-07 北京中科大洋科技发展股份有限公司 Video and audio data network transmission system and method oriented to real-time application
CN104270684B (en) * 2014-09-24 2018-05-11 北京中科大洋科技发展股份有限公司 It is a kind of towards the video/audio network transmission system and method applied in real time

Also Published As

Publication number Publication date
AU2012202123B2 (en) 2014-01-30

Similar Documents

Publication Publication Date Title
JP6345827B2 (en) Providing a sequence data set for streaming video data
JP6556907B2 (en) Video encoding apparatus, video encoding method, data stream storage method, and data stream generation method
US9769485B2 (en) Multi-layer encoding and decoding
KR20190009378A (en) METHOD FOR ENCODING MEDIA CONTENT, DEVICE, AND COMPUTER PROGRAM
US10154289B2 (en) Signaling DPB parameters in VPS extension and DPB operation
US9332254B2 (en) Coding, storage and signalling of scalability information
US10194160B2 (en) Signaling change in output layer sets
CA2972332C (en) Inter-layer prediction for scalable video coding and decoding
JP5973077B2 (en) Bitstream properties in video coding
US10728148B2 (en) Packet format of network abstraction layer unit, and algorithm and apparatus for video encoding and decoding using the format, QoS control algorithm and apparatus for IPv6 label switching using the format
KR102202143B1 (en) Method and apparatus of transmitting and receiving flexible MMT asset
US9705746B2 (en) Channel bonding for layered content
EP2596643B1 (en) Arranging sub-track fragments for streaming video data
US8442120B2 (en) System and method for thinning of scalable video coding bit-streams
JP5640104B2 (en) System and method for implementing signaling and time level switching in scalable video coding
CN1801944B (en) Method and device for coding and decoding video
CN102106156B (en) Real-time transport protocol (RTP) packetization method for fast channel change applications using scalable video coding (SVC)
US9319717B2 (en) System and method for providing and using predetermined signaling of interoperability points for transcoded media streams
AU2007311526B2 (en) System and method for providing picture output indications in video coding
US7039116B1 (en) Methods and apparatus for embedding and format conversion of compressed video data
RU2409910C2 (en) Backward-compatible aggregation of images in scalable video coding
AU2010259914B2 (en) Multiview video coding over MPEG-2 systems
US7720096B2 (en) RTP payload format for VC-1
US8436889B2 (en) System and method for videoconferencing using scalable video coding and compositing scalable video conferencing servers
AU2005242601B2 (en) Multiple interoperability points for scalable media coding and transmission

Legal Events

Date Code Title Description
FGA Letters patent sealed or granted (standard patent)