BR122023012858B1 - POSITIONING OF THE STRUCTURE ELEMENT IN STRUCTURES OF A CONTINUOUS DATA STREAM REPRESENTING AUDIO CONTENT - Google Patents

Abstract

A better fit between a very high data stream and high coding, on the one hand, and flexibility of structure element positioning, on the other hand, is achieved by arranging each of the sequences of structures of the data stream comprising a sequence of N structure elements and, on the other hand, the data stream comprising a configuration block comprising a field indicating the number of elements N and a type indication syntax portion indicating, for each element position of the sequence of N element positions, an element type from a plurality of element types with, in the sequences of N structure elements of the structures, each structure element being of the element type indicated, by the type indication portion, for the respective element position at which the respective structure element is positioned within the sequence of N structure elements of the respective structure in the data stream. Thus, the structures are equally structured wherein each structure comprises the same sequence of N structure elements of the structure element type indicated by the indication syntax portion (...).

Description

Specification

[0001] The present invention relates to audio coding, such as the so-called USAC codec [USAC = Unified Speech and Audio Coding] and in particular to the positioning of the structure element with respective data stream structures.

[0002] In recent years, several audio codecs have become available, each audio codec being specifically designed to suit a dedicated application. Primarily, these audio codecs are capable of encoding more than one audio channel or audio signal in parallel. Some audio codecs are also suitable for different encoding of audio content by differently grouping audio channels or audio objects of the audio content and subjecting these groups to different audio coding principles. Furthermore, some of these audio codecs allow the insertion of extension data into the data stream, to accommodate future extensions/developments of the audio codec.

[0003] An example of such audio codecs is the USAC codec, as defined in ISO/IEC CD 23003-3. This standard, named “Information Technologies — MPEG Audio Technologies — Part 3: Unified Audio and Speech Coding”, describes in detail the functional blocks of a reference model of a call for proposals on unified audio and speech coding.

[0004] Figures 5a and 5b illustrate block diagrams of the decoder and encoder. In the following, the overall functionality of the individual blocks is briefly explained. Then, the problems of putting all the parts of the resulting syntax together into a continuous data stream are explained with reference to Figure 6.

[0005] Figures 5a and 5b illustrate block diagrams of the encoder and decoder. The block diagrams of the USAC encoder and decoder reflect the structure of MPEG-D USAC encoding. The general structure can be described as follows: first, there is a common pre/post-processing consisting of an MPEG surround (MPEGS) functional unit to handle stereo or multichannel processing and an enhanced SBR (eSBR) unit that handles the parametric representation of the higher audio frequencies in the input signal. Then, there are two branches, one consisting of an Advanced Audio Coding (AAC) tool path and the other consisting of a linear prediction coding (LP)-based path, which in turn presents either a frequency domain representation or a time domain representation of the residual LPC. All transmitted spectra for both AAC and LPC are represented in the MDCT domain following arithmetic quantization and coding. The time domain representation uses an ACELP excitation coding scheme.

[0006] The basic structure of MPEG-D USAC is shown in Figure 5a and Figure 5b. The data flow in this diagram is from left to right, top to bottom. The functions of the decoder are to find the quantized audio spectrum description or time domain representation in the data stream payload and decode the quantized values and other reconstruction information.

[0007] In the case of transmitted spectral information, the decoder shall reconstruct the quantized spectrum, process the reconstructed spectrum through whatever tools are available in the payload of the continuous data stream to arrive at the spectrum of the actual signal as described by the payload of the continuous data stream of input, and finally convert the spectrum from the frequency domain to the time domain. Following the initial reconstruction and scaling of the spectrum reconstruction, there are optional tools that modify one or more of the spectra to provide more efficient encoding.

[0008] In the case of transmitted time domain signal representation, the decoder shall reconstruct the quantized time signal, process the reconstructed time signal through whatever tools are active in the data stream payload to arrive at the real time domain signal as described by the input data stream payload.

[0009] For each of the optional tools that operate on the signal data, the option to “pass” is retained, and in all cases where processing is omitted, the spectrum or time samples in its input are passed directly through the tool without modification.

[0010] At locations, where the continuous data stream changes its signal representation from time domain to frequency domain representation or from LP domain to non-LP domain or vice versa, the decoder shall facilitate the transition from one domain to another by means of a windowing with appropriate transition addition-overlap.

[0011] eSBR and MPEGS processing is applied in the same manner to both encoding paths after transition work.

[0012] The input to the data stream payload demultiplexer tool is the MPEG-D USAC data stream payload. The demultiplexer separates the data stream payload into parts for each tool and provides each tool with the data stream payload information related to that tool.

[0013] The outputs of the data stream payload demultiplexer tool are:

[0014] Depending on the type of core coding in the current structure:

[0015] the quantized noise-free encoded spectrum represented by:

[0016] scale factor information;

[0017] arithmetically coded spectral lines.

[0018] or: linear prediction parameters (LP) together with an excitation signal represented by:

[0019] arithmetically coded and quantized spectral lines (transform coded excitation, TCX); or coded time domain excitation ACELP.

[0020] Spectral noise padding information (optional);

[0021] The M/S decision information (optional);

[0022] Temporal noise shaping (TNS) information (optional);

[0023] The filter bank control information;

[0024] Time unwarping (TW) control information (optional);

[0025] Enhanced Spectral Bandwidth Replication (eSBR) control information (optional);

[0026] The MPEG surround control information (MPEGS).

[0027] The noiseless decoding tool with scale factor obtains information from the payload demultiplexer of the data stream, analyzes this information, and decodes the Huffman and DPCM encoded scale factors.

[0028] The input to the scale factor noiseless decoder tool is:

[0029] The scale factor information for the noise-free encoded spectrum.

[0030] The output of the scale factor noise-free decoder tool is:

[0031] The decoded integer representation of the scale factors.

[0032] The spectral noiseless decoder tool obtains information from the payload demultiplexer of the continuous data stream, analyzes this information, decodes the arithmetically encoded data, and reconstructs the quantized spectrum. The input to this noiseless decoder tool is:

[0033] The noise-free encoded spectrum.

[0034] The output of this noise-free decoder tool is:

[0035] The quantized values of the spectrum.

[0036] The inverse quantizer tool obtains the quantized values for the spectrum and converts the integer values to the reconstructed, unscaled spectrum. This quantizer is an expanded quantizer, whose expanded factor depends on the selected core coding mode.

[0037] The input to the Inverse Quantizer tool is:

[0038] The quantized values for the spectrum.

[0039] The output of the inverse quantizer tool is:

[0040] The inversely quantized, unscaled spectrum.

[0041] The noise fill tool is used to fill spectral gaps in the decoded spectrum that occur when the spectral value is quantized to zero, for example due to a strong data demand constraint at the encoder. The use of the noise fill tool is optional.

[0042] The inputs to the noise fill tool are:

[0043] The inversely quantized, unscaled spectrum;

[0044] Noise filling parameters;

[0045] The decoded integer representation of the scale factors.

[0046] The outputs for the noise fill tool are:

[0047] The unscaled, inversely quantized spectral values for spectral lines that were previously quantized to zero;

[0048] Modified integer representation of scale factors.

[0049] The rescaling tool converts the integer representation of the scale factors to the real values, and multiplies the unscaled inversely quantized spectrum by the relevant scale factors.

[0050] The inputs to the scale factors tool are:

[0051] The decoded integer representation of the scale factors;

[0052] The inversely quantized, unscaled spectrum.

[0053] The output of the scale factors tool is:

[0054] The inversely quantized, unscaled spectrum.

[0055] For an overview of the M/S tool, please refer to ISO/IEC 14496-3:2009, 4.1.1.2.

[0056] For an overview of the temporal noise modeling (TNS) tool, please refer to ISO/IEC 14496-3:2009, 4.1.1.2.

[0057] The block/filter bank switching tool applies the inverse of the frequency mapping that was performed on the encoder. An inverse modified discrete cosine transform (IMDCT) is used for the filter bank tool. The IMDCT can be configured to support spectral coefficients of 120, 128, 240, 256, 480, 512, 960, or 1024.

[0058] The inputs to the filter bank tool are:

[0059] The spectrum (inversely quantized);

[0060] The filter bank control information.

[0061] The output(s) of the filter bank tool is(are):

[0062] The reconstructed time-domain audio signal(s).

[0063] The time-warped filter bank/block switching tool replaces the normal filter bank/block switching tool when time-warping mode is enabled. The filter bank is the same (IMDCT) as for the normal filter bank, additionally the windowed time domain samples are mapped from the warped time domain to the linear time domain by time-varying resampling.

[0064] The inputs to the time-warped filter bank tools are: The inversely quantized spectrum; The filter bank control information; The time-warped control information.

[0065] The output(s) of the filter bank tool is (are): The reconstructed linear time domain audio signal(s).

[0066] The improved SBR (eSBR) tool regenerates the high band of the audio signal. It is based on the replication of harmonic sequences, truncated during encoding. It adjusts the spectral envelope of the generated high band and applies inverse filtering, and adds noise and sinusoidal components to recreate the spectral characteristics of the original signal.

[0067] The input to the eSBR tool is:

[0068] The quantized wrapper data;

[0069] General control data;

[0070] a time domain signal from the frequency domain core decoder or ACELP/TCX core decoder.

[0071] The output of the eSBR tool is either:

[0072] a time domain signal; or

[0073] A QMF-domain representation of a signal, for example, is used in the MPEG Surround tool.

[0074] The MPEG Surround (MPEGS) tool produces multiple signals from one or more input signals by applying a sophisticated upmix procedure to the input signal(s) controlled by suitable special parameters. In the USAC context, MPEGS is used to encode a multichannel signal, transmitting parametric side information along with a transmitted downmix signal.

[0075] The input to the MPEGS tool is:

[0076] a reduced time domain signal; or

[0077] a QMF-domain representation of a reduced signal from the eSBR tool.

[0078] The output of the MPEGS tool is:

[0079] a multichannel time domain signal.

[0080] The Signal Classifier tool analyzes the original input signal and generates from it control information that triggers the selection of different encoding modes. The input signal analysis is implementation dependent and will attempt to select the optimal core encoding mode for a given input signal structure. The output of the signal classifier can (optionally) also be used to influence the behavior of other tools, e.g. MPEG Surround, Enhanced SBR, Time-Warped Filter Bank, and others.

[0081] The input to the Signal Classifier tool is:

[0082] the original unmodified input signal;

[0083] additional implementation-dependent parameters.

[0084] The output of the Signal Classifier tool is:

[0085] a control signal for controlling the selection of the core codec (non-LP filtered frequency domain coding, LP filtered frequency domain, or LP filtered time domain coding).

[0086] The ACELP tool provides a way to efficiently represent a time domain excitation signal by combining a long term predictor (adaptive codeword) with a pulse-like sequence (innovation codeword). The reconstructed excitation is sent through an LP synthesis filter to form a time domain signal.

[0087] The input to the ACELP tool is:

[0088] adaptive and innovation codebook indexes;

[0089] gain values of adaptive and innovation codes;

[0090] other control data;

[0091] inversely interpolated and quantized LPC filter coefficients.

[0092] The output of the ACELP tool is:

[0093] The reconstructed time-domain audio signal.

[0094] The MDCT-based TCX decoder tool is used to transform the weighted LP residual representation of an MDCT domain back to a time domain signal and outputs a time domain signal including weighted LP synthesis filtering. The IMDCT can be configured to support 256, 512 or 1024 spectral coefficients.

[0095] The input to the TCX tool is:

[0096] The MDCT spectrum (inversely quantized);

[0097] Inversely interpolated and quantized LPC filter coefficients.

[0098] The output of the TCX tool is:

[0099] The reconstructed time-domain audio signal

[0100] The technology disclosed in ISO/IEC CD 23003-3, which is incorporated herein by reference, allows the definition of channel elements that are, for example, single channel elements containing only the payload for single channel elements, or channel pairs containing payload for two channels, or LFE (Low Frequency Enhancement) channel elements containing payload for one LFE channel.

[0101] Of course, the USAC codec is not the only codec capable of encoding and transferring information in a more complicated audio codec of more than one or two audio channels or audio objects through a continuous data stream. Therefore, the USAC codec served simply as a concrete example.

[0102] Fig. 6 shows a more general example of an encoder and decoder, respectively, both depicted in a common scenario where the encoder encodes audio content 10 into a continuous data stream 12, with the decoder decoding the audio content, or at least a portion thereof, from the continuous data stream 12. The result of the decoding, i.e., the reconstruction, is indicated at 14. As illustrated in Fig. 6, the audio content 10 may be composed of a number of audio signals 16. For example, the audio content 10 may be a spatial audio scene composed of a number of audio channels 16. Alternatively, the audio content 10 may represent a conglomeration of audio signals 16 with the audio signals 16 representing, individually and/or in groups, individual audio objects that may be placed together in an audio scene, at the discretion of the user of the decoder, to obtain the reconstruction 14 of the audio content 10 in the form of, for example, For example, a spatial audio scene for a specific speaker configuration. The encoder encodes the audio content 10 into units of consecutive time periods. Such a time period is exemplarily shown at 18 in Fig. 6. The encoder encodes the consecutive periods 18 of the audio content 10 using the same manner: that is, the encoder inserts into the data stream 12 a structure 20 per time period 18. In doing so, the encoder decomposes the audio content within the respective time period 18 into structure elements, the number and meaning/type of which are the same for each time period 18 and structure 20, respectively. With respect to the USAC codec described above, for example, the encoder encodes the same pair of audio signals 16 in every time period 18 into a channel pair element of the elements 22 of the structures 20, while using another coding principle, such as single channel coding for another audio signal 16 to obtain a single channel element 22, and so on. Parametric side information for obtaining an upmix of audio output signals from a downmix audio signal as defined by one or more structure elements 22 is collected to form another structure element within the structure 20. In this case, the structure element conveying this side information relates to, or forms a type of extension data for, other structure elements. Naturally, such extensions are not restricted to multi-channel or multi-object side information.

[0103] One possibility is to indicate within each structure element 22 what type the respective structure element is. Advantageously, such a procedure allows to handle future extensions of the data stream syntax. Decoders that are not able to handle certain structure element types would simply skip the respective structure elements within the data stream by exploiting the respective length information within these structure elements. Furthermore, it is possible to allow standard-compliant decoders of different types: some are able to understand a first set of types, while others understand and can deal with another set of types; alternative element types would simply be discarded by the respective decoders. Additionally, the encoder would be able to classify the structure elements at its discretion, such that decoders that are able to process such additional structure elements could be fed the structure elements within structures 20 in an order that, for example, minimizes buffer requirements within the decoder. Disadvantageously, however, the data stream would have to transmit the structure element type information per structure element, the necessity for which in turn negatively affects the compression ratio of the data stream 12 on the one hand and the decoding complexity on the other hand, since the additional analysis for inspection of the respective structure element type information takes place within each structure element.

[0104] Of course, it would be possible to adjust the order of the structure elements 22 in another way, such as by convention, but such a procedure prevents decoders from having the freedom to rearrange the structure elements due to, for example, specific properties of future extension structure elements, requiring or suggesting, for example, a different order among the structure elements.

[0105] Therefore, there is a need for another concept of a continuous data stream, encoder and decoder respectively.

[0106] Therefore, it is the object of the present invention to provide a continuous data stream, an encoder and a decoder that solve the aforementioned problem and allow obtaining a more efficient way of positioning the structure element.

[0107] This subject matter is met by the subject matter of the pending independent claims.

[0108] The present invention is based on the finding that a better compromise between a very high data stream and high decoding on the one hand and flexibility of structure element positioning on the other hand can be achieved if each of the sequence of structures of the data stream comprises a sequence of N structure elements and, on the other hand, the data stream comprises a configuration block comprising a field indicating the number of elements N and a type indication syntax portion indicating, for each element position of the sequence of N element positions, an element type from a plurality of element types with, in the sequences of N structure elements of the structures, each structure element being of the element type indicated, by the type indication portion, for a respective element position at which the respective structure element is positioned within the sequence of N structure elements of the respective structure in the data stream. Then, the structures are equally structured in that each structure comprises the same sequence of N structure elements of the structure element type indicated by the type indication syntax portion, positioned within the data stream in the same sequential order. This sequential order is generally adjustable for the sequence of structures by use of the type indication syntax portion that indicates, for each element position of the sequence of N element positions, one element type from a plurality of element types.

[0109] By this measurement, the structure element types may be arranged in any order, as at the encoder's discretion, to choose the order that is most appropriate for the structure element types used, e.g.

[0110] The plurality of structure element types may, for example, include a span element type with simply structure elements of the span element type containing the length information in the length of the respective structure element such that decoders that do not support the specific span element type may omit these structure elements of the span element type using the length information as an omission interval length. On the other hand, decoders capable of handling these structure elements of the span element type properly process the content or part of the payload thereof. Structure elements of other element types may not contain such length information. If, according to the specific application just mentioned, the encoder is capable of freely positioning these structure elements of the span element type within the sequence of structure elements of the structures, the overhead buffer in the decoders may be minimized by choosing the order of the structure element type appropriately and signaling the same within the type indication syntax portion.

[0111] Advantageous implementations of the applications of the present invention are the subject of the dependent claims.

[0112] In addition, preferred applications of the present invention are described below in relation to the figures, among which:

[0113] Fig. 1 shows a schematic block diagram of an encoder and its input and output according to an application;

[0114] Fig. 2 shows a schematic block diagram of a decoder and its input and output according to an application;

[0115] Fig. 3 schematically shows a continuous data flow according to an application;

[0116] Figs. 4a to z and za to zc show pseudocode tables illustrating a concrete syntax of the continuous data flow according to an application; and

[0117] Figs. 5a and b show a block diagram of a USAC encoder and decoder; and

[0118] Fig. 6 shows a typical encoder and decoder pair.

[0119] Fig. 1 shows an encoder 24 according to an application. The encoder 24 is used to encode an audio content 10 into a continuous data stream 12.

[0120] As described in the introductory part of the specification of the present application, the audio content 10 may be a conglomeration of several audio signals 16. The audio signals 16 represent, for example, individual audio channels of a spatial audio scenario. Alternatively, the audio signals 16 form audio objects of a set of audio objects together defining an audio scenario for free mixing on the decoder side. The audio signals 16 are defined on a common time base t as illustrated at 26. That is, the audio signals 16 may relate to the same time interval and may therefore be time aligned with respect to each other.

[0121] The encoder 24 is configured to encode the consecutive time periods 18 of the audio content 10 into a sequence of frames 20 such that each frame 20 represents a respective of one of the time periods 18 of the audio content 10. The encoder 24 is configured to somehow encode each time period in the same way such that each frame 20 comprises a sequence of an element number N of frame elements. Within each frame 20, it is true that each frame element 22 is of a respective type of a plurality of element types and that the frame elements 22 positioned at a given element position are of the same element type. That is, the first structure elements 22 in the structures 20 are of the same element type and form a first sequence (or sub-stream) of structure elements, the second structure elements 22 of all the structures 20 are of an equal element type to each other and form a second sequence of structure elements, and so on.

[0122] According to one embodiment, for example, the encoder 24 is configured such that the plurality of element types comprises the following:

[0123] (a) Frame elements of a single channel element type, for example, may be generated by encoder 24 to represent a single audio signal. Therefore, the sequence of frame elements 22 at a certain element position within frames 20, for example, the ith element frames with 0 > i > N+1, which in turn form the ith substream of frame elements, would together represent the consecutive time period 18 of such a single audio signal. The represented audio signal could therefore correspond directly to any of the audio signals 16 of the audio content 10. Alternatively, however, and as will be described in more detail below, such a represented audio signal may be of a channel of a downmix signal, which, together with the payload data of structure elements of another type of structure elements, positioned in another element position within the structures 20, produces a number of audio signals 16 of the audio content 10 that is greater than the number of channels of the just-mentioned downmix signal. In the case of the application described in more detail below, the structure elements of such a single-channel element are denoted UsacSingleChannelElement. In the case of MPEG Surround and SAOC, for example, there is only a single downmix signal which may be mono, stereo, or even multichannel in the case of MPEG Surround. In the latter case, for example, the 5.1 downmix consists of two-channel pair elements and a single channel element. In this case, the single channel element, as well as the two-channel pair elements, are only a part of the downmix signal. In the case of a stereo downmix box, a two-channel pair element will be used.

[0124] b) Structure elements of a channel pair element type may be generated by encoder 24 to represent a stereo pair of audio signals. That is, structure elements 22 of this type, which are positioned at a common element position within structures 20, would together form a respective substream of structure elements representing a consecutive time period 18 of such a stereo audio pair. The stereo pair of audio signals thus represented could directly be any pair of audio signals 16 of the audio content 10, or could represent, for example, a downmix signal, which together with the payload data of structure elements of another element type that are positioned at another element position results in a number of audio signals 16 of the audio content 10 that is greater than 2. In the application described in more detail below, structure elements of such a channel pair element type are denoted as UsacChannelPairElement.

[0125] c) To transmit information about audio signals 16 of the audio content 10 that require less bandwidth, such as subwoofer channels or the like, the encoder 24 may support structure elements of a specific type with structure elements of such a type that are positioned at a common element position, representing, for example, a consecutive time period 18 of a single audio signal. This audio signal may be any of the audio signals 16 of the audio content 10 directly, or it may be part of a downmix signal as described above with respect to the single channel element type and channel pair element type. In the application described in more detail below, structure elements of such a specific structure element type are denoted UsacLfeElement.

[0126] d) Structure elements of a type of extension element may be generated by the encoder 24 to transmit side information along with the continuous data stream to enable the decoder to upmix any of the audio signals represented by structure elements of any of the types a, b and/or c to obtain a higher number of audio signals. Structure elements of such a type of extension element, which are positioned at a certain common element position within the structures 20, would suitably transmit side information relating to the consecutive time period 18 that enables upmixing of the respective time period of one or more audio signals represented by any of the other structure elements to obtain the respective time period of a higher number of audio signals, where the latter may correspond to original audio signals 16 of the audio content 10. Examples for such side information may, for example, be parametric side information such as, for example, MPS or SAOC side information.

[0127] According to the application described in detail below, the available element types consist merely of the four element types described above, but other element types may be available as well. On the other hand, only one or two of the element types a to c may be available.

[0128] As has become clear from the above discussion, the omission of the structure elements 22 of the extension element type from the data stream 12 or the neglect of these structure elements in decoding does not render the reconstruction of the audio content 10 completely impossible: at least, the remaining structure elements of other element types would convey sufficient information to result in audio signals. These audio signals do not necessarily correspond to the original audio signals of the audio content 10 or a proper subset thereof, but may represent a kind of “amalgamation” of the audio content 10. That is, structure elements of the extension element type may convey information (payload data) that represents side information with respect to one or more structure elements positioned at different element positions within the structures 20.

[0129] In an application described below, however, structure elements of the extension element type are not restricted to such a type of side information transmission. Instead, structure elements of the extension element type are hereinafter denoted UsacExtElement and are defined to transmit payload data together with length information where the latter length information allows decoders receiving the continuous data stream 12 to omit these structure elements of the extension element type in the event that, for example, the decoder is unable to process the respective payload data within these structure elements. This is described in more detail below.

[0130] Before proceeding with the description of the encoder of Fig. 1, however, it should be noted that there are several possibilities for alternatives to the element types described above. This is especially true for the extension element type described above. In particular, in case the extension element type is configured such that the payload data thereof is omitted by decoders that are, for example, not capable of processing the respective payload data, the payload data of these structure elements of the extension element type could be any payload data types. These payload data could form side information with respect to the payload data of other structure elements of other structure element types, or could form independent payload data representing another audio signal, for example. Furthermore, even in the case where the payload data of the frame elements of the extension element type represents side information of payload data of frame elements of other frame element types, the payload data of these frame elements of the extension element type is not restricted to the type described above, namely multi-channel or multi-object side information. Multi-channel payload side information accompanies, for example, a downmix signal represented by any of the frame elements of another element type, with spatial cues such as binaural cue coding (BCC) parameters such as inter channel coherence (ICC) values, inter channel level differences (ICLD), and/or inter channel time differences (ICTD) and, optionally, channel loss coefficients, which parameters are known in the art from, for example, the MPEG Surround standard. The above mentioned spatial indicator parameters may, for example, be transmitted within the payload data of the structure elements of the extension element type in a time/frequency resolution, i.e. one parameter per time/frequency window of the time/frequency grid. In the case of multi-object side information, the payload data of the extension element type structure element may contain similar information such as inter-object cross-correlation (IOC) parameters, object level differences (OLD) as well as downmix parameters revealing how original audio signals have been downmixed into a channel(s) of a downmixed signal represented by any of the structure elements of another element type. Recent parameters are, for example, known in the art from the SAOC standard. However, an example of different side information that the payload data of the structure elements of the extension element type could represent is, for example, SBR data for parametrically encoding an envelope of a high frequency portion of an audio signal represented by any of the structure elements of the other structure element types positioned at different element positions within the structures 20 and enabling, for example, spectral band replication by using the low frequency portion as obtained from the last audio signal as the basis for the high frequency portion with them forming the envelope of the high frequency portion thus obtained by the envelope SBR data. More generally, the payload data of the structure elements of the extension element type could convey side information for modifying audio signals represented by structure elements of any of the other element types positioned at different element positions within the structure 20, whether in the time domain or in the frequency domain where the frequency domain may, for example, be a QMF domain or some other filter bank domain or transform domain.

[0131] Proceeding with the description of the functionality of the encoder 24 of Fig. 1, the same is configured to encode in the data stream 12 a configuration block 28 containing a field indicating the number of elements N and a type indication syntax portion indicating, for each element position of the sequence of element positions N, the respective element type. Therefore, the encoder 24 is configured to encode, for each frame 20, the sequence of N structure elements 22 in the data stream 12 such that each frame element 22 of the sequence of N structure elements 22, which is positioned at a respective element position within the sequence of N structure elements 22 in the data stream 12, is of the element type indicated by the type indication portion for the respective element position. In other words, the encoder 24 forms N substreams, each of which is a sequence of structure elements 22 of a respective element type. That is, for all these N substreams, the structure elements 22 are of the same element types, while structure elements of different substreams may be of a different element type. The encoder 24 is configured to multiplex all these structure elements into the data stream 12 by concatenating all the N structure elements of these substreams with respect to a common time period 18 to form a structure 20. Therefore, in the data stream 12 these structure elements 22 are arranged in structures 20. Within each structure 20, those representative of the N substreams, that is, the N structure elements with respect to the same time period 18, are arranged in the static sequential order defined by the sequence of element positions and the type indication syntax portion in the configuration block 28, respectively.

[0132] By use of the type indication syntax portion, the encoder 24 is able to freely select the order in which structure elements 22 of the N substreams are arranged within the structures 20. By this measure, the encoder 24 is able to keep, for example, the upper buffer of the decoder side as low as possible. For example, a substream of structure elements of the extension element type that transmits side information to structure elements of another substream (base substream), which are of a non-extension element type, can be positioned in an element position within the structures 20 immediately succeeding the element position in which these base substream structure elements are located in the structures 20. By this measure, the buffer time during which the decoding side has to store results, or intermediate results, of the decoding of the base substream for an application of the side information thereof, is kept low, and the upper buffer can be reduced. In the case of side information of the payload data of the structure elements of a substream, which are of the extension element type, being applied to an intermediate result, such as a frequency domain of the audio signal represented by another structure element substream 22 (base substream), positioning the structure element substream of the extension element type 22 such that it immediately follows the base substream not only minimizes the overhead buffer, but also the duration of time during which the decoder may have to interrupt further processing of the reconstruction of the represented audio signal because, for example, the payload data of the structure elements of the extension element type is to modify the reconstruction of the audio signal relative to the representation of the base substream. However, it may be advantageous to also position a dependent extension substream before the base substream representing an audio signal to which the extension substream refers. For example, the encoder 24 is free to position the extension payload substream within the upstream data stream relative to a channel element type substream. For example, the extension payload of substream i could carry dynamic range control (DRC) data and is transmitted prior to, or at an element position prior to i, relative to the encoding of the corresponding audio signal, such as via frequency domain (FD) coding, within the channel substream at element position i+1, for example. Then, the decoder is able to use DRC immediately when decoding and reconstructing the audio signal represented by a non-extension type substream i+1.

[0133] The encoder 24 as described thus far represents a possible application of the present application. However, Fig. 1 also shows a possible internal structure of the encoder which is to be understood merely as an illustration. As shown in Fig. 1, the encoder 24 may contain a distributor 30 and a sequencer 32 between which a plurality of encoding modules 34a-e are connected in the manner described in more detail in the following. In particular, the distributor 30 is configured to receive the audio signals 16 of the audio content 10 and to distribute the same into the individual encoding modules 34a-e. The manner in which the distributor 30 distributes the consecutive time period 18 of the audio signal 16 into the encoding modules 34a-34e is static. In particular, the distribution may be such that each audio signal 16 is routed to one of the encoding modules 34a-34e exclusively. An audio signal fed to the LFE encoder 34a is encoded by the LFE encoder 34a into a substream of structure elements 22 of type c (see above), for example. Audio signals fed to an input of the single-channel encoder 34b are encoded by the latter into a substream of structure elements 22 of type a (see above), for example. Similarly, a pair of audio signals fed to an input of the channel pair encoder 34c are encoded by the latter into a substream of structure elements 22 of type d (see above), for example. The above-mentioned encoding modules 34a to 34c are connected with an input and output thereof between the distributor 30 on the one hand and the sequencer 32 on the other hand.

[0134] However, as shown in Fig. 1 the inputs of encoder modules 34b and 34c are not simply connected to the output interface of distributor 30. Instead, it may be fed by an output signal from any of encoding modules 34d and 34e. The latter encoding modules 34d and 34e are examples of encoding modules that are configured to encode a number of input audio signals into a downmix signal of a smaller number of downmix channels on the one hand, and a substream of structure elements 22 of type d (see above) on the other hand. As is clear from the above discussion, encoding module 34d may be an SAOC encoder, and encoding module 34e may be an MPS encoder. The downmix signals are routed to any of encoding modules 34b and 34c. The substreams generated by the encoding modules 34a to 34e are forwarded to the sequencer 32 which sequences the substreams into the continuous data stream 12 as described above. Therefore, the encoding modules 34d and 34e have their input for the number of audio signals connected to the output interface of the distributor 30, while their substream output is connected to an input interface of the sequencer 32, and their downmix output is connected to the inputs of the encoding modules 34b and/or 34c, respectively.

[0135] It should be noted that according to the above description, the existence of multi-object encoders 34d and multi-channel encoders 34e was merely selected for illustrative purposes, and any of these coding modules 34d and 34e may be abandoned or replaced by another coding module, for example.

[0136] Having described the encoder 24 and the possible internal structure thereof, a corresponding decoder is described with reference to Fig. 2. The decoder of Fig. 2 is generally indicated with a reference signal 36 and has an input for receiving the continuous data stream 12 and an output for releasing a reconstructed version 38 of the audio content 10 or an amalgamation thereof. Therefore, the decoder 36 is configured to decode the data stream 12 containing the configuration block 28 and the sequence of frames 20 shown in Fig. 1, and to decode each frame 20 by decoding the frame elements 22 according to the element type indicated, by the type indicating part, to the respective element position at which the respective frame element 22 is positioned within the sequence of frame elements N 22 of the respective frame 20 in the data stream 12. That is, the decoder 36 is configured to assign each frame element 22 to one of the possible element types depending on its element position within the current frame 20 rather than on any information within the frame element itself. By this measure, the decoder 36 obtains N substreams, the first substream consisting of the first structure elements 22 of the structures 20, the second substream consisting of the second structure elements 22 within the structures 20, the third substream consisting of the third structure elements 22 within the structures 20, and so on.

[0137] Before describing the functionality of the decoder 36 with respect to the extension element type structure elements in more detail, a possible internal structure of the decoder 36 of Fig. 2 is explained in more detail to correspond to the internal structure of the encoder 24 of Fig. 1. As described with respect to the encoder 24, the internal structure should be understood to be merely illustrative.

[0138] In particular, as shown in Fig. 2, the decoder 36 may internally contain a distributor 40 and an arranger 42 between which the decoding modules 44a to 44e are connected. Each decoding module 44a to 44e is responsible for decoding a substream of structure elements 22 of a certain structure element type. Therefore, the distributor 40 is configured to distribute the N substreams of the continuous data stream 12 into the decoding modules 44a to 44e correspondingly. The decoding module 44a, for example, is an LFE decoder that decodes a substream of structure elements 22 of type c (see above) to obtain a narrow band (for example) audio signal at its output. Similarly, single channel decoder 44b decodes an input substream of structure elements 22 of type a (see above) to obtain a single audio signal at its output, and channel pair decoder 44c decodes an input substream of structure elements 22 of type b (see above) to obtain a pair of audio signals at its output. Decoding modules 44a to 44c have their input and output connected between the output interface of distributor 40 on the one hand and the input interface of arranger 42 on the other hand.

[0139] The decoder 36 may merely have decoding modules 44a through 44c. The other decoding modules 44e and 44d are responsible for extension element type structure elements and are therefore optional with respect to the concern for audio codec compliance. If both or either of these extension modules 44e and 44d are missing, the distributor 40 is configured to omit the respective structure element substreams in the data stream 12 as described in more detail below, and the reconstructed version 38 of the audio content 10 is merely an amalgamation of the original version having the audio signals 16.

[0140] If present, however, that is, if decoder 36 supports SAOC and/or MPS extension structure elements, multichannel decoder 44e may be configured to decode substreams generated by encoder 34e, while multi-object decoder 44d is responsible for decoding substreams generated by multi-object encoder 34d. Therefore, in the case of decoder module 44e and/or 44d being present, a switch 46 may connect the output of any of decoding modules 44c and 44b to an input downmix signal from decoding module 44e and/or 44d. Multichannel decoder 44e may be configured to upmix the downmix input signal using side information within the input substream from distributor 40 to obtain an increased number of audio signals at its output. The multi-object decoder 44d can act upon the difference that the multi-object decoder 44d treats the individual audio signals as audio objects while the multi-channel decoder 44e treats the audio signals at its output as audio channels.

[0141] The audio signals thus reconstructed are forwarded to arranger 42 which arranges them to form reconstruction 38. Arranger 42 may be further controlled by user input 48, which user input indicates, for example, an available speaker configuration or increased number of reconstruction channels 38 allowed. Depending on user input 48, arranger 42 may disable any of the decoding modules 44a to 44e such as, for example, any of the extension modules 44d and 44e, even if present and even if extension structure elements are present in data stream 12.

[0142] Before describing other possible details of the decoder, encoder and data stream, respectively, it should be noted that due to the ability of the encoder to interleave substream structure elements that are of the extension element type between substream structure elements that are not of the extension element type, the decoder upper buffer 36 can be reduced by the encoder 24 by appropriately choosing the order between the substreams and the order between the substream structure elements within each structure 20, respectively. Imagine, for example, that the decoder for the channel pair entering substream 44c would be placed in the first element position within structure 20, while the multi-channel substream for decoder 44e would be placed at the end of each structure. In this case, the decoder 36 would have to buffer the intermediate audio signal representing the downmix signal for the multichannel decoder 44e for a period of time involving the time between the arrival of the first frame element and the last frame element of each frame 20, respectively. Only then is the multichannel decoder 44e able to begin its processing. This delay can be avoided by the encoder 24 by arranging the dedicated substream for the multichannel decoder 44e in the second element position of frames 20, for example. On the other hand, the distributor 40 need not inspect each frame element with respect to its membership in any of the substreams. Instead, the distributor 40 is able to infer the membership of a current frame element 22 of a current frame 20 to any of the N substreams merely from the configuration block and the type indication syntax portion contained therein.

[0143] Reference is now made to Fig. 3 showing the data stream 12 which contains, as already described above, a configuration block 28 and a sequence of structures 20. The data stream portions on the right follow other positions of the data stream portion on the left when viewed in Fig. 3. In the case of Fig. 3, for example, the configuration block 28 precedes the structures 20 shown in Fig. 3 where, for illustrative purposes only, only three structures 20 are fully shown in Fig. 3.

[0144] Furthermore, it should be noted that configuration block 28 may be inserted into data stream 12 between frames 20 on a periodic or intermittent basis to allow for random access points in data stream transmission applications. Generally speaking, configuration block 28 may be a simply connected portion of data stream 12.

[0145] Configuration block 28 contains, as described above, a field 50 indicating the number of elements N, i.e., the number of structure elements N within each structure 20 and the number of multiplexed substreams in the data stream 12 as described above. In the following application describing an application for a concrete syntax of the data stream 12, field 50 is denoted numElements and configuration block 28 called UsacConfig in the following specific syntax example of Fig. 4a-z and za-zc. Furthermore, configuration block 28 contains type indication syntax portion 52. As already described above, this portion 52 indicates for each element position an element type from a plurality of element types. As shown in Fig. 3 and as is the case with respect to the following specific syntax example, the type indicating syntax portion 52 may contain the sequence of N syntax elements 54 with each syntax element 54 indicating the element type for the respective element position at which the respective syntax element 54 is positioned within the type indicating syntax portion 52. In other words, the in syntax element 54 within the portion 52 may indicate the element type of the in substream and in structure element of each structure 20, respectively. In the subsequent concrete syntax example, the syntax structure element is denoted UsacElementType. Although the type indicating syntax portion 52 may be contained within the data stream 12 as a contiguous or simply connected portion of the data stream 12, it is exemplarily shown in Fig. 3 that the elements 54 thereof are integrated with other syntax element portions of the configuration block 28 that are present for each of the N element positions individually. In the applications described below, these embedded syntax parts belong to the subflow-specific configuration data 55 meaning which is described below in more detail.

[0146] As already described above, each structure 20 is composed of a sequence of N structure elements 22. The element types of these structure elements 22 are not signaled by respective type indicators within the structure elements 22 themselves. Instead, the element types of the structure elements 22 are defined by their element position within each structure 20. The structure element 22 occurring first in the structure 20, denoted structure element 22a in Fig. 3, has the first element position and conforms to the element type that is indicated for the first element position by the syntax portion 52 within the configuration block 28. The same applies with respect to the following structure elements 22. For example, the structure element 22b occurring immediately after the first structure element 22a within the data stream 12, i.e., the one having element position 2, is of the element type indicated by the syntax portion 52.

[0147] According to a specific application, the syntax elements 54 are arranged within the data stream 12 in the same order as the structure elements 22 to which they refer. That is, the first syntax element 54, i.e., the one occurring first in the data stream 12 and being positioned on the outermost left side in Fig. 3, indicates the element type of the first occurring structure element 22a of each structure 20, the second syntax element 54 indicates the element type of the second structure element 22b, and so on. Naturally, the sequential order or arrangement of the syntax elements 54 within the data stream 12 and syntax parts 52 may be altered relative to the sequential order of the structure elements 22 within the structures 20. Other permutations would also be possible although less preferred.

[0148] For decoder 36, this means that it may be configured to read this sequence and N syntax elements 54 from type indicating syntax portion 52. To be more precise, decoder 36 reads field 50 such that decoder 36 knows the number of N syntax elements 54 to be read from data stream 12. As mentioned, decoder 36 may be configured to associate the syntax elements and the indicated element type, therefore, with structure elements 22 within structures 20 such that ith syntax element 54 is associated with ith structure element 22.

[0149] In addition to the above description, the configuration block 28 may contain the sequence 55 of N configuration elements 56 with each configuration element 56 containing configuration information for the element type for the respective element position at which the respective configuration element 56 is positioned in the sequence 55 of N configuration elements 56. In particular, the order in which the sequence of configuration elements 56 is written into the data stream 12 (and read from the data stream 12 by the decoder 36) may be the same order as that used for the structure elements 22 and/or syntax elements 54, respectively. That is, the configuration element 56 occurring first in the data stream 12 may contain the configuration information for the first structure element 22a, the second configuration element 56, the configuration information for the structure element 22b, and so on. As already mentioned above, the type indication syntax portion 52 and the element-position specific configuration data 55 are shown in the application of Fig. 3 as being interleaved with each other such that the configuration element 56 belonging to element position i is positioned in the data stream 12 between the type indicator 54 for element position i and element position i+1. Even in other words, the configuration elements 56 and the syntax elements 54 are arranged in the data stream alternately and read from it alternately by the decoder 36, but another positioning of these data in the data stream 12 within the block 28 would also be possible, as mentioned before.

[0150] By transmitting a configuration element 56 for each element position 1...N in the configuration block 28, respectively, the data stream allows to differently configure structure elements belonging to the different substreams and element positions, respectively, but being of the same element type. For example, a data stream 12 may contain two single channel substreams and therefore two single channel element type structure elements within each structure 20. The configuration information for both substreams may, however, be set differently in the data stream 12. This, in turn, means that the encoder 24 of Fig. 1 is enabled to differently set coding parameters within the configuration information for these different substreams and the single channel decoder 44b of the decoder 36 is controlled using these different coding parameters when decoding these two substreams. This is also true for the other decoding modules. More generally speaking, the decoder 36 is configured to read the sequence of N configuration elements 56 from the configuration block 28 and decode the structure element 22 according to the element type indicated by the ith syntax element 54, and using the configuration information contained by the ith configuration element 56.

[0151] For illustrative purposes, it is assumed in Fig. 3 that the second subflow, i.e., the subflow composed of the structure elements 22b occurring at the second element position within each structure 20, has an extension subflow element type composed of the structure elements 22b of the extension element type. Naturally, this is merely illustrative.

[0152] Furthermore, it is for illustrative purposes only that the data stream or configuration block 28 contains one configuration element 56 per element position not respective to the element type indicated by that element position per syntax portion 52. According to an alternative implementation, for example, there may be one or more element types for which no configuration element is contained by configuration block 28 such that, in the latter case, the number of configuration elements 56 within configuration block 28 may be less than N depending on the number of structure elements of such element types occurring in syntax portion 52 and structures 20, respectively.

[0153] In any case, Figure 3 shows an additional example for the construction of configuration elements 56 concerning the extension element type. In the specific syntax application subsequently explained, these configuration elements 56 are denoted as UsacExtElementConfig. For completeness only, it is noted that in the specific syntax application subsequently explained, the configuration elements for other element types are denoted as UsacSingleChannelElementConfig, UsacChannelPairElementConfig and UsacLfeElementConfig.

[0154] However, before describing a possible structure for the configuration element 56 for the extension element type, reference is made to the part of Figure 3 showing a possible structure of a structure element of the extension element type, here illustratively the second structure element 22b. As shown here, the structural elements of the extension element type may comprise a length information 58 in a length of the respective structure element 22b. The decoder 36 is configured to read from each structure element 22b of the extension element type of each structure 20 this length information 58. If the decoder 36 is not able to, or is instructed by the user not to, process the data stream to which the structure element of the extension element type belongs, the decoder 36 skips the structure element 22b using the length information 58 as the length of the skipped interval, i.e. the length of the part of the data stream that will be skipped. In other words, the decoder 36 may use the length information 58 to compute the number of bytes or any other suitable measure for defining the length of the interval of the data stream that will have to be skipped until accessing or visiting the next frame element within the current frame 20 or the beginning of the next following frame 20 in order to proceed with the next reading of the data stream 12.

[0155] As will be described in greater detail below, the structural elements of the extension element type may be configured to accommodate future and alternative extensions or developments of the audio codec, and accordingly the structural elements of the extension element type may have different statistical length distributions. In order to take advantage of the possibility that, in accordance with some applications, the structural elements of the extension element type of certain substreams are of constant length or have a rather narrow statistical length distribution, in accordance with some applications of the present application, the configuration elements 56 for the extension element type may comprise standard payload length information 60 as shown in Figure 3. In that case, it is possible for the structural elements 22b of the extension element type of the respective data stream to refer to this payload length information 60 contained within the respective configuration element 56 for the respective bit rate, rather than explicitly transmitting the payload length. Specifically, as shown in Figure 3 , in that case, the length information 58 may comprise the conditional syntax portion 62 in the form of the default extension payload length flag 64 followed, if the default payload length flag 64 is not set, by an extension payload length value 66. Any structure element 22b of the extension element type has a default extension payload length as indicated by the information 60 in the corresponding configuration element 56 in the case that the default extension payload length flag 64 of the length information 62 of the respective structure element 22b of the extension element type is set, and has an extension payload length corresponding to the extension payload length value 66 of the length information 58 of the respective structure element 22b of the extension element type in the case that the default extension payload length flag 64 of the length information 58 of the respective structure 22b of the extension element type is not set. That is, explicit encoding of the length value of the extension payload 66 may be avoided by the encoder 24 whenever possible, to merely refer to the length of the standard extension payload as indicated by the length information of the standard payload 60 within the configuration element 56 of the corresponding data stream and an element position, respectively. The decoder 36 acts as follows. It reads the length information of the standard payload 60 while reading the configuration elements 56. When reading the structural elements 22b of the corresponding data stream, the decoder 36, in reading the length information of these structural elements, reads the length flag of the standard extension payload 64 and checks whether it is set or not. If the length flag of the standard payload 64 is not set, the decoder proceeds with reading the length value of the length of the extension payload 66 from the conditional syntax portion 62 of the data stream so as to obtain the length of the length of the extension payload of the respective structure element. However, if the default payload flag 64 is set, the decoder 36 sets the length of the extension payload of the respective frame to be equal to the length of the default extension payload as derived from information 60. Skipping the decoder 36 may then involve skipping a section of payload 68 of the current frame element using the newly determined length of the extension payload as the skipped interval length, i.e., the length of a portion of the continuous data stream 12 that will be skipped in order to access the next frame element 22 of the current frame 20 or the beginning of the next frame 20.

[0156] Accordingly, as described above, repeated transmission of the payload length structure type of structure elements of a certain subflow extension element type may be avoided using a signaling mechanism 64 whenever the payload length variety of these structure elements is quite low.

[0157] However, since it is not a priori clear whether the payload carried by the structure elements of an extension element type of a certain subflow has such statistics concerning the payload length of the structure elements, and consequently, whether it is worthwhile to transmit the default payload length explicitly in the configuration element of such a subflow of the structure elements of the extension element type, in accordance with the further implementation, the default payload length information 60 is also implemented by a conditional syntax portion comprising a flag 60a called UsacExtElementDefaultLengthPresent in the following specific syntax example, and indicating whether an explicit transmission of the default payload length takes place or not. Merely if defined, the conditional syntax portion comprises the explicit transmission 60b of the default payload length called UsacExtElementDefaultLength in the following specific syntax example. In contrast, the default payload length is set by default to 0. In the latter case, bit consumption in the data stream that is saved as an explicit transmission of the default payload length is avoided. That is, the decoder 36 (and the distributor 40, which is responsible for all the reading procedures described above and hereinafter), can be configured to, upon reading the information of the default payload length 60, read a flag present in the default payload length 60a of the data stream 12, check whether or not the flag present in the default payload length 60a is set, and if the flag present in the default payload length 60a is set, set the default extension payload length to zero, and if the flag present in the default payload length 60a is not set, explicitly read the default extension payload length 60b of the data stream 12 (namely, the field 60b following the flag 60a).

[0158] Additionally, or alternatively to the standard payload mechanism, the length information 58 may comprise a flag present in the extension payload 60 of which the length information 58 is not defined, merely consists of the flag present in the extension payload and that is it. That is, there is no payload section 68. On the other hand, the length information 58 of any structure element 22b of the extension element type, the flag present in the payload data 70 of the length information 58 of which is configured, further comprises a syntax portion 62 or 66 indicating the extension payload length of the respective structure 22b, i.e. the length of its payload section 68. In addition to the default payload length mechanism, i.e. in combination with the default extension payload length flag 64, the flag present in the extension payload 70 allows each structure element of the extension element type to have two effectively encodeable payload lengths, namely 0 on the one hand and default payload length, i.e. the most probable payload length, on the other hand.

[0159] In parsing or reading the information of the length 58 of the current structure element 22b of the extension element type, the decoder 36 reads the flag present in the extension payload 70 of the data stream 12, checks whether the flag present in the extension payload 70 is set, and if the flag present in the extension payload 70 is not set, stops reading the respective structure element 22b, and continues with another reading, the next structure element 22 of the current structure 20 or starts with the reading or parsing of the next structure 20. Considering whether the flag present in the payload 70 is set, the decoder 36 reads the syntax portion 62 or at least the part 66 (if the flag 64 is non-existent, since this mechanism is not available) and ignores, if the payload of the current structure element 22 has to be ignored, the section 68 using the extension payload length of the respective structure element 22b of the span element type as the ignored range length.

[0160] As described above, structure elements of the extension element type may be provided in order to accommodate future extensions of the audio codec or alternative extensions for which the current decoder is not suitable, and consequently structure elements of the extension element type should be configurable. Specifically, in accordance with an implementation, the configuration block 28 comprises, for each element position for which the type indicating part 52 indicates the extension element type, a configuration element 56 comprising configuration information for the extension element type, characterized in that the configuration information comprises, in addition to or alternatively to the components highlighted above, an extension element type field 72 indicating a payload data type from a plurality of payload data types. The plurality of payload data types, in accordance with an implementation, comprises a multi-channel side information type and a multi-object encoder side information type in addition to other data types that are, for example, reserved for future developments. Depending on the specified payload data type, the configuration element 56 additionally contains the specific configuration data of the payload data type. Accordingly, the structure elements 22b in the corresponding element position and the respective substream respectively carry in their payload sections 68 the payload data corresponding to the specified payload data type. In order to allow an adaptation of the length of the payload data type-specific configuration data 74 to the payload data type, and to allow a reserve for future developments of additional payload data types, the specific syntax implementations described below present the extension element type configuration elements 56 additionally comprising a configuration element length value called UsacExtElementConfigLength so that decoders 36 that are not informed on the indicated payload data type and data for the current substream, are able to bypass the configuration element 56 and its payload data type-specific configuration data 74 to access the immediately following part of the continuous data stream 12, such as the element type syntax element 54 of the next element position (or in the alternative implementation not shown, the configuration element of the next element position) or the beginning of the first structure following the configuration block 28 or some other data, as will be shown respective to Figure 4a. Specifically, in the following specific application for a syntax, the multi-channel side information configuration data is contained within SpatialSpecificConfig, while the multi-object type configuration data is contained within SaocSpecificConfig.

[0161] In accordance with the latter aspect, the decoder 36 would be configured to, upon reading the configuration block 28, perform the following steps for each element or substream position for which the type indicating portion 52 indicates the extension element type:

[0162] Reading the configuration element 56, including reading the extension element type field 72 indicating the payload data type of the plurality of available payload data types,

[0163] If the extension element type field 72 indicates the multi-channel side information type, reading the multi-channel side information configuration data 74 as part of the configuration information from the data stream 12, and if the extension element type field 72 indicates the multi-object side information type, reading the multi-object side information configuration data 74 as part of the configuration information from the data stream 12.

[0164] Then, when decoding the corresponding structure elements 22b, that is, those of the corresponding element position and the substream, respectively, the decoder 36 would configure the multi-channel decoder 44e using the multi-channel side information configuration data 74, while feeding the thus configured multi-object decoder 44d with the payload data 68 of the respective structure elements 22b as the multi-channel side information, in the case of the payload data type indicating the multi-channel side information type, and decode the corresponding structure elements 22b by configuring the multi-object decoder 44d using the multi-object side information configuration data 74, and feeding the thus configured multi-object decoder 44d with the payload data 68 of the respective structure element 22b, in the case of the payload data type indicating the multi-object side information type.

[0165] However, if an unknown payload data type is indicated by field 72, decoder 36 would ignore the configuration data of payload data type 74 using the aforementioned configuration of the configuration length value also comprised by the current configuration element.

[0166] For example, the decoder 36 could be configured to, for any element position for which the type indicating portion 52 indicates the extension element type, read a configuration data length field 76 from the data stream 12 as part of the configuration element configuration information 56 for the respective element position so as to obtain a configuration data length, and check whether the payload data type indicated by the extension element type field 72 of the configuration element configuration information for the respective element position belongs to the predetermined set of payload data types being a subset of the plurality of payload data types. If the payload data type indicated by an extension element type field 72 of the configuration information of the configuration element for the respective element position belongs to the predetermined set of data types, the decoder 36 would read the payload data depending on the configuration data 74 as part of the configuration information of the configuration element for the respective element position from the data stream 12, and decode the frame elements of the extension element type at the respective element position in the frames 20 using the payload data dependent on the configuration data 74. But, if the payload data type indicated by the extension element type field 72 of the configuration information of the configuration element for the respective element position does not belong to the predetermined set of payload data types, the decoder will ignore the payload data of data dependent on the configuration data 74 using the configuration data length, and ignore the frame elements of the extension element type at the respective frame element position 20. using the length information of 58 in it.

[0167] In addition to, or alternatively to, the above mechanisms, the structure elements of a given substream may be configured to be transmitted in fragments rather than one in each complete frame. For example, the extension element type configuration elements could comprise a fragmentation usage flag 78, the decoder may be configured to, upon reading structure elements 22 positioned at any element position for which the chunk type indication indicates the extension element type, and for which the fragmentation usage flag 78 of the configuration element is defined, read an information fragment 80 from the continuous data stream 12, and use the fragment information to put the payload data of those structure elements of the consecutive frames together. In the following specific syntax example, each extension-type structure element of a subflow for which the usage flag for fragmentation 78 is defined comprises a pair of a start flag indicating the start of a payload of the subflow and an end flag indicating the end of a payload item of the subflow. These flags are called usacExtElementStart and usacExtElementStop in the following specific syntax example.

[0168] Furthermore, in addition to or alternatively to the above mechanism, the same variable length code may be used to read the length information 80, the extension element type field 72, and the configuration data length field 76, thus decreasing the complexity to implement the decoder, for example, and saving the bits requiring additional bits merely in rare occurrence cases, such as future extension element types, larger extension element type lengths, and so on. In the specific example explained below, this VCL code is derivable from Figure 4m.

[0169] Summarizing the above, the following can be applied to the functionality of the decoder: (1) Reading the configuration block 28, and (2) Reading/parsing the sequence of structures 20. Steps 1 and 2 are performed by the decoder 36 and, more precisely, the distributor 40. (3) A reconstruction of the audio content is restricted to those substreams, i.e., to those sequences of structure elements at the element positions, the decoding of which is supported by the decoder 36. Step 3 is performed within the decoder 36, for example, in the decoding modules (see Figure 2).

[0170] Accordingly, in step 1 the decoder 36 reads the substream number 50 and the number of structure elements 22 per structure 20, respectively, as well as the element syntax portion 52 revealing the element type of each of these substreams and element positions, respectively. For analysis of the data stream in step 2, the decoder 36 then critically reads the structure elements 22 of the sequence of structures from the data stream 12. In doing so, the decoder 36 ignores the structure elements, or the remaining parts/payload thereof, using the length information 58 as has been described above. In the third step, the decoder 36 performs reconstruction by decoding the structure elements not having to be ignored.

[0171] In deciding in step 2 which of the element and substream positions to ignore, the decoder 36 may inspect the configuration elements 56 within the configuration block 28. In order to do so, the decoder 36 may be configured to critically read the configuration elements 56 from the configuration block 28 of the data stream 12 in the same order as used for the element type indicators 54, and the structure elements 22 themselves. As denoted above, the cyclic reading of the configuration elements 56 may be interleveled with the cyclic reading of the syntax elements 54. Specifically, the decoder 36 may inspect the extension element type field 72 within the configuration elements 56 of the extension element type substreams. If the extension element type is not a supported type, the decoder 36 ignores the respective substream, and the structure elements 22 corresponding to the respective positions of the structure elements within the structures 20.

[0172] In order to facilitate the bit rate required for transmission of the length information 58, the decoder 36 is configured to inspect the configuration elements 56 of the substreams of the type of extension elements, and specifically the default payload length information 60 contained in step 1. In the second step, the decoder 36 inspects the length information 58 of the extension structure elements 22 that will be ignored. Specifically, first, the decoder 36 inspects the signaling 64. If defined, the encoder 36 uses the default length indicated for the respective substream by the default payload length information 60 as the remaining payload length that will be ignored in order to continue with the cyclic reading/parsing of the structure elements of the structures. If flag 64, however, is not set, then decoder 36 explicitly reads the payload length 66 from the continuous data stream 12. Although not explicitly explained above, it should be made clear that decoder 36 may derive the number of bits or bytes that will be skipped in order to access the next structure element of the current or next structure by some additional computation. For example, decoder 36 may take into account whether the fragmentation mechanism is enabled or not, as explained above with respect to flag 78. If it is enabled, decoder 36 may take into account that the structure elements of the substream with flag 78 set in any case have fragmentation information 80 and that consequently the payload data 68 starts later than it would in the case that fragmentation flag 78 is not set.

[0173] When decoding in step 3, the decoder acts as usual: that is, the individual substreams are subjected to the decoding engines or decoding modules as shown in Figure 2, characterized in that some substreams can form the side information for other substreams, as has been explained above with respect to the specific examples of extension substreams.

[0174] With respect to other possible details concerning the functionality of the decoders, reference is made to the above discussion. For the sake of completeness only, it is noted that the decoder 36 may skip further parsing of the configuration elements 56 in step 1, namely for those element positions that will be skipped, e.g. the extension element type indicated by field 72 is not suitable for a supported set of extension element types. Then, the decoder 36 may use the configuration length information 76 in order to skip respective configuration elements in cyclic reading/parsing in the configuration elements 56, i.e. skipping the respective number of bits/bytes in order to access the next syntax element of the continuous data stream such as the next element position indicator 54.

[0175] Before proceeding with the specific syntax application mentioned above, it should be noted that the present invention is not restricted to be implemented with unified speech and audio coding and its facets, such as core coding switching using a mixture or switching between AAC-type frequency domain coding and LP coding using parametric coding (ACELP) and transform coding (TCX). Rather, the above-mentioned substreams represent audio signals using any coding scheme. Furthermore, although in the specific syntax application described below it is assumed that SBR is an coding option of the main codec used to represent audio signals using single channel and dual channel element type substreams, SBR may also not be a possibility of the latter element types, but merely usable using extension element types.

[0176] In the following, a specific syntax example for the data stream 12 is explained. It should be noted that the specific syntax example represents a possible implementation for the application of Figure 3, and the correspondence between the syntax elements of the following syntax and the structure of the data stream of Figure 3 is indicated or derivable from the respective observations in Figure 3, as well as the description of Figure 3. The basic aspects of the following specific example are now highlighted: In this regard, it should be noted that any additional details beyond those already described above with respect to Figure 3 are not to be understood as a possible extension of the application of Figure 3. All these extensions can be individually constructed within the application of Figure 3. As a final preliminary note, it should be understood that the specific syntax example described below explicitly refers to the decoder and encoder environment of Figures 5a and 5b, respectively.

[0177] High-level information, such as the sample rate, exact channel configuration, about the contained audio content is present in the audio data stream. This makes the data stream more self-contained and makes transport of the configuration and payload easier when integrated into transport schemes, which may not have means to explicitly convey this information.

[0178] The configuration structure contains a combined frame length and an SBR sampling rate ratio index (coreSbrFrameLengthIndex). This ensures efficient transmission of both values and ensures that non-meaningful combinations of frame length and SBR rate cannot be signaled. The latter simplifies the implementation of a decoder.

[0179] The configuration may be extended by means of a specific configuration extension mechanism. This will avoid the bulky and inefficient transmission of configuration extensions as known from MPEG-4 AudioSpecificConfig().

[0180] The configuration allows for free signaling of the speaker positions associated with each transmitted audio channel. The channel signaling commonly used for speaker mappings can be efficiently signaled via a channelConfigurationIndex.

[0181] The configuration of each channel element is contained in a separate structure such that each channel element can be configured independently.

[0182] SBR configuration data (the “SBR header”) is split into an SbrInfo() and an SbrHeader(). For the SbrHeader(), a standard version is defined (SbrDfltHeader()), which can be efficiently referenced in the data stream. This reduces bit demand at locations where retransmission of SBR configuration data is required.

[0183] Configuration changes most commonly applied to SBR can be efficiently signaled with the help of the SbrInfo() syntax element.

[0184] The configuration for parametric bandwidth extension (SBR) and parametric stereo coding tools (MPS212, also known as MPEG Surround 2-1-2) are fully integrated into the USAC configuration framework. This better represents how both technologies are actually employed in the standard.

[0185] The syntax has an extension mechanism that allows existing and future extensions to be passed to the codec.

[0186] Extensions may be placed (i.e. interleaved) with the channel elements in any order. This allows extensions to be read before or after a specific channel element to which the extension is to be applied.

[0187] A default length can be defined by a syntax extension, which makes transmitting constant length extensions very efficient, because the length of the payload extension does not need to be transmitted every time.

[0188] The most common case of signaling a value with the help of an escape mechanism to extend the value range if necessary has been modularized into a specific genuine syntax element (escapedValue()), which is flexible enough to cover all desired escape value constellations, and bit field extensions. Data Stream Configuration UsacConfig() (Fig. 4a)

[0189] UsacConfig() has been extended to contain information about the contained audio content, as well as everything needed for full decoder configuration. High-level information about the audio (sampling rate, channel configuration, output frame length) is gathered at the beginning to facilitate access from higher (application) layers. UsacChannelConfig() (Fig. 4b)

[0190] These elements provide information about the elements of the contained data stream and their mapping to the loudspeakers. The channelConfigurationIndex allows an easy and convenient way of signaling one of a number of predefined mono, stereo or multichannel configurations that have been found to be practically relevant.

[0191] For more elaborate configurations that are not covered by channelConfigurationIndex UsacChannelConfig() allows free assignment of elements to the speaker position from a list of 32 speaker positions, which cover all currently known speaker positions in all known speaker configurations for home or cinema sound reproduction.

[0192] This list of speaker positions is a superset of the list given in the MPEG Surround Standard (see Table 1 and Figure 1 in ISO/IEC 23003-1). Four speaker positions have been added to cover the newly introduced 22.2 speaker configuration (see Figures 3a, 3b, 4a and 4b). UsacDecoderConfig() (Fig. 4c)

[0193] This element is the heart of the decoder configuration and as such contains all other information needed by the decoder to interpret the continuous data stream.

[0194] Specifically, the structure of the data stream is defined here by explicitly stating the number of elements and their order in the data stream.

[0195] A circuit over all elements then allows the configuration of all elements of all types (single, double, lfe, extension). UsacConfigExtension() (Fig. 4l)

[0196] To consider future extensions, the configuration has a powerful mechanism to extend the configuration of configuration extensions not yet existing for USAC. UsacSingleChannelElementConfig() (Fig. 4d)

[0197] This element configuration contains all the information needed to configure the decoder to decode a single channel. This is essentially the information related to the core encoder and, if SBR is used, the information related to SBR. UsacChannelPairElementConfig() (Fig. 4e)

[0198] Analogously to the above, this element configuration contains all the information needed to configure the decoder to decode a dual channel. In addition to the above-mentioned core configuration and the SBR configuration, it includes stereo-specific settings such as the exact type of stereo encoding applied (with or without MPS212, residual, etc.). Note that this element covers all types of stereo encoding options available in USAC. UsacLfeElementConfig() (Fig. 4f)

[0199] The LFE element configuration does not contain configuration data as an LFE element has a static configuration. UsacExtElementConfig() (Fig. 4k)

[0200] This element configuration can be used for any type of existing or future extension configuration for the codec. Each extension element type has its own specific ID value. A length field is included in order to conveniently ignore configuration extensions unknown to the decoder. Optionally defining a default payload length further increases the efficiency of encoding the payload extension present in the actual data stream.

[0201] Extensions that are already planned to be combined with USAC include: MPEG Surround, SAOC, and some kind of FIL element as known from MPEG-4 AAC. UsacCoreConfig() (Fig. 4g)

[0202] This element contains configuration data that impacts the core encoder configuration. Currently, these are switches for the time warp tool and the noise fill tool. SbrConfig() (Fig. 4h)

[0203] In order to reduce the bit overhead produced by frequent retransmission of sbr_header(), default values for sbr_header() elements, which are normally kept constant, are now realized in the SbrDfltHeader() configuration element. In addition, static SBR configuration elements are also realized in SbrConfig(). These static bits include flags for enabling or disabling advanced SBR-specific features, such as harmonic transposition or Inter TES. SbrDfltHeader() (Fig. 4i)

[0204] This takes the elements of sbr_header() that are typically held constant. Elements that affect things like amplitude resolution, intersection bandwidth, and pre-leveling spectrum are now held in SbrInfo(), which allows them to be efficiently changed on the fly. Mps212Config() (Fig. 4j)

[0205] Similar to the SBR configuration above, all configuration parameters for the MPEG Surround 21-2 tools are assembled in this configuration. All elements of SpatialSpecificConfig() that are not relevant or redundant in this context have been removed. UsacFrame() Data Stream Payload (Fig. 4n)

[0206] This is the outermost wrapper around the USAC payload data stream and represents a USAC access unit. It contains circuitry over all the constant channel elements and the extension elements as stated in the configuration part. This makes the data stream format much more flexible in terms of what it can contain, and is future-proof for any future extensions. UsacSingleChannelElement() (Fig. 4o)

[0207] This element contains all the data for decoding a mono stream. The content is split into a core encoder-related part and a ESBR-related part. The latter is now much more closely coupled to the core, which also reflects much better the order in which the data is needed by the decoder. UsacChannelPairElement() (Fig. 4p)

[0208] This element comprises data for all possible ways to encode dual stereo. Specifically, all types of unified stereo coding are covered, ranging from legacy M/S-based coding to full parametric stereo coding with the help of MPEG Surround 2-1-2. The stereoConfigIndex indicates which type is actually used. The appropriate eSBR data and MPEG Surround 2-1-2 data are sent in this element. UsacLfeElement() (Fig. 4q)

[0209] The old lfe_channel_element() has been renamed just to follow a consistent naming scheme. UsacExtElement() (Fig. 4r)

[0210] The extension element has been carefully designed to be maximally flexible, yet maximally performant, even for extensions that have a small payload (or often none at all). The length of the extension payload is flagged to foolish decoders to ignore it. User-defined extensions can be flagged via a range of reserved extension types. Extensions can be placed freely in the element order. A range of extension elements have already been considered, including a mechanism for writing padding bytes. UsacCoreCoderData() (Fig. 4s)

[0211] This new element summarizes all the information affecting the core encoders and therefore also contains the information from fd_channel_stream() and lpd_channel_stream(). StereoCoreToolInfo() (Fig. 4t)

[0212] To facilitate syntax readability, all stereo-related information has been captured in this element. It handles the numerous bit dependencies in stereo encoding modes. UsacSbrData() (Fig. 4x)

[0213] The functionality and legacy CRC of the scalable audio coding descriptive elements has been removed from what used to be the sbr_extension_data() element. In order to reduce the overhead caused by frequently resending SBR information and header data, their presence can be explicitly signaled. SbrInfo() (Fig. 4y)

[0214] SBR configuration data that is frequently changed on the fly. This includes elements that control things like amplitude resolution, intersection bandwidth, pre-leveling spectrum, which previously required the transmission of a full sbr_header(). (see 6.3 in [N11660], “Efficiency”). SbrHeader() (Fig. 4z)

[0215] In order to keep SBR's ability to change values in sbr_header() working, it is now possible to perform a SbrHeader() inside UsacSbrData(), in case other values than those sent in SbrDfltHeader() need to be used. The bs_header_extra mechanism has been kept, in order to keep the overhead as low as possible for the most common cases. sbr_data() (Fig. 4za)

[0216] Again, SBR scalable encoding remainders have been removed because they are not applicable in the USAC context. Depending on the number of channels sbr_data() contains either a sbr_single_channel_element() or a sbr_channel_pair_element(). usacSamplingFrequencyIndex

[0217] This table is a superset of the table used in MPEG-4 to indicate the sampling frequency of the audio codec. The table has been extended to also cover the sampling rates that are currently used in USAC operating modes. Some multiples of the sampling frequencies have also been added. channelConfigurationIndex

[0218] This table is a superset of the table used in MPEG-4 to signal the channelConfiguration. It has been extended to allow signaling of future viewing installations of commonly used loudspeakers. The index into this table is signaled with 5 bits to allow future extensions. usacElementType

[0219] There are only 4 element types. One for each of the four basic data stream elements: UsacSingleChannelElement(), UsacChannelPairElement(), UsacLfeElement(), UsacExtElement(). These elements provide the necessary high-level structure while maintaining all the necessary flexibility. usacExtElementType

[0220] Within UsacExtElement(), this element allows signaling of a multitude of extensions. In order to be future-proof, the bit field was chosen large enough to allow all possible extensions. Out of the currently known extensions, a few are already proposed for consideration: padding element, MPEG Surround, and SAOC. usacConfigExtType

[0221] Should it be necessary at some point to extend the configuration, then this can be handled via UsacConfigExtension() which would then allow a type to be assigned to each new configuration. Currently the only type that can be flagged is a padding mechanism for the configuration. coreSbrFrameLengthIndex

[0222] This table shall flag multiple configuration aspects of the decoder. Specifically, these are the output frame length, the SBR range and the resulting core encoder frame length (CCFL). At the same time, it indicates the number of QMF analysis and synthesis bands used in SBR stereoConfigIndex

[0223] This table determines the internal structure of a UsacChannelPairElement(). It indicates whether a mono or stereo core is used, whether MPS212 is used, whether stereo SBR is applied, and whether residual coding is applied in MPS212.

[0224] By moving large parts of the ESBR header fields to a standard header that can be referenced via a standard header flag, the bit demand for sending ESBR control data has been greatly reduced. Former sbr_header() bit fields that were considered to be more likely to change in a real-world system have been outsourced to the sbrInfo() element since it now consists of only four elements covering a maximum of 8 bits. Compared to sbr_header(), which consists of at least 18 bits, this represents a saving of 10 bits.

[0225] It is more difficult to assess the impact of this change on the total bit rate because it depends heavily on the data transmission rate of the ESBR control in sbrInfo(). However, for the common use case where the SBR intersection is changed in a continuous data stream, the bit savings can be as high as 22 bits per occurrence by sending an sbrInfo() instead of a fully transmittable sbr_header().

[0226] The output of the USAC decoder may be further processed by MPEG Surround (MPS) (ISO/IEC 230031), or SAOC (ISO/IEC 23003-2). If the SBR tool in USAC is active, a USAC decoder can typically be efficiently combined with a subsequent MPS/SAOC decoder by connecting them in the QMF domain in the same way as described for HE-AAC in ISO/IEC 23003-1 4.4. If a connection in the QMF domain is not possible, they need to be connected in the time domain.

[0227] If MPS/SAOC side information is embedded in a continuous USAC data stream via the usacExtElement mechanism (with usacExtElementType being ID_EXT_ELE_MPEGS or ID_EXT_ELE_SAOC), the alignment time between the USAC data and the MPS/SAOC data assumes the most efficient connection between the USAC decoder and the MPS/SAOC decoder. If the SBR tool in the USAC is active, and if the MPS/SAOC employs a QMF 64 band domain representation (see ISO/IEC 23003-1 6.6.3), the most efficient connection is in the QMF domain. Otherwise, the most efficient connection is in the time domain. This corresponds to the alignment time for the combination of HE-AAC and MPS as defined in ISO/IEC 230031 4.4, 4.5 and 7.2.1.

[0228] The additional delay introduced by adding MPS decoding after USAC decoding is given by ISO/IEC 23003-1 4.5 and depends on whether either the HQ MPS or the LP MPS is used, and whether the MPS is connected to the USAC in the QMF domain or in the time domain.

[0229] ISO/IEC 23003-1 4.4 clarifies the interface between USAC and MPEG systems. Each access unit delivered to the audio decoder from the systems interface shall result in a corresponding composition unit delivered from the audio decoder to the systems interface, i.e. the compositor. This shall include the start-up and stop-down conditions, i.e. when the access unit is the first or last of a finite sequence of access units.

[0230] For an audio composition unit, ISO/IEC 14496-1 7.1.3.5 Composition Time Stamp [CTS] specifies that the composition time applies to the n-th audio sample within the composition unit. For USAC, the value of n is always 1. Note that this applies to the output of the USAC decoder itself. In the case where a USAC decoder, for example, is being combined with an MPS decoder, account needs to be taken of the composition units delivered at the output of the MPS decoder.

[0231] If MPS/SAOC side information is integrated into a continuous USAC data stream via the usacExtElement mechanism (with the usacExtElementType being ID_EXT_ELE_MPEGS or ID_EXT_ELE_SAOC), the following restrictions may optionally be applied:

[0232] The MPS/SAOC sacTimeAlign parameter (see ISO/IEC 23003-1 7.2.5) must have the value 0.

[0233] The MPS/SAOC sampling frequency must be the same as the USAC sampling output frequency.

[0234] The MPS/SAOC bsFrameLength parameter (see ISO/IEC 23003-1 5.2) must have one of the permitted values from the predetermined list.

[0235] The syntax of the USAC payload data stream is shown in Figures 4n through 4r, and the syntax of payload auxiliary elements is shown in Figures 4s-w and the enhanced SBR payload syntax is shown in Figures 4x through 4zc.

Brief Description of Data Elements

[0236] UsacConfig() This element contains information about the contained audio content, as well as everything needed for complete decoder configuration.

[0237] UsacChannelConfig() This element provides information about the elements of the contained data stream and their mapping to speakers.

[0238] UsacDecoderConfig() This element contains all the information needed by the decoder to interpret the data stream. Specifically, the SBR resampling ratio is signaled here, and the structure of the data stream is defined here by explicitly specifying the number of elements and their order in the data stream.

[0239] UsacConfigExtension() The configuration extension mechanism to extend the configuration for future configuration extensions to USAC.

[0240] UsacSingleChannelElementConfig() Contains all the information needed to configure the decoder to decode a single channel. This is essentially the core encoder related information and, if SBR is used, the SBR related information.

[0241] UsacChannelPairElementConfig() Analogously to the above, this element configuration contains all the information needed to configure the decoder to decode a dual channel. In addition to the above mentioned core configuration and the sbr configuration, it includes stereo specific settings such as the exact type of stereo encoding applied (with or without MPS212, residual, etc.) This element covers all types of stereo encoding options available in USAC.

[0242] UsacLfeElementConfig() The LFE element configuration does not contain configuration data, as an LFE element has a static configuration.

[0243] UsacExtElementConfig() This element configuration can be used for any type of existing or future extension configuration for the codec. Each extension element type has its own specific type value. A length field is included in order to ignore configuration extensions unknown to the decoder.

[0244] UsacCoreConfig() Contains configuration data that impacts the core encoder configuration.

[0245] SbrConfig() Contains default values for sSRR configuration elements that are typically held constant. In addition, static SBR configuration elements are also held in SbrConfig(). These static bits include enabling or disabling flags for features specific to Advanced SBR, such as Harmonic Transposition or Inter TES.

[0246] SbrDfltHeader() This element realizes a standard version of the SbrHeader() elements that can be referred to if no differentiating value for these elements is desired.

[0247] Mps212Config() All configuration parameters for the MPEG Surround 2-1-2 tools are assembled into this configuration.

[0248] escapedValue() This element implements a general method for transmitting an integral value using a varying number of bits. It has a two-level escape mechanism, which allows the range of representable values to be extended by successively transmitting additional bits.

[0249] usacSamplingFrequencyIndex This index determines the sampling frequency of the audio signal after decoding. The value of usacSamplingFrequencyIndex and its associated sampling frequencies are described in Table C.

[0250] Table C - Values and meanings of usacSamplingFrequencyIndex usacSamplingFrequencyIndex

[0251] usacSamplingFrequency Decoder sampling frequency encoded as an unsigned integrated value in case usacSamplingFrequencyIndex is equal to zero.

[0252] channelConfigurationIndex This index determines the channel configuration. If channelConfigurationIndex > 0, the index uniquely defines the number of channels, channel elements, and associated speaker mapping according to Table Y. The names of the speaker positions, the abbreviations used, and the general position of the available speakers can be deduced from Figures. 3a, 3b and Figures 4a and 4b.

[0253] bsOutputChannelPos This index describes the positions of the loudspeakers that are associated with a given channel according to Table XX. Figure Y indicates the position of the loudspeaker in the listener's 3D environment. In order to facilitate the understanding of the loudspeaker positions, Table XX also contains the loudspeaker positions according to IEC 100/1706/CDV which are listed here as information for the interested reader.

[0254] Table - Values of coreCoderFrameLength, sbrRatio, outputFrameLength and numSlots dependent on coreSbrFrameLengthIndex

[0255] usacConfigExtensionPresent Indicates the presence of extensions for the configuration.

[0256] numOutChannels If the value of channelConfigurationIndex indicates that none of the predefined channel configurations are used, then this element determines the number of audio channels to which a specific speaker position should be associated.

[0257] numElements This field contains the number of elements that will follow in the circuit in the element types in UsacDecoderConfig().

[0258] usacElementType[elemIdx] Defines the USAC channel element type of the element at position elemIdx in the data stream. There are four element types, one for each of the four basic data stream elements: UsacSingleChannelElement(), UsacChannelPairElement(), UsacLfeElement(),UsacExtElement(). These elements provide the necessary high-level structure while maintaining all the necessary flexibility. The meaning of usacElementType is defined in Table A. Table A - usacElementType Value

[0259] stereoConfigIndex This element determines the internal structure of a UsacChannelPairElement(). It indicates the use of a mono or stereo core, the use of MPS212, whether stereo SBR is applied, and whether residual coding is applied in MPS212 according to the ZZ Table. This element also defines the values of the auxiliary elements bsStereoSbr and bsResidualCoding.

[0260] Table ZZ - StereoConfigIndex values and their meaning and implicit designation of bsStereoSbr and bsResidualCoding

[0261] tw_mdct This indicator signals the use of MDCT stopped in time in this flow.

[0262] noiseFilling This flag signals the use of noise that fills the spectral holes in the FD core encoder.

[0263] harmonicSBR This indicator signals the use of harmonic repair for SBR.

[0264] bs_interTes This indicator signals the use of the intra-TES tool in SBR.

[0265] dflt_start_freq This is the default value for the bs_start_freq data stream element, which is applied in case the sbrUseDfltHeader flag indicates that the default values for SbrHeader() elements should be assumed.

[0266] dflt_stop_freq This is the default value for the bs_stop_freq data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0267] dflt_header_extra1 This is the default value for the bs_header_extra1 data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0268] dflt_header_extra2 This is the default value for the bs_header_extra2 data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0269] dflt_freq_scale This is the default value for the bs_freq_scale data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0270] dflt_alter_scale This is the default value for the bs_alter_scale data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0271] dflt_noise_bands This is the default value for the bs_noise_bands data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0272] dflt_limiter_bands This is the default value for the bs_limiter_bands data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0273] dflt_limiter_gains This is the default value for the bs_limiter_gains data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0274] dflt_interpol_freq This is the default value for the bs_interpol_freq data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0275] dflt_smoothing_mode This is the default value for the bs_smoothing_mode data stream element, which is applied in case the sbrUseDfltHeader flag indicates that default values for SbrHeader() elements should be assumed.

[0276] usacExtElementType This element allows to signal the types of extensions of the data stream. The meaning of usacExtElementType is defined in Table B.

[0277] Table B - Value of usacExtElementType

[0278] usacExtElementConfigLength flags the length of the extension configuration in bytes (octets).

[0279] usacExtElementDefaultLengthPresent This flag signals whether a usacExtElementDefaultLength is passed in UsacExtElementConfig().

[0280] usacExtElementDefaultLength signals the default length of the extension element in bytes. Only if the extension element in a given access unit deviates from this value, an additional length needs to be transmitted in the data stream. If this element is not explicitly transmitted (usacExtElementDefaultLengthPresent==0) then the value of usacExtElementDefaultLength should be set to zero.

[0281] usacExtElementPayloadFrag This flag indicates whether the payload of this extension element can be fragmented and sent as multiple segments in consecutive USAC frames.

[0282] numConfigExtensions If configuration extensions are present in UsacConfig() this value indicates the number of flagged configuration extensions.

[0283] confExtIdx Index to the configuration extensions.

[0284] usacConfigExtType This element allows to flag configuration extension types. The meaning of usacExtElementType is defined in Table D.

[0285] Table D - Value of usacConfigExtType

[0286] usacConfigExtLength flags the length of the configuration extension in bytes (octets).

[0287] bsPseudoLr This flag signals that an inverse lateral/average rotation should be applied to the center signal prior to Mps212 processing.

[0288] Table - bsPseudoLr

[0289] bsStereoSbr This flag signals the use of stereo SBR in combination with MPEG Sound decoding.

[0290] Table - bsStereoSbr

[0291] bsResidualCoding indicates whether residual coding is applied according to the Table below. The value of bsResidualCoding is set by stereoConfigIndex (see X).

[0292] Table X - bsResidualCoding

[0293] sbrRatioIndex indicates the ratio between the central sampling rate and the sampling rate after eSBR processing. At the same time it indicates the number of QMF analysis and synthesis tracks used in SBR according to the Table below.

[0294] Table - Definition of sbrRatioIndex

[0295] elemIdx Index to the elements present in UsacDecoderConfig() and UsacFrame(). UsacConfig()

[0296] UsacConfig() contains information about the output sampling frequency and channel configuration. This information must be identical to the information signaled outside this element, for example in an MPEG-4 AudioSpecificConfig().

Usac Output Sampling Frequency

[0297] If the sampling rate is not one of the rates listed in the right column of Table 1, sampling frequency dependent tables (code tables, scale factor range tables, etc.) must be derived in order to analyze the payload of the data stream. Since a given sampling frequency is associated with only one sampling frequency table, and since maximum flexibility is desired in the range of possible sampling frequencies, the following table should be understood to associate an implied sampling frequency with the desired sampling frequency dependent tables.

[0298] Table 1 - Sampling frequency mapping UsacChannelConfig()

[0299] The channel configuration table covers most of the common speaker positions. For added flexibility channels can be mapped to any of the selection of 32 speaker positions found in modern speaker configurations in various applications (see Figures 3a, 3b).

[0300] For each channel contained in the data stream UsacCanalConfig() specifies the associated speaker position to which this particular channel should be mapped. The speaker positions that are indexed by bsOutputChannelPos are listed in Table X. In the case of multiple channel elements the index i of bsOutputChannelPos[i] indicates the position at which the channel appears in the data stream. Figure Y provides an overview of the speaker positions relative to the listener.

[0301] More precisely, channels are numbered in the sequence in which they appear in the data stream starting with 0 (zero). In the common case of a UsacSingleCanalElement() or UsacLfeElement() the channel number is assigned to this channel and the channel count is raised by one. In the case of a UsacChannelPairElement() the first channel in this element (with index ch==0) is enumerated first, where the second channel in this same element (with index ch==1) is given the next higher number and the channel count is raised by two.

[0302] It follows that numOutChannels must be equal to or greater than the cumulative sum of all channels contained in the data stream. The cumulative sum of all channels is equivalent to the number of all UsacSingleChannelElement() plus the number of all UsacLfeElement() plus twice the number of all UsacChannelPairElement()’s.

[0303] All entries in the bsOutputChannelPos array must be mutually distinct to avoid double assignment of speaker positions in the data stream.

[0304] In the special case that channelConfigurationIndex is 0 and numOutChannels is less than the cumulative sum of all channels contained in the data stream, then the management of unassigned channels is outside the scope of this specification. Information about this, for example, can be conveyed by appropriate means in higher application layers or specifically by designed extension payloads (private). UsacDecoderConfig()

[0305] UsacDecoderConfig() contains all the information needed by the decoder to interpret the data stream. First, the value of sbrRatioIndex determines the ratio between the length of the core encoder frame (ccfl) and the length of the output frame. Following sbrRatioIndex is a loop over all channel elements in the current data stream. For each iteration the element type is signaled in usacElementType[], immediately followed by its corresponding configuration structure. The order in which the various elements are present in UsacDecoderConfig() must be identical to the order of the corresponding payload in UsacFrame().

[0306] Each moment of an element can be configured independently. When reading each channel element in UsacFrame(), for each element the corresponding configuration of this moment, i.e. with the same elemIdx, must be used. UsacSingleChannelElementConfig()

[0307] UsacSingleChannelElementConfig() contains all the information needed to configure the decoder to decode a single channel. SBR configuration data is only transmitted if SBR is actually employed. UsacChannelPairElementConfig()

[0308] The UsacChannelPairElementConfig() contains configuration data related to the core encoder as well as SBR configuration data dependent on the use of SBR. The exact type of stereo coding algorithm is indicated by stereoConfigIndex. In USAC a channel pair can be encoded in several ways. These are: 1. Stereo core encoder pair using traditional joined stereo coding techniques, extended by the possibility of complex prediction in the MDCT domain. 2. Mono core encoder channel in combination with MPS212-based MPEG Surround for full parametric stereo coding. Mono SBR processing is applied to the core signal. 3. Stereo core encoder pair in combination with MPS212-based MPEG Surround, where the first channel of the core encoder carries a downmix signal and the second channel carries a residual signal. The residual can be band-limited to realize partial residual coding. Mono SBR processing is applied only to the downmix signal before MPS212 processing. 4. Stereo center encoder pair in combination with MPS212-based MPEG Surround, where the first channel of the center encoder carries a downmix signal and the second channel carries a residual signal. The residual can be range-limited to realize partial residual encoding. Stereo SBR is applied to the reconstructed stereo signal after MPS212 processing.

[0309] Option 3 and 4 can further be combined with a pseudo LR channel rotation after the central decoder. UsacLfeElementConfig()

[0310] Since the use of MDCT with still time and noise padding is not allowed for LFE channels, there is no need to transmit the common central encoder indicator to these tools. They should be set to zero.

[0311] Furthermore, the use of SBR is neither allowed nor meaningful in an LFE context. Thus, SBR configuration data is not transmitted. UsacCoreConfig()

[0312] UsacCoreConfig() contains only flags to enable or disable the use of MDCT with still time and spectral noise filling on a global data stream level. If tw_mdct is set to zero, still time should not be applied. If noiseFilling is set to zero, spectral noise filling should not be applied. SbrConfig()

[0313] The data stream element SbrConfig() serves the purpose of signaling the exact eSBR configuration parameters. On the one hand SbrConfig() signals the general employment of eSBR tools. On the other hand, it contains a standard version of SbrHeader(), SbrDfltHeader(). The values of this standard header should be assumed if no different SbrHeader() is transmitted in the data stream. The background of this mechanism is that typically only one set of SbrHeader() values is applied in a data stream. The transmission of SbrDfltHeader() then allows to refer to this standard set of values very efficiently using only one bit in the data stream. The possibility to bypass the immediate SbrHeader values is still retained, allowing the transmission in the range of a new SbrHeader in the data stream itself. SbrDfltHeader()

[0314] The SbrDfltHeader() is what can be called the basic model of SbrHeader() and should contain the values for the predominantly used eSBR configuration. In the data stream this configuration can be referenced by setting the sbrUseDfltHeader flag. The structure of SbrDfltHeader() is identical to that of SbrHeader(). In order to distinguish between the values of SbrDfltHeader() and SbrHeader(), the bit fields in SbrDfltHeader() are prefixed with “dflt_” instead of “bs_”. If the use of SbrDfltHeader() is indicated, then the SbrHeader() bit fields should assume the values of the corresponding SbrDfltHeader(), i.e. bs_start_freq = dflt_start_freq; bs_stop_freq = dflt_stop_freq; etc. (continue for all elements in SbrHeader(), like: bs_xxx_yyy = dflt_xxx_yyy; Mps212Config()

[0315] Mps212Config() resembles MPEG Surround's SpatialSpecificConfig() and was largely derived from it. It is however reduced in scope to contain the information relevant for mono to stereo upmixing in the context of USAC. Consequently MPS212 configures only one OTT box. UsacExtElementConfig()

[0316] UsacExtElementConfig() is a general container for extension element configuration data for USAC. Each USAC extension has a unique type identifier, usacExtElementType, which is defined in Table X. For each UsacExtElementConfig() the length of the contained extension configuration is passed in the usacExtElementConfigLength variable and allows decoding to safely skip over extension elements whose usacExtElementType is unknown.

[0317] For USAC extensions that typically have a constant payload length, UsacExtElementConfig() allows the transmission of a usacExtElementDefaultLength. Setting a default payload length in the configuration allows for highly efficient signaling of the usacExtElementPayloadLength within UsacExtElement(), where bit consumption needs to be kept low.

[0318] In the case of USAC extensions where a larger amount of data is accumulated and transmitted not on a per-frame basis, but only on every second frame or even more rarely, this data may be transmitted in fragments or segments over several USAC frames. This can be useful to keep the bit pool more equalized. The use of this mechanism is signaled by the usacExtElementPayloadFrag flag. The fragmentation mechanism is further explained in the description of usacExtElement in 6.2.X. UsacConfigExtension()

[0319] UsacConfigExtension() is a general container for UsacConfig() extensions. It provides a convenient way to amend or extend the information exchanged at decoder initialization or configuration time. The presence of configuration extensions is indicated by UsacConfigExtensionPresent. If configuration extensions are present (UsacConfigExtensionPresent==1), the exact number of these extensions is given in the numConfigExtensions bit field. Each configuration extension has a unique type identifier, usacConfigExtType, which is defined in Table X. For each UsacConfigExtension the length of the contained configuration extension is transmitted in the usacConfigExtLength variable and allows the configuration data stream parser to safely skip configuration extensions where usacConfigExtType is unknown. Top-level payloads for the USAC audio object type Terms and Definitions

[0320] UsacFrame() This data block contains audio data for a time period of a USAC frame, related information, and other data. As signaled in UsacDecoderConfig(), UsacFrame() contains numElements elements. These elements can contain audio data for one or two channels, audio data for low-frequency enhancement, or extension payload.

[0321] UsacSingleChannelElement() Abbreviation SCE. Syntactical element of the data stream containing encoded data for a single audio channel. A single_channel_element() basically consists of UsacCoreCoderData(), containing data for both the FD and LPD core encoder. In case SBR is active, the UsacSingleChannelElement also contains SBR data.

[0322] UsacChannelPairElement() Abbreviation CPE. Syntactic element of the payload of the continuous data stream containing data for a channel pair. The channel pair can be obtained by transmitting two discrete channels or by transmitting one discrete channel and related to the Mps212 payload. This is signaled by means of stereoConfigIndex. The UsacChannelPairElement still contains SBR data in case SBR is active.

[0323] UsacLfeElement() Abbreviation LFE. Syntactic element that contains a low-frequency enhancement channel. LFEs are always encoded using the fd_channel_stream() element.

[0324] UsacExtElement() Syntactic element containing extension payload. The length of an extension element is either flagged as a default length in the configuration (USACExtElementConfig()) or flagged in UsacExtElement() itself. If present, the extension payload is of type usacExtElementType as flagged in the configuration.

[0325] usacIndependencyFlag Indicates whether the current UsacFrame() can be decoded completely without knowing the information of the previous structures according to the Table below.

[0326] NOTE: Please consult X.Y for recommendations on the use of usacIndependencyFlag.

[0327] usacExtElementUseDefaultLength Indicates whether the length of the extension element matches the usacExtElementDefaultLength, which was set in UsacExtElementConfig().

[0328] usacExtElementPayloadLength Shall contain the length of the extension element in bytes. This value shall only be explicitly transmitted in the data stream if the length of the extension element in this access unit deviates from the default value, usacExtElementDefaultLength.

[0329] usacExtElementStart Indicates whether the present usacExtElementSegmentData begins a data block.

[0330] usacExtElementStop Indicates whether the present usacExtElementSegmentData ends a data block.

[0331] usacExtElementSegmentData The concatenation of all consecutive usacExtElementSegmentData UsacExtElement() structures starting from the UsacExtElement() with usacExtElementStart==1 and including the UsacExtElement() with usacExtElementStop==1 forms a data block. In case a complete data block is contained in one UsacExtElement(), usacExtElementStart and usacExtElementStop must be set to 1. Data blocks are interpreted as a byte-aligned extension payload depending on the usacExtElementType according to the following Table:

[0332] Table - Interpretation of data blocks for decoding the USAC extension payload fill_byte Octet of bits that can be used to fill the data stream with bits that carry no information. The exact bit pattern used for fill_byte must be '10100101'. Help Elements

[0333] nrCoreCoderChannels In the context of a channel pair element this variable indicates the number of core encoder channels that form the basis for stereo encoding. Depending on the value of stereoConfigIndex this value should be either 1 or 2.

[0334] nrSbrChannels In the context of a channel pair element this variable indicates the number of channels to which SBR processing is applied. Depending on the value of stereoConfigIndex this value must be either 1 or 2. Subsidiary Payloads for USAC Terms and Definitions

[0335] UsacCoreCoderData() This data block contains the audio data from the core encoder. The payload element contains payloads for one or two channels of the core encoder, for either FD or LPD mode. The specific mode is signaled per channel at the beginning of the element.

[0336] StereoCoreToolInfo() All stereo related information is captured in this element. It handles the various bitfield dependencies in stereo coding modes. Help Elements

[0337] commonCoreMode On a CPE this flag indicates whether both encoded core encoder channels use the same mode.

[0338] Mps212Data() This data block contains payload for the Mps212 stereo module. The presence of this data is dependent on the stereoConfigIndex.

[0339] common_window Indicates whether channel 0 and channel 1 of a CPE use identical window parameters.

[0340] common_tw Indicates whether channel 0 and channel 1 of a CPE use identical parameters for MDCT with still time. UsacFrame() decoding

[0341] A UsacFrame() forms an access unit of the continuous USAC data stream. Each UsacFrame decodes at 768, 1024, 2048 or 4096 output samples according to the outputFrameLength determined from Table X.

[0342] The first bit in UsacFrame() is the usacIndependencyFlag, which determines whether a given frame can be decoded without any knowledge of the previous frame. If usacIndependencyFlag is set to 0, then dependencies on the previous frame may be present in the payload of the current frame.

[0343] The UsacFrame() is additionally made of one or more syntactic elements that must appear in the continuous data stream in the same order as their corresponding configuration elements in the UsacDecoderConfig(). The position of each element in the series of all elements is indexed by elemIdx. For each element the corresponding configuration, as transmitted in the UsacDecoderConfig(), of this moment, i.e. with the same elemIdx, must be used.

[0344] These syntactic elements are of one of the four types, which are listed in Table X. The type of each of these elements is determined by usacElementType. There may be several elements of the same type. Elements that occur at the same elemIdx position in different structures must belong to the same flow.

[0345] Table - Examples of possible simple payloads of continuous data streams

[0346] If these data stream payloads are to be transmitted over a constant rate channel then they may include an extension payload element with a usacExtElementType of ID_EXT_ELE_FILL to adjust the instantaneous bit rate. In this case, an example of an encoded stereo signal is:

[0347] Table - Examples of simple stereo data stream with extension payload to record padding bits. UsacSingleChannelElement() decoding

[0348] The simple UsacSingleChannelElement() structure is made up of a single UsacCoreCoderData() element with nrCoreCoderChannels set to 1. Depending on the sbrRatioIndex of this element a UsacSbrData() element follows with nrSbrChannels set to 1 as well. UsacExtElement() decoding

[0349] UsacExtElement() structures in a data stream may be decoded or skipped by a USAC decoder. Each extension is identified by a usacExtElementType, carried in the UsacExtElement() associated with UsacExtElementConfig(). For each usacExtElementType a specific decoder may be present.

[0350] If a decoder for the extension is available to the USAC decoder then the extension payload is forwarded to the extension decoder right after the UsacExtElement() is parsed by the USAC decoder.

[0351] If no decoder for the extension is available to the USAC decoder, a minimum of structure is provided within the data stream so that the extension can be ignored by the USAC decoder.

[0352] The length of an extension element is specified by a default length in octets, which can be signaled within the corresponding UsacExtElementConfig() and which can be overridden in UsacExtElement(), or by a length information explicitly provided in UsacExtElement(), which is one or three octets, using the escapedValue() syntactic element.

[0353] Extension payloads that span one or more UsacFrame() may be fragmented and their payload may be distributed among the multiple UsacFrame(). In this case the usacExtElementPayloadFrag flag is set to 1 and a decoder shall collect all fragments from the UsacFrame() with usacExtElementStart set to 1 and including the UsacFrame() with usacExtElementStop set to 1. When the usacExtElementStop is set to 1 then the extension is considered complete and is passed to the extension decoder.

[0354] Note that integrity protection for a fragmented extension payload is not provided by this specification and other means must be used to ensure the integrity of extension payloads.

[0355] Note that all extension payload data is assumed to be byte-aligned.

[0356] Each UsacExtElement() must obey the requirements resulting from the use of usacIndependencyFlag. To put it more explicitly, if usacIndependencyFlag is set (==1) the UsacExtElement() must be decoded without knowledge of the prior structure (and the extension payload that may be contained within it).

Decoding Process

[0357] The stereoConfigIndex, which is passed in UsacChannelPairElementConfig(), determines the exact type of stereo encoding that is applied to the given CPE. Depending on this type of stereo encoding either one or two core encoder channels are actually transmitted in the data stream and the nrCoreCoderChannels variable needs to be set accordingly. The UsacCoreCoderData() syntax element then provides the data for one or two core encoder channels.

[0358] Similarly there may be data available for one or two channels depending on the stereo encoding type and the use of eSBR (i.e. if sbrRatioIndex>0). The value of nrSbrChannels needs to be set correctly and the UsacSbrData() syntax element provides eSBR data for one or two channels.

[0359] Finally the Mps212Data() is transmitted, depending on the value of stereoConfigIndex.

Low frequency enhancement (LFE) channel element, UsacLfeElement() General

[0360] To maintain a regular structure in the decoder, UsacLfeElement() is defined as a standard element fd_channel_stream(0,0,0,0,x), i.e. it is equal to a UsacCoreCoderData() using the frequency domain encoder. Thus, decoding can be done using the standard procedure for decoding a UsacCoreCoderData()-element.

[0361] To accommodate a more efficient bit rate and hardware implementation of the LFE decoder, however, several restrictions apply to the options used for encoding this element:

[0362] The window_sequence field is always set to 0 (ONLY_LONG_sequence);

[0363] Only the lowest 24 spectral coefficients of any LFE may be nonzero;

[0364] No Temporal Noise Shape is used, i.e. tns_data_present to set to 0;

[0365] Warped time is not active;

[0366] No noise padding is applied. UsacCoreCoderData()

[0367] UsacCoreCoderData() contains all the information to decode one or two channels from the core encoder.

[0368] The decoding order is:

[0369] Get the core_mode[] for each channel;

[0370] In case of two central coded channels (nrChannels==2), parse the StereoCoreToolInfo() and determine all stereo related parameters;

[0371] Depending on the core_modes flag transmit either a lpd_channel_stream() or a fd_channel_stream() for each channel.

[0372] As can be seen from the above list, decoding a core encoder channel (nrChannels==1) results in getting the core_mode bit followed by either lpd_channel_stream or fd_channel_stream, depending on the core_mode.

[0373] In the case of two core encoder channels, some signaling redundancies between channels can be explained in particular if core_mode of both channels is 0. See 6.2.X (StereoCoreToolInfo() decoding) for more details. StereoCoreToolInfo()

[0374] StereoCoreToolInfo() allows to efficiently encode parameters whose values can be shared by the core encoder channels of a CPE in case both channels are encoded in FD mode (core_mode[0,1]==0). In particular, the following data elements are shared when the appropriate flag in the data stream is set to 1.

[0375] Table - Data stream element shared by channels of a central encoder channel pair

[0376] If the appropriate flag is not set then data elements are transmitted individually for each core encoder channel in both StereoCoreToolInfo()(max_sfb, max_sfb1) and fd_channel_stream() that follows the StereoCoreToolInfo() in the UsacCoreCoderData() element.

[0377] In case of common_window==1 the StereoCoreToolInfo() also contains the information about the M/S stereo coding and complex prediction data in the MDCT domain (see 7.7.2).

[0378] UsacSbrData() This data block contains the payload for extending the SBR bandwidth to one or two channels. The presence of this data is dependent on sbrRatioIndex.

[0379] SbrInfo() This element contains SBR control parameters that do not require decoder reset when changed.

[0380] SbrHeader() This element contains SBR header data with SBR configuration parameters, which typically do not change over the course of a continuous data stream. SBR payload for USAC

[0381] In USAC the SBR payload is transmitted in UsacSbrData(), which is an integral part of each single channel element or channel pair element. UsacSbrData() immediately follows UsacCoreCoderData(). There is no SBR payload for LFE channels.

[0382] numSlots The number of time slots in an Mps212Data structure.

[0383] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a function of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or function of a corresponding apparatus.

[0384] Depending on certain implementation requirements, applications of the invention may be implemented in hardware or in software. The implementation may be carried out using a digital storage medium, for example, a floppy disk, a DVD, a CD, a ROM memory, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored therein, which cooperate (or may cooperate) with a programmable computer system so that the respective method is carried out.

[0385] Some applications according to the invention comprise a non-transitory data carrier having electronically readable control signals, which can cooperate with a programmable computer system, so that one of the methods described herein is realized.

[0386] The encoded audio signal may be transmitted over a wired or wireless transmission medium or may be stored on a machine-readable medium or non-transitory storage medium.

[0387] Generally, applications of the present invention may be implemented as a computer program product having a product code, the product code being operative to perform one of the methods when the computer program product is executed on a computer. The product code may, for example, be stored on a machine-readable medium.

[0388] Other applications comprise the computer program for performing one of the methods described herein, stored in a machine-readable medium.

[0389] In other words, an application of the inventive method is therefore a computer program having product code for carrying out one of the methods described herein, when the computer program executes on a computer.

[0390] Another application of the inventive method is therefore a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for carrying out one of the methods described herein.

[0391] Another application of the inventive method is therefore a data stream or a sequence of signals representing the computer program for carrying out one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example via the Internet.

[0392] An additional application comprises a processing means, for example, a computer, or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0393] An additional application comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0394] In some applications, a programmable logic device (e.g., an array of programmable logic gates) may be used to perform some or all of the functionality of the methods described herein. In some applications, an array of programmable logic gates may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0395] The applications described above are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is intended, therefore, that they be limited only by the scope of the pending patent claims and not by the specific details set forth in the form of description and explanation of the applications contained herein.

Claims (9)

1. Decoder for decoding a continuous data stream (12) comprising an electronic circuit and a computer; wherein the continuous data stream comprises a configuration block (28) and a sequence of frames (20) respectively representing consecutive time periods of an audio content (10), characterized in that the configuration block (UsacConfig) comprises a field (numElements) indicating a number N of frame elements per frame, and a type indication syntax portion (52) indicating, for each element position of a sequence of N element positions, an element type of the plurality of element types, and wherein each of the sequence of frames comprises a sequence of N frame elements, the decoder being configured to decode each frame (20) by decoding each frame element (22) according to the element type indicated by the type indication syntax portion such that the i-th frame element of the sequence of N frame elements is decoded according to the element type indicated by the type indication syntax portion for the i-th element position. wherein the plurality of element types comprises a span element type, wherein the decoder is configured to read, from each structure element (22b) of the span element type of any structure (20), a length information (58) in a length of the respective structure element, skip at least a portion of at least some of the structure elements (22) of the span element type of the structures (20) using the length information (58) in the length of the respective structure element as the length of the skip interval. wherein the decoder is configured to read, for each element position in which the type-indicating syntax portion indicates the extension element type, a configuration element (74) comprising configuration information for the extension element type of the configuration block (28), with, when reading the configuration information for the extension element type, reading the standard payload length information (60) at a standard extension payload length of the data stream, the decoder is also configured to, when reading the length information (58) of the structure elements (22) of the extension element type, read a standard extension payload length indicator (64) of a conditional syntax portion (62) of the data stream (12), check whether the standard payload length indicator (64) is set, and, if the standard payload length indicator (64) is not set, read a value of the extension payload length (66) of the conditional syntax portion (62) of the data stream (12) to obtain an extension payload length of the respective structure element, and, if the standard payload length indicator (64) is set, setting the extension payload length of the respective structure element to be equal to the standard extension payload length, the decoder is also configured to skip a payload section (68) of at least some of the structure elements (22) of the extension element type of the structures (20) using the extension payload length of the respective structure element as the skip interval length. wherein the decoder is configured to, upon reading the standard payload length information (60), read a standard payload length present indicator from the data stream (12), check whether the standard payload length present indicator is set, if the standard payload length present indicator is not set, set the standard extension payload length to zero, if the standard payload length present indicator is set, explicitly read the standard extension payload length from the bit stream, and wherein the decoder is configured to reconstruct the audio content. 2. The decoder of claim 1, wherein the decoder is configured to read a sequence of N syntax elements (54) of the type indicating syntax portion (52), with each element indicating the element type for the respective element position at which the respective syntax element is positioned in the sequence of N syntax elements. 3. The decoder of claim 1, wherein the decoder is configured to read a sequence of N configuration elements (56) from the configuration block (28), with each configuration element comprising configuration information so that the i-th configuration element comprises configuration information about the element type for the i-th element position, wherein the decoder is configured to, in decoding the i-th structure element of the sequence of N structure elements, use the configuration information about the element type for the i-th element position. 4. The decoder of claim 3, wherein the type-indicating syntax portion (52) comprises a sequence of N syntax elements (54), with each syntax element indicating the element type for the respective element position at which the respective syntax element is positioned in the sequence of N syntax elements, and the decoder is configured to read the configuration elements (56) and the syntax elements (54) from the continuous data stream (12) alternately. 5. Decoder according to claim 1, characterized in that the decoder is configured to, upon reading the length information (58) of any structure element of the extension element type of the structures, read a present indicator of the extension payload (70) of the data stream (12), check whether the present indicator of the extension payload (70) is set, and, if the present indicator of the extension payload (70) is not set, stop reading the respective structure element (22b) of the extension element type and continue with the reading of another structure element (22) of a current structure (20) or a structure element of a subsequent structure (20), and if the present indicator of the extension payload (70) is set, read a portion of the syntax indicating a length of the extension payload of the respective structure of the extension element type of the data stream, and skip, for at least some of the structure elements (22) of the extension element type of the structures (20), the present indicator of the extension payload (70) of the length information that is defined, a payload section (68) thereof using the length of the extension payload of the respective structure element (22b) of the extension element type read from the continuous data stream as the length of the hop interval. 6. Decoder according to claim 1, characterized in that the decoder is configured so that the decoder, in decoding the structure elements (22) in the structures (20) at the element positions in which the type indication syntax portion indicates a channel pair element type, reconstructs the two audio signals. 7. Decoder for decoding a continuous data stream (12) comprising an electronic circuit and a computer; characterized in that the continuous data stream comprises a configuration block (28) and a sequence of frames (20) respectively representing consecutive time periods of an audio content (10), wherein the configuration block (UsacConfig) comprises a field (numElements) indicating a number N of structure elements per frames, and a type indication syntax portion (52) indicating, for each element position of a sequence of N element positions, an element type of the plurality of element types, and wherein each of the sequence of frames comprises a sequence of N structure elements, the decoder is configured to decode each frame (20) by decoding each structure element (22) according to the element type indicated by the type indication syntax portion such that the i-th structure element of the sequence of N structure elements is decoded according to the element type indicated by the type indication syntax portion for the i-th element position wherein the plurality of element types comprises a type of extension element, wherein the decoder is configured to read, from each structure element (22b) of the extension element type of any structure (20), a length information (58) in a length of the respective structure element, skip at least a portion of at least some of the structure elements (22) of the extension element type of the structures (20) using the length information (58) in the length of the respective structure element as the length of the skip interval; wherein the decoder is further configured to, upon reading a configuration block (28), for each element position in which the type indication syntax portion (52) indicates the extension element type, read a configuration element (56) comprising configuration information for the extension element type of the continuous data stream (12), wherein the configuration information comprises an extension element type field (72) indicating a payload data type of a plurality of payload data types; wherein the plurality of payload data types comprises a multi-channel information type and a multi-object coding side information type, the decoder is configured to, upon reading the configuration block (28), for each element position at which the type indicating portion (52) indicates the extension element type, if the extension element type field (72) indicates the multi-channel information type, read the configuration data of the multi-channel side information (74) as part of the configuration information of the continuous data stream (12), and if the extension element type field (72) indicates the multi-object coding side information type, read the configuration data of the multi-object coding side information (74) as part of the configuration information of the stream, and the decoder is configured to, upon decoding each frame, decode the frame elements of the extension element type positioned at any element position at which the type indicating syntax portion (52) indicates the extension element type, and at which the extension element type field (72) indicates the multi-channel information type, read the configuration data of the multi-object coding side information (74) as part of the configuration information of the stream, and the decoder is configured to, upon decoding each frame, decode the frame elements of the extension element type positioned at any element position at which the type indicating syntax portion (52) indicates the extension element type, and at which the extension element type field (72) indicates the multi-channel information type, read the configuration data of the multi-object coding side information (74) as part of the configuration information of the stream, extension element type of the configuration element (56) indicates the multi-channel information type, configuring a multi-channel decoder (44e) using the configuration data of the multi-channel side information (74) and feeding the then configured multi-channel decoder (44e) with the payload data (68) of the respective structure elements (22b) of the extension element type as the multi-channel side information, decoding the structure elements of the extension element type positioned at any position of the element at which the type indicating part indicates the extension element type, and at which the extension element type of the configuration element (56) indicates the multi-object coding side information type, configuring a multi-object decoder (44d) using the configuration data of the multi-object coding side information (74) and feeding the then configured multi-object decoder (44d) with payload data (68) of the respective structure elements (22) of the extension element type as the multi-channel side information on the multi-object coding side; and where the decoder is configured to reconstruct the audio content. 8. A method for decoding the continuous data stream (12), wherein the continuous data stream comprises a configuration block (28) and a sequence of frames (20) respectively representing consecutive time periods of an audio content, characterized in that the configuration block (28) comprises a field (50) indicating a number of elements N, and a type indication syntax portion (52) indicating, for each element position of a sequence of N element positions, an element type of a plurality of element types, and wherein each of the sequence of frames comprises a sequence of N frame elements, wherein the method comprises decoding each frame (20) by decoding each frame element (22) according to the element type indicated, by the type indication syntax portion, for a respective element position at which the respective frame element is positioned within the sequence of N frame elements (22) of the respective frame (20) in the continuous data stream (12); wherein the plurality of element types comprises a span element type, wherein the method further comprises reading, from each structure element (22b) of the span element type of any structure (20), a length information (58) on a length of the respective structure element, skipping at least a portion of at least some of the structure elements (22) of the span element type of the structures (20) using the length information (58) on the length of the respective structure element as the length of the skip interval. wherein the decoder is further configured to, upon reading a configuration block (28), for each element position in which the type indicating synax portion (52) indicates the span element type, read a configuration element (56) comprising configuration information for the span element type of the data stream (12), wherein the configuration information comprises a span element type field (72) indicating a payload data type of a plurality of payload data types; wherein the plurality of payload data types comprises a multi-channel information type and a multi-object coding side information type, in reading the configuration block (28) comprises, for each element position in which the type indication syntax portion (52) indicates the extension element type, if the extension element type field (72) indicates the multi-channel information type, reading the configuration data of the multi-channel side information (74) as part of the configuration information of the stream (12), and if the extension element type field (72) indicates the multi-object coding side information type, reading the configuration data of the multi-object coding side information (74) as part of the configuration information of the stream, and the method comprises, in decoding each frame, decoding the frame elements of the extension element type positioned at any element position in which the type indication syntax portion indicates the extension element type, and in which the extension element type of the configuration element (56) indicates the multi-channel information type, configuring a multi-channel decoder (44e) using the configuration data of the multi-channel side information (74) and feeding the then configured multi-channel decoder (44e) with the payload data (68) of the respective structure elements (22b) of the extension element type as the multi-channel side information, decoding the structure elements of the extension element type positioned at any element position at which the type indicating part indicates the extension element type, and at which the extension element type of the configuration element (56) indicates the multi-object coding side information type, configuring a multi-object decoder (44d) using the configuration data of the multi-object coding side information (74) and feeding the then configured multi-object decoder (44d) with payload data (68) of the respective structure elements (22) of the extension element type as the multi-object coding side information; and decode the structures when reconstructing the audio content. 9. Non-transitory readable medium characterized by having, stored therein, instructions for carrying out, when executed on a computer, the method as defined in claim 8.