RU2838373C1 - Quantisation and entropy coding of parameters for low-latency audio codec - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to the field of entropy coding of parameters for audio codecs with low latency and to mechanisms for achieving target bitrates of parameters by iterative refinement of bitrate of parameters using a range of technologies of quantisation and entropy coding. Disclosed is a method for frame-by-frame encoding of metadata for an input signal, wherein the metadata contain a plurality of at least partially interrelated parameters calculated from the input signal. Method involves iterative execution of the following steps for each frame using a looped process: determining a processing strategy from a plurality of processing strategies for calculating and quantizing parameters, calculating and quantizing parameters based on the determined processing strategy to obtain quantized parameters, and encoding the quantized parameters. In particular, each of the plurality of processing strategies comprises a corresponding first indicator indicating the order related to the calculation and quantization of the individual parameters. Processing strategy is determined based on at least one bitrate threshold value.

EFFECT: minimizing additional information (parameters) while maintaining a high frame refresh rate for the audio data base.

17 cl, 4 dwg, 3 tbl

Description

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application Nos. 63/037,784 and 63/194,010, filed June 11, 2020 and May 27, 2021, respectively, each of which is incorporated by reference in its entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

The present invention generally relates to the field of entropy parameter coding (additional information) for low-latency audio codecs (encoders/decoders) and to mechanisms for achieving target parameter bitrates by iteratively refining the parameter bitrate using a range of quantization and entropy coding technologies.

LEVEL OF TECHNOLOGY

When the frame period (frame size) of the audio codec (encoder/decoder) approaches 20 milliseconds (ms) or less, the audio data base is updated in short frame sizes. If we follow the approach of updating both the audio data base and the parameters every frame, the additional information for each frame will also be embedded and transmitted at the same rate.

However, it is generally known in the art that the additional information does not need to be updated so frequently. For example, spatial parameters can typically be calculated and updated, for example, every 40 ms. For codecs with frame periods of 40 ms or more, this usually means that the parameter update rate matches the frame rate, and thus the parameters can be encoded in each frame independently. However, in codecs with short frame periods, for example less than 40 ms, this means that the parameters will actually be oversampled if they are all included in each frame.

Thus, in a broad sense, the aim of the present invention is to provide mechanisms for minimizing additional information (which is sometimes also referred to as parameters) as much as possible, but maintaining a high frame rate for the underlying audio data.

ESSENCE OF THE INVENTION

Taking into account the above, the present invention generally provides a method for frame-by-frame encoding of metadata for an input signal, as well as a corresponding program, a machine-readable storage medium and a device having the features specified in the corresponding independent claims of the invention.

According to one aspect of the present invention, a method for frame-by-frame encoding of metadata for an input signal is provided. In particular, the metadata can be calculated or computed (e.g., extracted) from the input (audio or video) signal using a suitable codec (encoder/decoder). Generally speaking, the metadata can be used to reconstruct the input signal on the decoder side. The metadata can comprise a plurality of at least partially interrelated parameters calculated from the input signal. In other words, at least some parameters of the input signal can be calculated (e.g., generated or reconstructed) depending on at least some other parameters, so that, depending on various circumstances, not all parameters need always be transmitted explicitly.

In particular, the method may comprise/include, for each frame, iteratively performing, using a looped process, the following steps: determining a processing strategy from a plurality of processing strategies for calculating and quantizing parameters; calculating and quantizing parameters based on the determined processing strategy to obtain quantized parameters; and encoding the quantized parameters. Since the looped process is generally directed (among other things) to processing related to quantization, in some cases the looped process may also be called a quantization loop (or simply a loop, for short). Similarly, since the processing strategy is also generally directed (among other things) to processing that is related to quantization, in some cases the processing strategy may also be called a quantization strategy (or, in some other cases, interchangeably referred to as a quantization scheme). In addition, it should be noted that the encoding process may use any suitable encoding procedure, including, but not limited to, entropy encoding (e.g., Huffman encoding or arithmetic encoding) or entropy-free encoding (e.g., base 2 encoding). Depending on various implementations and/or requirements, any other suitable encoding mechanism may be used.

As will be clear and obvious to a person skilled in the art, the plurality of processing strategies for calculating and quantizing the parameters may be provided in any suitable manner, such as predetermined or preconfigured. Accordingly, the processing strategy may also be determined from the plurality of processing strategies in any suitable manner. For example, depending on the (current) bit rate requirement, a suitable processing strategy may be selected from the plurality of processing strategies, such that the resulting bit rate after performing the calculation, quantization and encoding (e.g. with or without entropy encoding) based on the processing strategy thus selected satisfies the (current) bit rate requirement. It should be noted that since the bit rate requirement may change from time to time (e.g. from frame to frame), the processing strategy thus determined may also be different for each frame or some frames.

In particular, each of the plurality of processing strategies may contain a corresponding first pointer indicating an order (or sequence) related to the calculation and quantization of individual parameters. In other words, the first pointer may contain sequence information indicating when and in what order to perform the calculation and quantization of individual parameters. As an example (but not limitation), the first pointer may contain information indicating that all parameters are first calculated before any of them are quantized.

More specifically, the processing strategy is determined based on at least one bit rate threshold. As will be understood and apparent to a person skilled in the art, the bit rate threshold(s) may, for example, be predetermined or preconfigured, depending on different implementations and/or requirements.

Generally speaking, in the above-described configuration, the proposed method according to the present invention can be considered as implementing the idea of an iterative and stepwise approach to selecting an optimal parameter quantization scheme/strategy, which generally searches for the "best" (or optimal) quantization scheme from several alternatives. However, it should be noted that in the present case, the term "best" does not necessarily have to refer to the quantization scheme with the lowest (final) parameter bitrate (i.e. after quantization and possible encoding), but may refer to a scheme that is able to alleviate the loss of state for the decoder. As will be understood by one skilled in the art, in general, the "state" of the decoder refers to the history of information that the decoder stores from previous frames in order to be able to correctly decode the current frame. For example (but not as a limitation), in some cases, the encoder side may apply the so-called time-differential encoding. However, the use of time-differential coding in general may have a disadvantage, which is primarily that state is usually introduced from frame to frame, which may cause problems when the audio stream may experience packet loss during transmission. In this case, both the audio data and the parameters associated with the audio data may be lost during transmission, so that any parameters that have been updated using time-differential coding may experience potential artifacts in several subsequent frames. In this sense, the above-mentioned alleviation of state loss refers to an attempt to avoid time-differential coding where possible, so that the decoder does not have to rely on metadata received in previous frames in order to decode the metadata of the current frame. However, when time-differential coding is required, it must be implemented in such a way that the system quickly recovers from packet loss. In particular, by carefully choosing an appropriate quantization scheme, as described in the present invention, the above-presented undesirable behavior associated with packet loss can be limited (reduced) as much as possible. In other words, the present invention generally proposes a coding reduction (encoder side), which includes an iterative process of choosing for quantization and coding (with or without entropy) that attempts to minimize the degree of possible introduction of artifacts caused by packet loss, for example due to the use of time-differential coding.

In some examples, the processing strategy may be defined such that the (resulting) bitrate of the encoded quantized parameters is equal to or less than the (metadata/parameters) bitrate threshold. As such, the resulting bitrate after quantization and encoding using a certain (e.g., selected) processing strategy is within the (at least one) bitrate threshold, thereby satisfying a bitrate requirement, e.g., pre-approved or pre-defined by standardization specifications.

In some examples, each of the plurality of processing strategies may further comprise a respective second pointer indicating information for performing parameter quantization.

In some examples, the information for performing quantization of the parameters comprises corresponding quantization ranges and/or quantization levels for a plurality of parameters. For example, the information may relate to a maximum value, a minimum value, a number of quantization levels, or any other suitable value required for each of the corresponding parameters (e.g., one corresponding parameter for each type of parameter). Generally speaking, as will be understood and apparent to one skilled in the art, these quantization-related values/parameters provide or determine a coarser or finer quantization overall and, accordingly, an accompanying better or worse spatial reproduction. As will be understood and apparent to one skilled in the art, generally speaking, some (quantization) parameters are generally considered to be more sensitive to quantization than others, and, in general, there may be no absolute fine/coarse quantization technique for all parameters.

In the above configuration, each of the plurality of processing strategies can be regarded as comprising a first pointer (its first part/chunk) related to the order/sequence associated with the computation and quantization; and a second pointer (its second part/chunk) related to the actual quantization process. By carefully designing the processing strategy (e.g., using different combinations of the first pointer and the second pointer), it is possible to efficiently and flexibly meet different bitrate configurations/requirements, such as for different use cases or scenarios. In particular, in some cases, there may be one processing strategy (e.g., the coarsest quantization strategy among the plurality of quantization strategies) that can be regarded as guaranteeing a bitrate lower than (or equal to) a target bitrate threshold.

In some examples, the encoding of the parameters may include time-differential and/or frequency-differential encoding. Generally speaking, one metadata parameter may be quantized from a continuous numerical value into an index representing a discrete value. In non-differential encoding, the information encoded for this metadata parameter corresponds directly to this index. It should be noted that the term "non-differential encoding" as used in the present invention may refer to non-time-differential encoding, non-frequency-differential encoding, or non-differential encoding of all kinds, if necessary, as will be understood and obvious to one skilled in the art. In time-differential encoding, the information encoded is the difference between the index of the metadata parameter from the current frame and the index of the same metadata parameter from the previous frame. As will be understood and obvious to one skilled in the art, the above-described general idea of time-differential encoding may be further extended, for example, to a plurality of frequency bands. Accordingly, the metadata parameter can be extended in a similar manner, for example, to a plurality of parameters, respectively related to a plurality of frequency bands (or each of them), as needed. Frequency differential coding follows a similar principle, but the encoded difference is the difference between the metadata of one frequency band of the current frame and the metadata of another frequency band of the current frame (as opposed to the current frame minus the previous frame in time differential coding). As a simple example (but not limitation), suppose that a0, a1, a2 and a3 will denote the indices of the parameters in the 4 frequency bands of a particular frame, then in one example of implementation the frequency differential indices can be a0, a0-a1, a1-a2, a2-a3. As will be apparent to those skilled in the art, the general idea of (time- and/or frequency-) differential coding is that metadata may typically vary slowly from frame to frame or from frequency band to frequency band, so that even if the original metadata value was large, the difference between it and the metadata of the previous frame or the difference between it and the metadata of another frequency band is likely to be small. This is advantageous because, in general, parameters with statistical distributions that tend to zero can be encoded using fewer bits.

In some examples, the processing strategy determined for the current frame may differ from the processing strategy determined for the previous frame, and accordingly, the encoding of the parameters may include time-differential encoding in different processing strategies. In other words, in certain cases where different processing strategies are determined (for example, for different frames of the input signal), the method according to the present invention can still encode the parameters, for example, using time-differential encoding in these different processing strategies.

As indicated above, each of the plurality of processing strategies may contain a corresponding first pointer indicating the order (or sequence) related to the calculation and quantization of individual parameters.

In some examples, the first pointer may contain information indicating that all parameters are calculated before quantization.

In some examples, the first indicator may contain information indicating that the parameters are calculated separately and then quantized sequentially one after another. In particular, at least one parameter of the plurality of parameters may be calculated based on another quantized parameter of the plurality of parameters. As an example, but not limitation, suppose that a total of three parameters are to be calculated and quantized, then the first parameter may be calculated first (from the input signal) and then quantized; while the second parameter may be calculated based on the (quantized) first parameter and then the second parameter itself may be quantized; and finally, the third parameter is calculated based on the (quantized) first parameter and/or the (quantized) second parameter and then quantized. In one example, the third parameter is calculated based on the quantized first and second parameters.

In some examples, the first indicator may contain information indicating that all parameters are calculated before any parameter is quantized; and in particular, at least one of the parameters is recalculated based on another quantized parameter, and the recalculated parameter is quantized. Also taking the above-described three parameters as an example, all parameters are first calculated and then the first and second parameters are quantized; after that, the third parameter is recalculated, for example based on the quantized second parameters, and then the third parameter is quantized based on the recalculated value.

In some examples, the method may further include, before encoding the quantized parameters, mapping the indices of the quantized parameters from the previous frame to the indices of the current frame. In other words, if a different processing strategy (quantization scheme, for example, concerning different quantization levels and/or sequences) is determined (e.g., found/selected), the indices (quantizations) from the previous frame that were quantized using the other quantization scheme are mapped to the indices of the current frame. It should be noted that this allows for time-differential encoding to be performed between frames without the need to send a non-differential frame each time the quantization scheme is changed, thereby further improving the overall efficiency and flexibility of coding.

In some possible implementations, index mapping can be performed based on formulas: , Where denotes the indices of the current frame after display, denotes the index of the previous frame, denotes the quantization level of the current frame and indicates the quantization level of the previous frame.

As a simple illustrative example, assume the quantization range is 0 to 2, and assume the previous quantization levels are 11. In the case of uniform quantization, this would generally mean that each quantization step would be 0.2. Further assume the current quantization levels are 21, which means that each quantization step is 0.1 under uniform quantization. Based on these assumptions, if the quantized value in the previous frame was 0.4, then with 11 uniform quantization levels, the next previous index is obtained . The mapping provides the quantized indices of the previous frame's metadata as if they were quantized using the quantization levels of the current frame. So in this example, if the quantization levels in the current frame are 21, then the quantized value 0.4 would be mapped to . After the mapped indices are calculated, the difference between the indices of the current frame and the previous frame is calculated and this difference is encoded. Similar or similar approaches may also be applied to frequency differential coding, if necessary, as will be understood and obvious to a person skilled in the art.

It should be noted that the above formulas and the corresponding example are provided for illustrative purposes only and that any other suitable mechanism (e.g., a lookup table, etc.) may be used to perform the index mapping, as will be understood and apparent to one skilled in the art.

In some examples, the at least one bit rate threshold may comprise a target bit rate threshold. Accordingly, the loop process may comprise the following steps: quantizing and encoding the parameters in a non-differential and/or frequency-differential manner using an entropy coder in accordance with a (specified) processing strategy; estimating (e.g., calculating) a first parameter bit rate for the encoded parameters; and exiting the loop process if the first parameter bit rate is less than or equal to the target bit rate threshold. In particular, in some possible implementations, the first parameter bit rate may be estimated (calculated) from a minimum among non-differential and frequency-differential coding schemes encoded using (trained) entropy coders. As will be understood and apparent to one skilled in the art, entropy coders may be trained in any suitable manner, for example to adapt to individual coding schemes. For example, in some possible implementations, training entropy coders may involve developing probability models based on metadata computed from a large group of input signals. The specific signals chosen to develop these models are expected to represent the types of signals that are expected to flow through the system in everyday use. Thus, metadata from other similar signals should be encoded as efficiently as possible. In short, the goal of this training is to adapt entropy coders to be as efficient as possible with the expected probability distribution of the parameters.

In some examples, the loop process may further include the following steps: if the first parameter bit rate is greater than a target bit rate threshold, quantizing and encoding the parameters in a non-differential manner without entropy according to a processing strategy; estimating a second parameter bit rate for the encoded parameters; and exiting the loop process if the second parameter bit rate is less than or equal to the target bit rate threshold.

In some examples, the loop process may further include the following steps: if the second parameter bit rate is greater than a target bit rate threshold, quantizing and encoding the parameters in a time-differential manner using a (trained) entropy encoder according to the processing strategy; estimating a third parameter bit rate for the encoded parameters; and exiting the loop process if the third parameter bit rate is less than or equal to the target bit rate threshold.

In some examples, the time-differential quantization and encoding may be performed on a subset of the parameters in a frequency-interleaved manner relative to the previous frame. In particular, as will be understood and apparent to one skilled in the art, the frequency-interleaved manner may typically refer to cases where different frequency bands (e.g., corresponding to different subsets of the parameters) are processed (e.g., quantized and encoded) for different frames. In other words, the time-differential quantization and encoding of (at least a subset) of the parameters for the current frame may be performed in a different frequency band (corresponding to the parameters currently being processed) that is different from the frequency band of the previous frame.

In some examples, time-differential quantization and encoding may be accomplished by cycling through a series of frequency-interleaved time-differential encoding schemes such that for each cycle, a different subset of the parameters (corresponding to a different set of frequency bands) are quantized and encoded time-differentially, while the remaining parameters are quantized and encoded non-differentially.

In some examples, the determined processing strategy may be considered as a first processing strategy, and accordingly the loop process may further comprise the following steps: if the third parameter bit rate is greater than the target bit rate threshold, determining a second processing strategy from the plurality of processing strategies such that the (resulting) bit rate after applying the second processing strategy is expected to be less than the bit rate of using the first processing strategy; and repeating the above-described steps of the loop process. As will be understood and obvious to a person skilled in the art, in such cases the determined (e.g., selected) second processing strategy in this way may simply be considered as a processing strategy that is coarser than the previously determined (e.g., selected) first processing strategy. As such, the size of the set of possible quantized values/indices may be reduced, which (usually) also leads to a corresponding reduction in the bit rate.

In some examples, the parameters may be represented in a first number of frequency bands, and the loop process may further include the following steps: if the third bit rate of the parameters is greater than a target bit rate threshold, reducing the number of frequency bands representing the parameters to a second number that is less than the first number, such that the total number of parameters that need to be quantized and encoded is reduced; and repeating the above steps of the loop process.

In some examples, the parameters are represented in a first number of frequency bands, and the looping process may further include the following steps: if the third bit rate of the parameters is greater than a target bit rate threshold, then reusing (which in some cases is called "freezing") the parameters in one or more frequency bands from the previous frame in the current frame; and repeating the steps of the looping process described above. As an example, when encoding using a particular coding scheme, it is possible to freeze the parameters in certain frequency band(s) (e.g., frequency bands 2, 6, and 10). As a further illustrative example, if all bands are frozen for a period of 2 frames, then the encoder can transmit half of the bands (e.g., even-numbered bands) in frame N, and the remaining half (e.g., odd-numbered bands) in frame N+1 (thereby reducing the total number of parameters that need to be transmitted), which generally means that the decoder will receive all (e.g., 12) updated bands every two frames. In such cases, if one frame is lost, it is usually possible to extrapolate from the last two usable frames. When recovering from packet loss, interpolation can be performed between the bands that were received with a given frame. In general, the result of the freezing process described above will be a decrease in entropy, which does not require changes to the decoder or the entropy coding scheme, with little impact on quality.

In summary, when it is necessary to reduce the total number of bands, this can be done by at least two methods. The first method is frequency resolution reduction, where instead of using N bands, only M bands are used (where M < N), and one or more bands are wider in the M-band configuration than in the N-band configuration. These M bands may be derivatives of the N bands, such that adjacent bands may be grouped together in pairs, threes, etc., or any other groupings that are perceptually meaningful. The second method is time resolution reduction, where the widths of all N bands may remain exactly the same in the frequency domain, but the bands are frozen for a period of x frames (where x > 1). This means that updates for the N bands may be transmitted over a period of x frames, or in other words, only N/x bands out of the N bands need to be updated and transmitted to the decoder with each frame.

In some examples, at least one bit rate threshold may further comprise, in addition to the above-described target bit rate threshold, a maximum bit rate threshold that is greater than the target bit rate threshold. Accordingly, the looped process may further comprise the following steps: before determining the second processing strategy, or reducing the number of frequency bands, or reusing the parameters, obtaining a minimum among the first, second, and third bit rates of the parameters; and exiting the looped process if the minimum is less than or equal to the maximum bit rate threshold.

It is worth noting that if the processing loop terminates at a particular step as described above, this generally means that the final parameter bitrate is the bitrate that is computed at that step (i.e., at the exit of the processing loop). In addition, as noted above, for greater certainty, there may be a certain (e.g., the coarsest) quantization strategy among the given quantization strategies available for parameter quantization that is guaranteed to provide a bitrate that is less than or equal to the target bitrate threshold or the maximum bitrate threshold. In this way, it is possible to guarantee that there is always a solution for a suitable parameter bitrate within the target bitrate threshold or the maximum bitrate threshold.

In some examples, the parameters may comprise one or more prediction parameters (sometimes simply called PR parameters), cross-prediction parameters (sometimes simply called C parameters), and decorrelation parameters (sometimes simply called P parameters). As indicated above, at least some of the parameters are at least partially interrelated, such that they can be calculated based on each other. Of course, as will be understood and apparent to one skilled in the art, there may be any other parameters (types of parameters), depending on different implementations and/or requirements (e.g., use of specific codecs).

As indicated above, the order (or sequence) of parameter computation and quantization can be specified by the first processing strategy indicator.

In some examples, the prediction parameters may be computed and quantized first, the cross-prediction parameters are computed from the quantized prediction parameters and then quantized, and the decorrelation parameters are first computed from the quantized cross-prediction parameters and the quantized prediction parameters and then quantized.

In some examples, the parameters (i.e., the prediction parameters, the cross-prediction parameters, and the decorrelation parameters) may be first computed, then the decorrelation parameters and prediction parameters are quantized, and the cross-prediction parameters are recomputed from the quantized prediction parameters and then quantized.

In some examples, the method can be applied to encoding metadata of an immersive voice and audio services (IVAS) codec or an ambiphony codec. The ambiphony codec can be a first-order ambiphony codec (FOA) or even a higher-order ambiphony codec (HOA). Of course, as will be clear and obvious to a person skilled in the art, any other suitable codecs can be used, depending on the various implementations.

In some examples, the frame size is less than 40 ms, and in particular equal to or less than 20 ms.

According to another aspect of the present invention, a device is provided comprising a processor and a memory coupled to the processor. The processor may be adapted to cause the device to perform all the steps of the example methods described in the present invention.

According to a further aspect of the present invention, a computer program is provided. The computer program may include instructions that, when executed by a processor, cause the processor to perform all the steps of the illustrative methods described in the present invention.

According to another aspect, a machine-readable storage medium is provided. The machine-readable storage medium can store the above-mentioned computer program.

It will be understood that the features of the device and the steps of the method can be interchanged with each other in many ways. In particular, the details of the disclosed method(s) can be implemented by the corresponding device (or system) and vice versa, as will be understood by a person skilled in the art. In addition, it will be understood that any of the above statements regarding the method(s) also apply to the corresponding device (or system) and vice versa.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which

Fig. 1 is a schematic diagram of a block diagram of an encoder/decoder (“codec”) for encoding and decoding signals (bit streams) according to one embodiment of the present invention,

Fig. 2 is a block diagram illustrating an example of a method for frame-by-frame encoding of metadata for an input signal according to one embodiment of the present invention,

Fig. 3 is a block diagram illustrating an example of a processing cycle according to one embodiment of the present invention, and

Fig. 4 is a block diagram illustrating an example of a processing cycle according to another embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The figures (Fig.) and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods described herein will be readily apparent as suitable alternatives that can be used without departing from the stated principles.

Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It should be noted that, where applicable, like or similar reference numerals may be used in the figures and may denote like or similar functionality. The figures depict embodiments of the described system (or method) for illustrative purposes only. It will be readily apparent to one skilled in the art from the following description that alternative embodiments of the structures and methods depicted herein may be employed without departing from the principles disclosed herein.

In addition, in the figures, where connecting elements such as solid or dotted lines or arrows are used to illustrate a connection, relationship or connection between or among two or more other schematic elements, the absence of any of such connecting elements does not imply that the connection, relationship or connection cannot exist. In other words, some connections, relationships or connections between elements are not shown in the drawings so as not to obscure the understanding of the present invention. Additionally, to simplify the illustration, one connecting element is used to represent a plurality of connections, relationships or connections between elements. For example, if a connecting element represents the transmission of signals, data or commands, those skilled in the art will understand that such an element represents one or a plurality of signal paths, as necessary, to affect the transmission.

As stated above, when the frame period of the audio codec (encoder/decoder) approaches 40 ms, or even 20 ms or less, the core audio data can be updated in short time intervals. However, it is generally known that the additional information (or metadata/parameter) does not need to be updated that often. In other words, in codecs with short frame periods, this may generally mean that the parameters will be oversampled if they are all included in every frame (like an audio signal). In some implementations, it may be possible to not send the metadata in every frame, but only update it in every Mth frame (e.g. up to M=4 in some cases). This would generally reduce the average metadata bitrate.

In view of the above, in a broad sense, the application of the technique described in this application can be applied to any parameters or additional information in audio coding where the temporal correlation of the parameters exceeds the codec pitch. For example (but not as a limitation), the frequency interleaved time differential entropy coding procedures can be applied to parameters in the 3rd Generation Partnership Project (3GPP) IVAS codec that model spatial relationships, or in any parametric stereo coding technology that attempts to minimize the codec pitch below 40 ms. However, as will be understood and apparent to those skilled in the art, although embodiments of the present invention may be applied to a first order ambiphony (FOA) codec with presence effect, the approach described herein is generally applicable to any other suitable audio codec (e.g., higher order ambiphony, HOA codecs) where the step or frame size is small, which would generally present some specific difficulties in timely encoding of additional information as mentioned above.

Referring now to Fig. 1, a schematic diagram of a (simplified) block diagram of an encoder/decoder ("codec") 100 for encoding and decoding signals (bitstreams) according to one embodiment of the present invention is shown. In particular, as may be understood by one skilled in the art, the illustrative example of Fig. 1 shows a first-order ambiphony (FOA) codec 100 with a spatial reconstructor (SPAR) for encoding and decoding IVAS bitstreams in FOA format. More particularly, as indicated in the figure, the FOA codec 100 of Fig. 1 includes both passive and active prediction, as will be understood and apparent to one skilled in the art.

In general, for encoding, the IVAS encoder may comprise a spatial analysis and downmix unit that receives audio data including, but not limited to, monophonic signals, stereophonic signals, binaural signals, spatial audio signals (e.g., multi-channel spatial audio objects), FOA, higher-order ambiphony (HoA), and any other suitable audio data. In some implementations, the spatial analysis and downmix unit may implement complex advanced coupling (CACPL) for analyzing/downmixing stereophonic signals/FOA audio signals and/or SPAR for analyzing/downmixing FOA audio signals. In other implementations, the spatial analysis and downmix unit may also implement any other suitable formats.

Now referring again to Fig. 1, the FOA codec 100 may include a SPAR FOA encoder 101, an enhanced voice services (EVS) encoder 105, a SPAR FOA decoder 106, and an EVS decoder 107. The SPAR FOA encoder 101 may be configured to convert the FOA input signal into a set of downmix channels and parameters used to reconstruct the input signal at the SPAR FOA decoder 106. Depending on the various implementations, the downmix signals may range from 1 to 4 channels, and the parameters (which are sometimes also referred to as coefficients) may include, but are not limited to, prediction coefficients (PR), cross-prediction coefficients (C), and decorrelation coefficients (P). It should be noted that SPAR is a method used to reconstruct an audio signal from a downmixed version of the audio signal using the PR, C and P parameters, as will be described in more detail below.

Depending on the number of downmix channels, one of the FOA input signals may always be transmitted unchanged (e.g., channel W, as shown in the present example in Fig. 1), and 1 to 3 other channels (e.g., channels Y, Z, and X, as shown in the present example in Fig. 1) may be transmitted either as residual signals or fully parametrically.

In particular, the prediction parameters may remain the same regardless of the number of downmix channels and may be used to minimize the predicted energy in the residual downmix channels. On the other hand, the cross-prediction parameters may be used to further assist in reconstructing the fully parameterized channels from the residual signals. As such, these parameters would not be needed in the 1- and 4-channel downmix cases, where in the former case there are no residual channels to predict from and in the latter case there are no parameterized channels to predict. In addition, the decorrelator parameters may be used to compensate for the remaining energy not accounted for by the prediction and cross-prediction. As before, the number of decorrelation parameters may depend on the number of downmix channels in each band.

The example of Fig. 1 generally depicts an illustrative embodiment of such a system and how such parameters are accommodated at the decoder side. In particular, the example implementation shown in Fig. 1 depicts a nominal 2-channel downmix, where the channel representation W (which is W for passive prediction or W' for active prediction) is transmitted to the decoder 106 unchanged with one predicted channel Y'. The cross-prediction coefficients (C) allow at least some of the parametric channels to be reconstructed from the residual channels in cases where at least one channel is sent as residual and at least one is sent parametrically, i.e. for 2- and 3-channel downmixes. Thus, generally speaking, for 2-channel downmixes, the parameters C allow some of the channels X and Z to be reconstructed from Y', and the remaining channels are reconstructed using decorrelated versions of the channel W, as described in more detail below. In case of 3-channel downmixing, the residual Y' and X' channels are used to restore only Z.

As will also be clear and obvious to one skilled in the art, it should be noted that in some illustrative implementations, W may be an active channel (or, in other words, a channel with active prediction, hereinafter referred to as W’). As an example (but not limitation), an active channel W that allows some type of mixing of channels X, Y, Z into channel W may be defined as follows:

where f is a suitable constant (e.g. 0.5) allowing at least some of the channels X, Y, Z to be mixed into the channel W, and pr _y , pr _x and pr _z are the prediction coefficients (PR). Accordingly, in cases of passive W, f = 0, so there is no mixing of channels X, Y, Z into the channel W.

In the exemplary implementation of Fig. 1, the SPAR FOA encoder 101 may comprise a predictor (passive or active) unit 102, a remixer unit 103, and an extraction/downmix selection unit 104. In particular, the predictor 102 may receive FOA channels in a 4-channel B (W, Y, Z, X) format and calculates downmix channels (representation W, Y', Z', X').

The extraction/downmix selection unit 104 may extract the SPAR FOA metadata, for example from a payload section of the IVAS bitstream metadata. The predictor unit 102 and the remix unit 103 may then use the SPAR FOA metadata to generate remixed FOA channels (representation W, S ₁ ', S ₂ ' and S ₃ '), which may then be fed to the EVS encoder 105 for encoding them into an EVS bitstream, which may subsequently be encapsulated in an IVAS bitstream sent to the decoder 106.

With reference to the SPAR FOA decoder 106, the EVS bitstream is decoded by the EVS decoder 107, which results in a number (for example, N_dmx = 2, where N_dmx denotes the number of downmix channels) of downmix channels. In some implementations, the SPAR FOA decoder 106 may be configured to perform operations opposite to those performed by the SPAR encoder 101. For example, in the example of Fig. 1, the remixed FOA channels (representation W, S ₁ ', S ₂ ' and S ₃ ') may be reconstructed from the 2 downmix channels using the SPAR FOA spatial metadata. The remixed SPAR FOA channels may then be input to a reverse mixer 111 to reconstruct the downmixed SPAR FOA channels (W, Y', Z' and X' representation). The predicted SPAR FOA channels may then be input to a reverse predictor 112 to reconstruct the original, unmixed SPAR FOA channels (W, Y, Z and X).

It should be noted that in this two-channel example, decorrelator blocks 109-1 (dec ₁ ) and 109-2 (dec ₂ ) may be used to generate decorrelated versions of channel W using a decorrelator in the time domain or frequency domain. The downmix channels and decorrelated channels may be used in combination with SPAR FOA metadata to parametrically reconstruct channels X and Z. Block 108 C may relate to multiplying the residual channel with a 2×1 matrix of coefficients C, thereby creating two cross-prediction signals that may be summed into parametrically reconstructed channels, as shown in the example of Fig. 1. In addition, block 110-1 P ₁ and block 110-2 P ₂ may relate to multiplying the decorrelator output signals with columns of a 2×2 matrix of coefficients P, thereby creating four output signals that may be summed into parametrically reconstructed channels, as shown in the example of Fig. 1.

As indicated above, in some implementations, depending on the number of downmix channels, one of the FOA input signals may be passed to the SPAR FOA decoder 106 unchanged (e.g., the exemplary W channel), and one to three other channels (Y, Z, and X) may be passed to the SPAR FOA decoder 106 either as residual signals or fully parametrically. The PR coefficients, which remain the same regardless of the number N_dmx of downmix channels, may be used to minimize the predicted energy in the residual downmix channels. The C coefficients may be used to further assist in reconstructing the fully parametrically configured channels from the residual signals. As such, the C coefficients may not be required in cases of single- and four-channel downmixes where there would be no residual channels or parametrically configured channels to predict based on them. P factors are used to compensate for the remaining energy not accounted for by the PR and C factors. The number of P factors usually depends on the number N of downmix channels in each band.

In some implementations, SPAR PR coefficients (passive W only) are calculated as follows.

Step 1. Predict all side signals (Y, Z, X) from the main signal W using a prediction matrix consisting of prediction coefficients as follows:

where, as an example, the prediction parameter for the predicted channel Y’ can be calculated as:

Where are elements of the input covariance matrix corresponding to signals A and B, and can be calculated for each band. Similarly, the residual channels Z' and X' have corresponding prediction parameters, namely pr _z and pr _x . The above matrix is known as the prediction matrix.

Step 2. Perform a remix of W and the predicted signals (Y’, Z’, X’) from most to least acoustically significant, where “remix” means changing the order or changing the combination of signals based on some methodology,

One possible implementation of remixing is to reverse the order of the input signals on W, Y', X', and Z', assuming that left-right audio cues are more acoustically significant or important than front-back cues, and front-back cues are more acoustically significant/important than top-bottom cues.

Step 3. Calculate the covariance of the 4-channel post-prediction and downmix remix as:

where the [prediction] and [remix] matrices refer to the matrices used in equations (2) and (4), respectively. The final post-prediction and remix downmix matrix can be written as

where d represents the residual channels (i.e. channels 2 through N_dmx, where N_dmx is the number of downmix channels), and u represents the parametric channels that need to be fully reconstructed (i.e. channels (N_dmx+1) through 4).

For an example of downmixing WS ₁ S ₂ S ₃ with 1-4 channels d and u can represent the following channels shown in Table 1:

N Channels d Channels u 1 -- S ₁ ', S ₂ ', S ₃ ' 2 S ₁ ' S ₂ ', S ₃ ' 3 S ₁ ', S ₂ ' S ₃ ' 4 S ₁ ', S ₂ ', S ₃ ' --

Table 1. Representations of channels d and u

Of greatest interest for calculating SPAR FOA metadata are the values R _dd , R _ud and R _uu .

Step 4. Based on the values of R _dd , R _{ud ,} and R _{uu ,} the codec 100 can determine whether any remaining portion of the fully parametric channels can be cross-predicted from the residual channels passed to the decoder. In some possible implementations, the required additional coefficients C can be calculated as:

Therefore, the C parameter will generally have the form (1×2) for a 3-channel downmix and (2×1) for a 2-channel downmix.

Step 5. Calculate the remaining energy in the parameterized channels that need to be reconstructed by decorrelators 109-1 and 109-2 as:

Where is a constant scaling factor. It should be noted that the residual energy in the upmix channels Res _uu is the difference between the actual energy R _uu (post-prediction) and the reconstructed energy Reg _uu of the cross-prediction.

In some possible implementations, after the off-diagonal elements of the normalized matrix Res _uu have been set to zero, the square root of the matrix can be extracted. P can also be a covariance matrix, and thus be Hermitian symmetric. Thus, only the parameters from the upper or lower triangle need to be transmitted to the decoder 106. The diagonal elements can be real, while the off-diagonal elements can be complex. In some additional possible implementations, the coefficients of P can be further divided into diagonal and off-diagonal elements P _d and P _o , respectively. In some implementations, only the diagonal elements of the coefficients of P are calculated and transmitted to the decoder, and they can be calculated as follows:

Now, on the encoder side, quantization of these parameters may become necessary. In particular, given the dependencies between the three types of parameters (i.e. PR, C and P) as stated above, the order (or sequence) of their calculation and quantization may generally be considered important for the sound quality. According to the present invention, three possible embodiments of methods for achieving this are presented below:

1. All in one

In this embodiment, decorrelators are generally prohibited from compensating for quantized prediction errors.

More precisely, in the first step, the parameters PR are calculated, then C, and then P, as shown above, without quantization. Then, all parameters PR, C, and P are quantized according to a quantization strategy or scheme (e.g., based on suitable quantization ranges and/or quantization levels, as will be understood by a person skilled in the art).

2. Cascade

In general, this particular embodiment allows for accurate prediction and cross-prediction, and decorrelators can compensate for errors caused by quantization.

More precisely, in the first stage, the parameter PR is calculated and then quantized. After that, the parameter C is calculated from the quantized parameters PR and then quantized. Finally, the parameter P is also calculated from the quantized parameters C and then quantized.

3. Partial cascade

In general, this particular embodiment would minimize the P coefficients, thereby allowing accurate cross-prediction, but not allowing decorrelators to compensate for prediction errors.

More precisely, in the first step, the parameters PR, C and P are calculated without quantization, as in the above-described all-in-one embodiment, and then the parameter P is quantized. After that, the parameters PR are also quantized. And finally, the parameter C is recalculated from the quantized parameters PR and then quantized.

In each of the embodiments depicted above, the downmix (including residual signals) can always be computed using quantized prediction coefficients.

As will be understood and obvious to one skilled in the art, the quantization process itself can be defined using a suitable (quantization) range. For example, a range [-a, a] can be defined for some parameters (e.g., parameters PR, C, and off-diagonal elements of parameter P), while another range [0, a] can be defined for others. In addition, several quantization levels can also be defined, which should be uniformly distributed between these end points. In other words, different constraints and step sizes can be configured or defined for each type of parameter (e.g., PR, C, P _d , P _o ). In addition, in some implementations, if the parameters are complex values, the real and imaginary parts can be quantized using the same/different ranges and number of steps, according to the distribution of the parameters.

A possible implementation of the quantization process can be defined as:

or

where x denotes the quantization indices, a denotes the quantization range and qlvl denotes the quantization level.

In some possible implementations, it may be desirable to choose odd values for the quantization levels (i.e. qlvl) to ensure that the quantization point at 0 is accessible, for example for two-sided parameters, as will be apparent to one skilled in the art.

It will be worth noting, as already indicated above, that the example of Fig. 1 generally shows an implementation of passive prediction (i.e. channel W). However, as will be clear and obvious to a person skilled in the art, in some other possible implementations active prediction may be used. Generally speaking, the active channel W may allow some kind of mixing of at least some channels X, Y, Z into channel W, and such active prediction may typically be used in the case of 1-channel downmixing. Accordingly, in cases of passive prediction there will typically be no mixing of channels X, Y, Z into channel W.

Fig. 2 is a block diagram illustrating an example of a method 200 for frame-by-frame encoding of metadata for an input signal according to one embodiment of the present invention. The method 200, as described herein, can be applied, for example, to the codec 100 shown in Fig. 1 (or to any other suitable codec). Metadata can be calculated/computed (e.g., extracted) from the input (audio or video) signal by using a suitable codec (encoder/decoder). Generally speaking, metadata can be used to assist in reconstructing the input signal on the decoder side. Metadata can comprise a plurality of at least partially interrelated parameters that can be calculated from the input signal. In other words, at least some parameters of the input signal can be calculated (e.g., generated or reconstructed) depending on at least some other parameters, so that, depending on various circumstances, not all parameters need to be explicitly transmitted all the time.

The method 200 can be performed iteratively, for example using a looped process (which will be described in detail below) for each frame of the input signal. In particular, the method 200 (or more precisely, the looped process) begins with step S210, in which a processing strategy is determined from a plurality of processing strategies for calculating and quantizing the parameters.

After determining (for example, selecting) the processing strategy in step S210, the loop process proceeds to step S220, in which calculation and quantization of parameters are performed based on the determined processing strategy to obtain quantized parameters.

Thereafter, in step S230, the (quantized) parameters are respectively encoded, and then the (final) bit rate is estimated (e.g. calculated) from the encoded parameters and a decision is made based on the estimated bit rate together with at least one target bit rate threshold value (e.g. predetermined or preconfigured) in step S240.

If the bit rate threshold condition is met, for example the estimated bit rate is equal to or less than the bit rate threshold, the method 200 exits the processing loop. Otherwise, the loop returns to step S210 and continues with steps S210-S240. In particular, when re-entering the loop, a new processing strategy may be determined to satisfy the target bit rate threshold condition.

As will be clear and obvious to a person skilled in the art, the plurality of processing strategies for calculating and quantizing the parameters may be provided in any suitable manner, such as predetermined or preconfigured. Accordingly, the processing strategy may also be determined from the plurality of processing strategies in any suitable manner. For example, depending on the (current) bit rate requirement, a suitable processing strategy may be selected from the plurality of processing strategies, such that the resulting bit rate after performing the calculation, quantization and encoding (e.g. with or without entropy encoding) based on the processing strategy thus selected satisfies the (current) bit rate requirement.

Since the loop process is generally directed (among other things) to quantization-related processing, in some cases the loop process may also be referred to as a quantization loop (or simply a loop for brevity). Similarly, since the processing strategy is also generally directed (among other things) to quantization-related processing, in some cases the processing strategy may also be referred to as a quantization strategy (or, in some other cases, interchangeably referred to as a quantization scheme). In addition, it should be noted that the encoding process may use any suitable encoding procedure, including, but not limited to, entropy encoding or entropy-free encoding (e.g., radix 2 encoding). Of course, any other suitable encoding mechanism may be used, depending on various implementations and/or requirements.

In particular, each of the plurality of processing strategies may comprise a corresponding first pointer indicating an order (or sequence) with respect to the calculation and quantization of individual parameters. In other words, the first pointer may comprise sequence information indicating when and in what order to perform the calculation and quantization of individual parameters. As an example (but not limitation), the first pointer may comprise information indicating that all parameters are first calculated before any of them are quantized.

The loop process will now be described in more detail with reference to the examples shown in Figs. 3 and 4.

As stated above, in codecs with short steps or frame updates, the parameters may be over-sampled if they are all included in each frame. Thus, the main objective of the present invention is to provide mechanisms to minimize the additional information as much as possible, but at the same time maintain a high frame update rate for the basis of audio data and parameters.

In a broad sense, to solve the above-described problem, in particular to estimate the propagation of additional information, the author of the present invention generally proposes a mechanism for introducing time-differential estimates for the parameters of some (frequency) bands together with non-differential estimates for the parameters of other (frequency) bands. The proposed approach alternates coding of the bands in a time-differential manner and in a non-differential manner so that each band is regularly updated by a non-differential calculation without the need for a complete update of the parameters. The key idea is that as the frame size decreases, the correlation of the parameters from frame to frame increases and, thus, by using time-differential coding of the parameters, it is possible to achieve increased coding efficiency.

In addition to frequency interleaving of time-differential coding, the idea of an iterative and stepwise approach to the selection of the optimal parameter quantization scheme is also introduced, which searches for the "best" (or optimal) quantization scheme from several alternatives. In this case, the term "best" or "optimal" may not necessarily refer to the quantization scheme with the lowest parameter bitrate, but to the scheme that facilitates the state for the decoder.

For example, the use of time-differential coding in general may have the disadvantage that it primarily introduces frame-to-frame state that may cause problems when the audio stream may experience packet loss during transmission. In this case, both audio data and parameters may be lost, and any parameters updated using time-differential coding may experience potential artifacts in several subsequent frames. The present invention does not generally consider mitigating this problem using a decoder. Instead, this problem is generally solved (alleviated) by selecting an appropriate quantization scheme that will limit such behavior as much as possible. In a broad sense, mitigation by coding (on the encoder side) generally includes an iterative process of selecting quantization and entropy coding that attempts to minimize the possible extent of packet loss artifacts due to the use of time-differential coding.

Now turning back to the figures, Fig. 3 is a block diagram schematically depicting an example of a processing cycle 300 according to one embodiment of the present invention.

The processing cycle 300 begins with step S310, in which the first bit rate (hereinafter denoted b1) is calculated (or estimated). In some possible implementations, the entropy of the non-differentially and/or frequency-differentially quantized parameters is estimated for each frame. In some other possible implementations, the first bit rate b1 can be calculated at least among non-differential and frequency-differential coding schemes encoded using (trained) entropy coders (e.g., Huffman coding or arithmetic coding).

In step S320, the first bit rate b1 is compared with the target bit rate (hereinafter denoted t). If the estimated bit rate b1 of the parameters is within (i.e. equal to or less than) the target bit rate t, then the processing loop is exited. As a result, the parameters are encoded such that any additional available bits are provided to the audio encoder to increase the bit rate of the underlying audio data.

If step S320 fails (i.e. the estimated bit rate b1 is greater than the target bit rate t), then step S330 calculates the second bit rate (hereinafter denoted b2) of the quantized parameters. In some possible implementations, the second bit rate b2 can be calculated in a non-differential manner without entropy coding (e.g. using radix 2 coding).

Then, in step S340, the second bit rate b2 is compared with the target bit rate t. If the second bit rate b2 is within (i.e., equal to or less than) the target bit rate t, then the processing loop is exited.

Otherwise, in step S350, the third bit rate (hereinafter denoted b3) of the parameters is calculated. In some possible implementations, the third bit rate b3 can be calculated by time-differential coding using (trained) entropy coders. In some further possible implementations, a subset of the parameter values in the current frame can be quantized and then subtracted from the quantized parameter values in the previous frame, and the differential quantized value and entropy of the parameter can be calculated.

In step S360, if the calculated bit rate b3 is equal to or less than the threshold value t, then the processing loop is exited and the parameters are encoded with the provided bit rate and additional bits are provided for encoding the audio data.

Otherwise, at step S370, various measures may be implemented to ultimately ensure compliance with the target bit rate threshold t.

For example, in some possible implementations, a second, coarser, processing strategy (quantization strategy) may be selected from a plurality of processing strategies. In such cases, as will be understood and apparent to one skilled in the art, the quantization process may include several levels of increasingly coarser quantization, such as, for example, fine, average, coarse, and very coarse quantization strategies. Then, after determining (e.g., selecting) a coarser quantization strategy, the processing loop repeats steps S310 through S360.

In some other possible implementations, the reduction of the number of frequency bands may be performed in step S370. Then the above steps (i.e. steps S310 to S360) may be repeated with the reduced band configuration. This will generally reduce the total number of parameters for quantization and may often result in a low bit rate for (at least) some frames.

Alternatively or additionally, in some other implementations it is also possible to perform a step of freezing (i.e. reusing) the parameters in the band from the previous frame. This will essentially prevent the parameter from changing over time, which will result in a decrease in entropy for time-differential entropy coding. For example, as shown in Table 2 (which will be described in detail below), when coding using the coding scheme 4a, it is possible to freeze the parameters in frequency bands 2, 6, and 10. This will typically result in a decrease in entropy, no change in the decoder or the entropy coding scheme, and little impact on quality. It should be noted that the above example with 2, 6, and 10 is only an illustrative example and many band configurations can be frozen in several frames, as will be understood and apparent to one skilled in the art. For example, if all the bandwidths are frozen in a 2-frame period, the encoder can send half the bandwidths in frame N and the remaining half in frame N+1 (thereby reducing the total number of parameters to send), which generally means that the decoder will receive all (say, 12) updated bandwidths every two frames. In such cases, if one frame is lost, it is usually possible to extrapolate from the last two usable frames. When recovering from packet loss, it is possible to interpolate between the bandwidths that were received with a given frame.

It should be noted that if the loop exit is performed at step x, then the final bitrate of the parameters is the bitrate calculated at that step x.

In addition, in some implementations it may be possible (or even desirable) to consider designing the bitrate b3 with the coarsest quantization strategy (out of a given set of quantization strategies available for quantizing the parameters) as guaranteeing a bitrate that is less than the target bitrate threshold t. In such cases, it may be possible to guarantee that there is always a solution for a suitable parameter bitrate within the target bitrate t.

Fig. 4 is a block diagram illustrating an example of a processing cycle 400 according to another embodiment of the present invention. In particular, identical or similar reference numerals in the cycle 400 of Fig. 4 generally refer to identical or similar elements in the cycle 300 that are shown in Fig. 3, so that their repeated description may be omitted for the sake of brevity.

In particular, the processing loop of Fig. 4 may be particularly suitable for cases where two bit rate thresholds are used (represented as a target bit rate threshold t1 and a maximum bit rate threshold t2), as opposed to the scenario with a single target bit rate threshold depicted in Fig. 3. In a broad sense, the target bit rate threshold t or t1 may be considered a goal or target that is desirable to achieve, while the maximum bit rate threshold t2 may be considered simply as a "hard" threshold that should not be exceeded.

More specifically, steps S410 to S470 are similar to the steps (i.e., steps S310 to S370) shown in Fig. 3, so that their repeated description may be omitted for the sake of brevity.

However, instead of directly switching to step S470, if the condition of step S460 is not met, an additional step S461 is introduced with calculating the fourth bit rate (b4) as the minimum among the bit rates b1, b2 and b3. Then, the fourth bit rate b4 is compared with the maximum threshold value t2 of the bit rate in step S462.

If the fourth bit rate b4 is equal to or less than the maximum bit rate threshold value t2, the processing loop 400 is exited; otherwise, the processing loop 400 continues to step S470 (which is essentially similar to step S370 in Fig. 4) and steps S410 to S462 are repeated.

As shown in Fig. 3, if the loop exits at step x, then the final parameter bitrate is the bitrate calculated at that step x.

Moreover, in some implementations it is also possible (or even desirable) to consider designing the bitrate b3 with the coarsest quantization strategy (from a given set of quantization strategies available for quantizing the parameters) as guaranteeing a bitrate that is less than the maximum bitrate threshold t2. In such cases, it is possible to guarantee that there will always be a solution for a suitable parameter bitrate within the maximum bitrate t2.

To summarize, steps S310, S330 and S350 of Fig. 3 and, accordingly, also steps S410, S430 and S450 of Fig. 4 generally do not affect the audio quality. However, step S461 of Fig. 4 could degrade the quality due to the effect on both the bit rate of the audio data and the bit rate of the parameters. In addition, any possible technologies mentioned above in steps S370 of Fig. 3 and S470 of Fig. 4 (for example, switching to coarser quantization, reducing the bandwidth by reducing the frequency resolution, reducing the bandwidth by reducing the time resolution, etc.) will generally have a negative effect on the quality. Thus, the steps in the examples of Figs. 3 and 4 are arranged in such an order as to minimize the deterioration in quality or to cope with limitations in other areas. In a broad sense, the method described in the present invention seeks to select one or more of the above-depicted technologies to maintain a balance between metadata bitrate reduction and perceived quality.

There are also two additional considerations regarding the specific order of the above steps and the reason for the possible presence of two target parameter bitrates (i.e. bitrates t1 and t2).

In particular, the stepwise order allows the procedure to be terminated if constraints are satisfied. This will generally reduce the computational burden of sequential computations, since it will usually not be necessary to go through all available steps.

In addition, the ordering also ensures that alternatives are implicit. For example, placing non-differential entropy coding as the first stage will typically mean that this alternative is preferred if it satisfies the constraints. This represents a simplification of the encoder to minimize state in order to improve performance under packet loss.

Moreover, the ability to use two targets (t1 and t2) generally provides more control over the trade-off between audio data bitrate and parameter bitrate.

The interleaving used to achieve time-differential coding will be described in more detail below.

Some possible implementations for controlling the interleaving of time-differential entropy coding are shown in Table 2.

Coding scheme Time-differential encoding, bands 1-12 base 0 0 0 0 0 0 0 0 0 0 0 0 4a 0 1 1 1 0 1 1 1 0 1 1 1 4b 1 0 1 1 1 0 1 1 1 0 1 1 4c 1 1 0 1 1 1 0 1 1 1 0 1 4d 1 1 1 0 1 1 1 0 1 1 1 0

Table 2. Alternating time-differential coding schemes

In this particular example, a total of 5 configurations are proposed for encoding the metadata bitstream, each consisting of 12 (frequency) bands. More specifically, the band labeled 0 is encoded non-differentially, and the band labeled 1 is encoded time-differentially (i.e., the parameter is quantized and subtracted from the quantized parameter in the previous frame).

As described in this example, the bit rate of the parameters of each frame is first estimated by coding in a non-differential manner (i.e. bases) using quantization of the parameters (e.g. see step S410 or S510). Then, in step S450 or S550, a time-differential coding scheme is selected (if necessary) based on the coding scheme of the previous frame.

An example of mapping from the previous frame's time-differential coding scheme to the current frame's time-differential coding scheme is shown below in Table 3:

Previous frame encoding scheme Time-differential coding scheme for the current frame base 4a 4a 4b 4b 4c 4c 4d 4d 4a

Table 3. Display of time-differential coding schemes

It should be noted that in the present example the term "base" as used in Table 3 generally refers to a non-differential coding scheme. Thus, as can be seen from Table 3, the time-differential coding always cycles from 4a to 4d (and back). It is possible to continue cycling without even requiring the implementation of non-differential coding. And in this particular example the maximum information or "state" of the codec to be stored is the current frame and the three previous frames (i.e. a total of four frames). Of course, as will be understood and apparent to one skilled in the art, such numbers as 5 configurations and 12 (frequency) bands, etc. are used merely as examples for illustrative purposes, and any other suitable number may be used, depending on different implementations and/or requirements. Similar or similar arguments apply to the switching between coding schemes as depicted in Table 3, which may likewise use any suitable technique.

It should be noted that if a different quantization scheme is selected, then the indices from the previous frame, quantized using a different quantization scheme, may first be mapped to the indices of the current frame. In general, the mapping step may be necessary to allow time-differential encoding of parameters, for example when the number of quantization levels changes from one frame to the next, thereby allowing time-differential encoding to be performed between frames without resorting to transmitting a non-differential frame each time the quantization scheme is changed.

As a possible example, the mapping of indices can be done based on the formulas:

Where denotes the indices of the current frame after display, denotes the indices of the previous frame, denotes the quantization level of the current frame and indicates the quantization level of the previous frame.

As a simple illustrative example, assume the quantization range is 0 to 2, and assume the previous quantization levels are 11. In the case of uniform quantization, this would generally mean that each quantization step would be 0.2. Further assume the current quantization levels are 21, which means that each quantization step is 0.1 under uniform quantization. Based on these assumptions, if the quantized value in the previous frame was 0.4, then with 11 uniform quantization levels, the next previous index is obtained . The mapping provides the quantized indices of the previous frame's metadata as if they were quantized using the quantization levels of the current frame. So in this example, if the quantization levels in the current frame are 21, then the quantized value 0.4 would be mapped to . After the mapped indices are calculated, the difference between the indices of the current frame and the previous frame is calculated and this difference is encoded. Similar or similar approaches may also be applied to frequency differential coding, if necessary, as will be understood and obvious to a person skilled in the art.

Of course, any other suitable mapping schemes (e.g. using a lookup table or similar) may be used, depending on different implementations and/or requirements.

Moreover, as indicated above, one metadata parameter can be quantized from a continuous numerical value into an index representing a discrete value. In non-differential coding, the information encoded for this metadata parameter corresponds directly to this index. In time-differential coding, the encoded information is the difference between the index of the metadata parameter from the current frame and the index of the same metadata parameter from the previous frame. As will be understood and obvious to a person skilled in the art, the above-described general idea of time-differential coding can be further extended, for example to a plurality of frequency bands. Accordingly, the metadata parameter can be similarly extended, for example to a plurality of parameters, respectively related to a plurality of frequency bands, if necessary. Frequency-differential coding follows a similar principle, but the encoded difference is the difference between the metadata of one frequency band of the current frame and the metadata of another frequency band of the current frame (as opposed to the current frame minus the previous frame in time-differential coding). As a simple example (but not limitation), suppose that a0, a1, a2 and a3 denote indices of parameters in 4 frequency bands of a particular frame, then in one example of implementation the frequency differential indices may be a0, a0-a1, a1-a2, a2-a3. As will be apparent to one skilled in the art, the general idea of (time- and/or frequency-) differential coding is that metadata may typically vary slowly from frame to frame or from frequency band to frequency band, so that even if the original metadata value was large, the difference between it and the metadata of the previous frame or the difference between it and the metadata of another frequency band is likely to be small. This is advantageous since in general parameters with statistical distributions that tend to zero can be encoded using fewer bits. Thus, even if some of the illustrative implementations briefly or simply refer to time-differential coding, one skilled in the art will understand that frequency-differential coding can also be applied to them (possibly with some suitable adaptation).

Some additional possible examples of the present invention may relate to a process for processing an input audio signal represented in sub-bands to obtain a downmixed signal and associated metadata, which may be performed by one or more processors. The process may include, for each sub-band, determining a downmix matrix and associated metadata; and re-mixing each of said sub-bands according to said downmix matrix to obtain said downmixed signal. One or more quantization strategies and one or more encoding strategies may be used to encode the metadata, taking into account a limitation of the target and/or maximum bitrate of the metadata.

In some implementations, the process may include non-differential entropy coding of all subbands. The process may further include frequency-differential entropy coding of all subbands. The process may further include combining frequency interleaving with time-differential coding of quantized parameters corresponding to selected subbands for a low-latency audio codec, as described in detail below.

The process may further include non-entropy encoding of sub-band metadata. Iteratively performing the steps to find a suitable encoding strategy to meet the bit rate and audio quality requirements and to reduce the decoder state. The process may further include reducing the frequency resolution by reducing the number of sub-bands in which the spatial metadata will be encoded, for example from 12 bands to 6 bands. The process may include reducing the temporal resolution by freezing the metadata of one or more sub-bands in time so that the sub-band metadata does not need to be transmitted. The process may include using multiple quantization strategies, where each strategy is a combination of quantization levels for different parameters of the spatial metadata, and the process may further include selecting one of these three quantization strategies to ensure that the target bit rates are achieved. The process may include iteratively performing the steps to find a suitable quantization scheme to meet the bit rate and audio quality requirements. The iterative method aims to obtain the desired metadata bitrate with the desired quantization scheme, minimal computational complexity, and reduced decoder state. If the desired quantization level does not fall within the desired bitrate range, then revert to a (e.g., coarser) quantization scheme by ensuring minimal impact on audio quality.

In some implementations, mapping indices from previous frames quantized to a different number of levels to indices of the current frame allows time-differential coding to be performed between frames without having to transmit a non-differential frame each time a different quantization level is needed.

In various implementations, quantization (transforming continuous values into discrete indices for encoding) may involve determining the best value for the coefficients according to current needs, by changing the order of calculation and quantization of successive metadata coefficients.

A computing device implementing the above-described techniques may have the following exemplary architecture. Other architectures are possible, including architectures with more or fewer components. In some implementations, an exemplary architecture includes one or more processors (e.g., dual-core Intel® Xeon® processors), one or more output devices (e.g., an LCD display), one or more network interfaces, one or more input devices (e.g., a mouse, keyboard, touch display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components may exchange messages and data over one or more communication channels (e.g., buses), which may use different hardware and software to facilitate the transfer of data and control signals between the components.

The term "machine-readable medium" refers to a medium that is involved in providing instructions to a processor for execution, including, but not limited to, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include, but are not limited to, coaxial cables, copper wire, and optical fiber.

The machine-readable medium may further include an operating system (e.g., a Linux® operating system), a network data transfer module, an audio interface manager, an audio processing manager, and a live content distributor. The operating system may be multi-user, multi-processor, multi-tasking, multi-threaded, real-time, etc. The operating system performs basic tasks, including, but not limited to: recognizing incoming data from network interfaces 706 and/or devices 708 and transmitting output data to them; tracking and managing files and directories on machine-readable media (e.g., in memory or a storage device); managing peripheral devices; and managing traffic of one or more communication channels. The network data transfer module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP/IP, HTTP, etc.).

The architecture may be implemented in a parallel or peer-to-peer infrastructure or on a single device with one or more processors. The software may include multiple software components or may be a single body of code.

The described features can be advantageously implemented in one or more computer programs that can be executed on a programmable system that includes at least one programmable processor connected with the ability to receive data and commands from a data storage system and transmit them to it, at least one input device and at least one output device. A computer program is a set of commands that can be used, directly or indirectly, in a computer to perform a certain activity or to obtain a certain result. A computer program can be written in a programming language of any form (for example, Objective-C, Java), including compiled or interpreted languages, and it can be deployed in any form, for example as a stand-alone program or as a module, component, subroutine, web application executed in a browser, or another element suitable for use in a computing environment.

Suitable processors for executing a program consisting of instructions include, by way of example, both general-purpose and special-purpose microprocessors, and a single processor or one of several processors or cores in a computer of any type. In general, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The key elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer may also include, or be operatively connected for data exchange with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by application-specific integrated circuits (ASICs) or may be integrated into such circuits.

To provide interaction with a user, the features may be implemented on a computer comprising a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or a device with a Retina display, for displaying information to the user. The computer may comprise an input device with a touch surface (e.g., a touch screen) or a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. The computer may comprise a voice input device for receiving voice commands from the user.

The features may be implemented in a computer system comprising a server component, such as a data server, or comprising a middleware component, such as an application server or an Internet server, or comprising a client component, such as a client computer comprising a graphical user interface or an Internet browser, or any combination thereof. The system components may be connected via any form or medium of digital data transmission, such as a communication network. Examples of communication networks include, for example, LANs, WANs, and computers and networks that form the Internet.

A computing system may include clients and servers. The client and server are typically remote from each other and typically interact via a communications network. The client and server are interconnected by computer programs running on the respective computers and having a client-server interconnection with each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying the data to a user and receiving user input from a user interacting with the client device). Data generated on the client device (e.g., as a result of user interaction) may be received from the client device by the server.

A system of one or more computers may be configured to perform specific actions by having software, firmware, hardware, or a combination thereof installed in the system that, when running, cause the system to perform actions. One or more computer programs may be configured to perform specific actions by including instructions that, when executed by a data processing device, cause the device to perform actions.

Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as operating in certain combinations, and even initially claimed as such, one or more features from a claimed combination may in some cases be excluded from the combination, and the claimed combination may be directed to a subcombination or a variant of a subcombination.

Similarly, although operations are depicted in the drawings in a particular order, this should not be construed as a requirement that these operations be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be understood as requiring the same separation in all embodiments, and it should also be understood that the software components and systems as a whole described may be integrated together into a single software product or packaged into multiple software products.

Unless specifically stated otherwise, as is evident from the following discussions, it is understood that in all discussions of the present invention, where terms such as "processing," "calculating," "computing," "determining," "analyzing," or the like are used, they refer to the action and/or processes of a computer or computing system or similar electronic computing devices that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

Reference throughout this specification to "one exemplary embodiment," "some exemplary embodiments," or "an exemplary embodiment" means that a particular feature, structure, or characteristic described in connection with the exemplary embodiment is included in at least one exemplary embodiment of the present invention. Thus, appearances of the phrases "in one exemplary embodiment," "in some exemplary embodiments," or "in an exemplary embodiment" in various places throughout this specification are not necessarily all referring to the same exemplary embodiment. Furthermore, particular features, structures, or characteristics may be combined in one or more exemplary embodiments in any suitable manner, as will be apparent from this specification to one skilled in the art.

In the context of this document, unless otherwise indicated, the use of the ordinal numerals "first," "second," "third," etc., to describe an ordinary object merely indicates that reference is being made to different instances of similar objects, and is not intended to indicate that the objects so described are to be in a given sequence, whether in time, space, rank, or in any other sense.

In the claims below and in the present description, any of the terms "comprising", "consisting of" or "which comprises" is an open term, which means including at least the elements/features that follow it, but not excluding the rest. Therefore, the term "comprising" when used in the claims should not be interpreted as limiting with respect to the means or elements or steps listed after it. For example, the scope of the expression "a device comprising A and B" should not be limited to devices consisting only of elements A and B. Any of the terms "comprising" or "which comprises" used in this document is also an open term, which also means including at least the elements/features that follow this term, but not excluding the rest. Thus, "comprising" means "comprising" and is synonymous with it.

It should be understood that in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together in a single exemplary embodiment, figure, or description thereof for the purpose of simplifying the description and promoting an understanding of one or more different aspects of the invention. Such manner of disclosure, however, should not be interpreted as reflecting an intent that the claims require more features than those expressly recited in each claim. Instead, as the following claims reflect, aspects of the present invention reside in less than all of the features of a single exemplary embodiment described above. Accordingly, the claims following the Description section are hereby expressly incorporated into this Description section, with each claim independently representing a separate exemplary embodiment of the present invention.

Furthermore, although some exemplary embodiments described herein include some and not other features included in other exemplary embodiments, combinations of features from different exemplary embodiments are intended to be within the scope of the present invention and to form different exemplary embodiments, as will be understood by those skilled in the art. For example, in the following claims, any of the claimed exemplary embodiments may be used in any combination.

The description provided herein sets forth numerous specific details. However, it is understood that exemplary embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail in order not to obscure the understanding of this description.

Thus, although what are believed to be the best embodiments of the present invention have been described herein, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present invention, and all such changes and modifications are intended to be claimed as being within the scope of the present invention. For example, any formulas given above are merely examples of procedures that may be used. Functionality may be added to or deleted from block diagrams, and operations may be interchanged among functional blocks. Steps may be added to or deleted from methods described within the scope of the present invention.

Various aspects and implementations of the present invention may also be understood from the following numbered exemplary embodiments (EEEs), which do not constitute claims.

PPVO 1. A method for processing an input audio signal, represented in sub-bands, to obtain a downmixed signal and associated metadata, the method comprising:

for each sub-band, defining a downmix matrix and associated metadata; and;

re-mixing each of said sub-bands according to said downmix matrix to obtain said downmixed signal.

PPVO 2. The method according to PPVO 1, in which the encoding of metadata is performed using one or more quantization strategies and one or more encoding strategies taking into account the limitation of the target and/or maximum bitrate of the metadata.

PPVO 3. A method according to PPVO 2, which includes non-time-differential entropy coding of all sub-bands.

PPVO 4. The method according to PPVO 3, which includes combining frequency interleaving with time-differential coding of quantized parameters corresponding to selected subbands, for a low-latency audio codec.

PPVO5. A method according to PPVO 4 that includes non-entropy coding of sub-band metadata.

PPVO 6. The method according to PPVO 5, which includes iteratively going through steps 3) through 5) to find a suitable coding strategy to meet the bit rate and audio quality requirements and to reduce the decoder state.

PPVO 7. A method according to PPVO 6, which includes reducing the number of bands transmitted by means of a combination of metadata in sub-bands.

PPVO 8. A method according to PPVO 7, which includes fixing in time the metadata of one or more sub-bands, such that the sub-band metadata does not need to be transmitted.

PPVO 9. A method according to PPVO 8, which involves using multiple quantization levels for given metadata to ensure that target bitrates are achieved.

PPVO 10. The method according to PPVO 9, which includes iteratively performing PPVO steps 3-9 to find a suitable quantization scheme to meet the bit rate and audio quality requirements.

PPVO 11. A method according to PPVO 3 or PPVO 9, in which the mapping of indices from previous frames quantized to a different number of levels to indices of the current frame allows time-differential coding to be performed between frames without resorting to transmitting a non-time-differential frame each time a different quantization level is needed.

PPVO 12. A method according to any of the above PPVO, wherein the quantization comprises determining the best value for the coefficients according to the current needs, by changing the order of calculation and quantization of successive metadata coefficients.

PPVO 13. A system containing:

one or more processors; and

a non-transitory machine-readable medium on which instructions are stored that, when executed by one or more processors, cause the one or more processors to perform operations according to any of PPVO 1-12.

PPVO 14. A non-transitory machine-readable medium that stores instructions that, when executed by one or more processors, cause one or more processors to perform operations according to any of PPVO 1-12.

Claims (50)

1. A method for frame-by-frame encoding of metadata for an input signal, wherein the metadata comprises a plurality of at least partially interrelated parameters calculated from the input signal, wherein the method includes for each frame: iterative execution, using a looped process, of the following steps: selecting a quantization strategy from a set of quantization strategies for calculating and quantizing parameters; calculating and quantizing the parameters based on the selected quantization strategy to obtain quantized parameters; and coding of quantized parameters, wherein each of the plurality of quantization strategies comprises a respective first pointer indicating an order related to the calculation and quantization of individual parameters; and in this case, the quantization strategy is selected based on the target bitrate threshold value, wherein the quantization strategy determined for the current frame differs from the quantization strategy determined for the previous frame; and wherein the parameter coding includes time-differential coding in different quantization strategies. 2. The method according to paragraph 1, characterized in that the quantization strategy is selected so that the bitrate of the encoded quantized parameters is equal to the target threshold bitrate value. 3. The method according to claim 1 or claim 2, characterized in that each of the plurality of quantization strategies additionally contains a corresponding second pointer indicating information for performing quantization of the parameters; and wherein, in particular, the information for performing quantization of the parameters contains corresponding quantization ranges and/or quantization levels for a plurality of parameters. 4. A method according to any of the preceding paragraphs, characterized in that the coding of the parameters includes time- and/or frequency-differential coding. 5. A method according to any of the preceding paragraphs, characterized in that the first pointer contains information indicating that: all parameters are calculated before quantization; the parameters are calculated separately and then quantized sequentially one after another, and at least one parameter of the plurality of parameters is calculated based on one or more other quantized parameters of the plurality of parameters; or all parameters are calculated before any parameter is quantized; and at least one of the parameters is recalculated based on another quantized parameter and the recalculated parameter is quantized. 6. A method according to any of the preceding paragraphs, characterized in that the method additionally includes, before encoding the quantized parameters: mapping the indices of quantized parameters from the previous frame to the indices of the current frame. 7. A method according to any of the preceding paragraphs, characterized in that the looped process includes the following steps: quantization and coding of parameters in a non-differential and/or frequency-differential manner using an entropy coder according to the quantization strategy; estimating the first parameter bitrate for the encoded parameters; and exit the loop process if the first parameter bitrate is less than or equal to the target bitrate threshold. 8. The method according to paragraph 7, characterized in that the looped process additionally includes the following steps: if the first parameter bitrate is greater than the target bitrate threshold: quantization and encoding of parameters in a non-differential manner without entropy according to the quantization strategy; estimating the second parameter bitrate for the encoded parameters; and exit the loop process if the second parameter bitrate is less than or equal to the target bitrate threshold. 9. The method according to paragraph 8, characterized in that the looped process additionally includes the following steps: if the second parameter bitrate is greater than the target bitrate threshold: quantization and encoding of parameters in a time-differential manner using an entropy coder according to the quantization strategy; estimating the third parameter bitrate for the encoded parameters; and exit the loop process if the third parameter bitrate is less than or equal to the target bitrate threshold. 10. The method according to paragraph 9, characterized in that time-differential quantization and coding are performed on a subset of parameters using a method with alternating frequencies relative to the previous frame. 11. The method according to claim 10, characterized in that the time-differential quantization and coding are performed by cyclically passing through a series of time-differential coding schemes with alternating frequencies, so that for each cycle a different subset of parameters are quantized and coded time-differentially, while the remaining parameters are quantized and coded non-differentially. 12. The method according to any one of paragraphs 7-11, characterized in that the selected quantization strategy is the first quantization strategy, and wherein the looped process additionally includes: if the third parameter bitrate is greater than the target bitrate threshold: selecting a second quantization strategy from a plurality of quantization strategies such that the bit rate of using the second quantization strategy is expected to be less than the bit rate of using the first quantization strategy; and repeating the stages of the looped process according to point 7. 13. The method according to any one of paragraphs 7-11, characterized in that the parameters are presented in a first number of frequency bands, and wherein the looped process additionally includes the following steps: if the third parameter bitrate is greater than the target bitrate threshold: reducing the number of frequency bands representing the parameters to a second number that is less than the first number, such that the total number of parameters that need to be quantized and encoded is reduced; or reusing parameters in one or more frequency bands from the previous frame in the current frame; and repeating the stages of the looped process according to point 7. 14. The method according to paragraph 12 or paragraph 13, characterized in that the looped process additionally includes the following steps: before choosing a second quantization strategy, or reducing the number of frequency bands, or reusing parameters: obtaining the minimum among the first, second and third bitrate parameters; and exit the loop process if the minimum is less than or equal to the maximum bitrate threshold. 15. A method according to any of the preceding paragraphs, characterized in that the method is applied to encoding metadata of a codec for voice and sound services with presence effect, IVAS, or an ambiphony codec. 16. A device for frame-by-frame encoding of metadata for an input signal, comprising a processor and a memory device connected to the processor, wherein the processor is adapted to cause the device to perform the method according to any of the preceding claims. 17. A machine-readable data carrier on which a program is stored containing instructions that, when executed by a processor, cause the processor to execute the method according to any of paragraphs 1-15.