WO2010042024A1 - Energy conservative multi-channel audio coding - Google Patents

Energy conservative multi-channel audio coding Download PDF

Info

Publication number
WO2010042024A1
WO2010042024A1 PCT/SE2009/051071 SE2009051071W WO2010042024A1 WO 2010042024 A1 WO2010042024 A1 WO 2010042024A1 SE 2009051071 W SE2009051071 W SE 2009051071W WO 2010042024 A1 WO2010042024 A1 WO 2010042024A1
Authority
WO
WIPO (PCT)
Prior art keywords
channel
energy
audio
energies
representation
Prior art date
Application number
PCT/SE2009/051071
Other languages
French (fr)
Inventor
Erik Norvell
Martin Sehlstedt
Anisse Taleb
Original Assignee
Telefonaktiebolaget Lm Ericsson (Publ)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget Lm Ericsson (Publ) filed Critical Telefonaktiebolaget Lm Ericsson (Publ)
Priority to CN2009801401393A priority Critical patent/CN102177542B/en
Priority to EP09819478.0A priority patent/EP2345027B1/en
Priority to US13/122,880 priority patent/US9330671B2/en
Priority to JP2011530989A priority patent/JP5608660B2/en
Publication of WO2010042024A1 publication Critical patent/WO2010042024A1/en

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing

Definitions

  • the present invention relates to an audio encoding method and a corresponding audio decoding method, as well as an audio encoder and a corresponding audio decoder.
  • Adaptation at intermediate gateways If a part of the network becomes congested, or has a different service capability, a dedicated network entity as illustrated in Fig. 1 , performs the transcoding of the service. With scalable codec this could be as simple as dropping or truncating media frames.
  • PS Parametric Stereo
  • MPEG4- SLS provides progressive enhancements to the core AAC/BSAC all the way up to lossless with granularity step down to 0.4 kbps.
  • AOT Audio Object Type
  • An Audio Object Type (AOT) for SLS is yet to be defined.
  • CfI Call for Information
  • the latest standardization efforts is an extension of the 3GPP2/VMR-WB codec to also support operation at a maximum rate of 8.55 kbps.
  • the Multirate G.722.1 audio/video conferencing codec has previously been updated with two new modes providing super wideband (14 kHz audio bandwidth, 32 kHz sampling) capability operating at 24, 32 and 48 kbps. Further standardization efforts were aiming to add an additional mode that would extend the bandwidth to 48 kHz full-band coding. The end result was the new stand-alone codec G.719, which provides low complex full-band coding from 32 to 128 kbps in steps of 16 kbps.
  • G.729 The main standardization effort is taking place in ITU-T, (Working Party 3, Study Group 16).
  • G.729.1 This extension is scalable from 8 to 32 kbps with 2 kbps granularity steps from 12 kbps.
  • the main target application for G.729.1 is conversational speech over shared and bandwidth limited xDSL-links, i.e. the scaling is likely to take place in a Digital Residential Gateway that passes the VoIP packets through specific controlled Voice channels (Vc's).
  • Vc's Digital Residential Gateway that passes the VoIP packets through specific controlled Voice channels
  • ITU-T has also recently (Sept. 2008) approved the recommendation for a completely new scalable conversational codec, G.718.
  • the codec comprises a core rate of 8.0 kbps and a maximum rate of 32 kbps., with scaling steps at 12.0, 16.0 and 24.0 kbps.
  • the G.718 core is a WB speech codec inherited from VMR-WB, but also handles NB input signals by upsampling to the core samplerate. Further a joint extension of the G.718 and G.729.1 codecs that will bring super wideband and stereo capabilities (32 kHz sampling/2 channels) is currently under standardization in ITU-T (Working Party 3, Study Group 16, Question 23). The qualification period ended July 2008.
  • G.718 have this feature. Typically this is achieved by stepwise re-encoding of the coding residual from the previous layer.
  • the embedded layered structure is attractive since lower bitrates can be decoded by simply discarding the upper layers.
  • the embedded layering may not be optimal when considering the higher bitrates and a layered codec usually performs worse than a fixed bitrate codec at the same bitrate.
  • Other codecs that can be mentioned here is the SNR scalable MPEG4-CELP and G.727 (Embedded ADPCM).
  • G722 Sub band ADPCM
  • G.729.1 operates with a cascaded CELP codec for the bitrates 8 and 12 kbps, but provides WB signals at 14 kbps using a bandwidth extension to fill the range from 4 kHz to 7 kHz.
  • the bandwidth extension typically creates an excitation signal from the lower band by spectral folding or other mappings, which is further gain adjusted and shaped with a spectral envelope to simulate the higher end frequency spectrum. Although the solution might sound good, the extended spectrum does not generally match the input signal in an MSE sense.
  • the bandwidth extension used at lower rates is typically replaced with coded content in higher layers. This is the case for G.729.1 where the spectrum is gradually replaced with coded spectrum on a subband basis.
  • G.718 exhibits the same feature and uses bandwidth extension from 6.4 kHz to 7.0 kHz for rates 8, 12 and 16 kbps. For the rates 24 and 32 kbps, the bandwidth extension is disabled and replaced with coded spectrum.
  • MPEG4-CELP specifies a bandwidth scalable coding system for 8 and 16 kHz sampled input signals.
  • audio scalability can be achieved by:
  • Dropping audio channels e.g., mono consist of 1 channel, stereo 2 channels, surround 5 channels
  • Dropping audio channels e.g., mono consist of 1 channel, stereo 2 channels, surround 5 channels
  • AAC-BSAC Advanced Audio Coding - Bit-Sliced Arithmetic Coding
  • the AAC-BSAC supports enhancement layers of around 1 Kbit/s/channel or smaller for audio signals.
  • bit-slicing scheme is applied to the quantized spectral data.
  • the quantized spectral values are grouped into frequency bands, each of these groups containing the quantized spectral values in their binary representation.
  • the bits of the group are processed in slices according to their significance and spectral content.
  • MSB most significant bits
  • scalability can be achieved in a two-dimensional space. Quality, corresponding to a certain signal bandwidth, can be enhanced by transmitting more LSBs, or the bandwidth of the signal can be extended by providing more bit-slices to the receiver. Moreover, a third dimension of scalability is available by adapting the number of channels available for decoding. For example, a surround audio (5 channels) could be scaled down to stereo (2 channels) which, on the other hand, can be scaled to mono (1 channels) if, e.g., transport conditions make it necessary.
  • FIG. 2 A general example of an audio transmission system using multi-channel (i.e. at least two input channels) coding and decoding is schematically illustrated in Fig. 2.
  • the overall system basically comprises a multi-channel audio encoder 100 and a transmission module 10 on the transmitting side, and a receiving module 20 and a multi-channel audio decoder 200 on the receiving side.
  • the simplest way of stereophonic or multi-channel coding of audio signals is to encode the signals of the different channels separately as individual and independent signals, as illustrated in Fig. 3.
  • Another basic way used in stereo FM radio transmission and which ensures compatibility with legacy mono radio receivers is to transmit a sum signal (mono) and a difference signal (side) of the two involved channels.
  • M/S stereo coding is similar to the described procedure in stereo FM radio, in a sense that it encodes and transmits the sum and difference signals of the channel sub-bands and thereby exploits redundancy between the channel sub-bands.
  • the structure and operation of a coder based on M/S stereo coding is described, e.g., in U. S patent No. 5285498 by J. D. Johnston.
  • Intensity stereo on the other hand is able to make use of stereo irrelevancy. It transmits the joint intensity of the channels (of the different sub-bands) along with some location information indicating how the intensity is distributed among the channels. Intensity stereo does only provide spectral magnitude information of the channels, while phase information is not conveyed. For this reason and since temporal inter-channel information (more specifically the inter-channel time difference) is of major psycho-acoustical relevancy particularly at lower frequencies, intensity stereo can only be used at high frequencies above e.g. 2 kHz.
  • An intensity stereo coding method is described, e.g., in European Patent 0497413 by R. Veldhuis et al.
  • a recently developed stereo coding method is described, e.g., in a conference paper with title 'Binaural cue coding applied to stereo and multi-channel audio compression', 112th AES convention, May 2002, Kunststoff (Germany) by C. Faller et al.
  • This method is a parametric multi-channel audio coding method.
  • the basic principle of such parametric techniques is that at the encoding side the input signals from the N channels d , c2 cN are combined to one mono signal m.
  • the mono signal is audio encoded using any conventional monophonic audio codec.
  • parameters are derived from the channel signals, which describe the multi-channel image.
  • the parameters are encoded and transmitted to the decoder, along with the audio bit stream.
  • the decoder first decodes the mono signal m' and then regenerates the channel signals c1 ⁇ c2 ⁇ ..., cN', based on the parametric description of the multi-channel image.
  • the principle of the binaural cue coding (BCC[2]) method is that it transmits the encoded mono signal and so-called BCC parameters.
  • the BCC parameters comprise coded inter-channel level differences and inter-channel time differences for sub-bands of the original multi-channel input signal.
  • the decoder regenerates the different channel signals by applying sub-band-wise level and phase adjustments of the mono signal based on the BCC parameters.
  • the advantage over e.g. M/S or intensity stereo is that stereo information comprising temporal inter-channel information is transmitted at much lower bit rates.
  • a stereo codec comprising a down-mixing module 120, a core mono codec 130, 230, a bitstream multiplexer/demultiplexer 150, 250 and a parametric stereo side information encoder/decoder 140, 240.
  • the down- mixing transforms the multi-channel (in this case stereo) signal into a mono signal.
  • the objective of the parametric stereo codec is to reproduce a stereo signal at the decoder given the reconstructed mono signal and additional stereo parameters.
  • MPEG Surround [4][5] divides the audio coding into two partitions: one predictive/parametric part called the Dry component and a non- predictable/diffuse part called the Wet component.
  • the Dry component is obtained using channel prediction from a down-mix signal which has been encoded and decoded separately.
  • the Wet component may be either one of the following three: a synthesized diffuse sound signal generated from the prediction and decorrelating filters, a gain adjusted version of the predicted part or simply by the encoded prediction residual. Summary
  • an audio encoding method based on an overall encoding procedure operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels.
  • a first encoding process is performed for encoding a first signal representation, including a down-mix signal, of the set of audio input channels.
  • Local synthesis is performed in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process.
  • a second encoding process is performed for encoding a second representation of the set of audio input channels, using at least the locally decoded down-mix signal as input.
  • Input channel energies of the audio input channels are estimated, and at least one energy representation of the audio input channels is generated based on the estimated input channel energies of the audio input channels.
  • the generated energy representation(s) is/are then encoded.
  • Residual error signals from at least one of the encoding processes, including at least the second encoding process, are generated, and residual encoding of the residual error signals is performed in a third encoding process.
  • the audio encoder device comprises a first encoder for encoding a first representation, including a down- mix signal, of the set of audio input channels in a first encoding process, a local synthesizer for performing local synthesis in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process, and a second encoder for encoding a second representation of the set of audio input channels in a second encoding process, using at least the locally decoded down-mix signal as input.
  • the audio encoder device further comprises an energy estimator for estimating input channel energies of the audio input channels, an energy representation generator for generating at least one energy representation of the audio input channels based on the estimated input channel energies of the audio input channels, and an energy representation encoder for encoding the energy representation(s).
  • the audio encoder device also comprises a residual generator for generating residual error signals from at least one of the encoding processes, including at least the second encoding process, and a residual encoder for performing residual encoding of the residual error signals in a third encoding process.
  • an audio decoding method based on an overall decoding procedure operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels.
  • a first decoding process is performed to produce at least one first decoded channel representation including a decoded down-mix signal based on a first part of the incoming bit stream.
  • a second decoding process is performed to produce at least one second decoded channel representation based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of audio input channels.
  • Input channel energies of audio input channels are estimated based on the estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of audio input channels.
  • Residual decoding is performed in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals.
  • the residual error signals and decoded channel representations from at least one of the first and second decoding processes, including at least the second decoding process, are then combined, and channel energy compensation is performed at least partly based on the estimated input channel energies for generating the multichannel audio signal.
  • the audio decoder device operates on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels.
  • the audio decoder device comprises a first decoder for producing at least one first decoded channel representation including a decoded down-mix signal based on a first part of the incoming bit stream, and a second decoder for producing at least one second decoded channel representation based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of audio input channels.
  • the audio decoder device further comprises an estimator for estimating input channel energies of audio input channels based on estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of audio input channels.
  • the audio decoder device also comprises a residual decoder for performing residual decoding in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals.
  • the audio decoder device also includes means for combining the residual error signals and decoded channel representations from at least one of the first and second decoding processes, including at least the second decoding process, and for performing channel energy compensation at least partly based on the estimated input channel energies for generating the multichannel audio signal.
  • Fig. 1 illustrates an example of a dedicated network entity for media adaptation.
  • Fig. 2 is a schematic block diagram illustrating a general example of an audio transmission system using multi-channel coding and decoding.
  • Fig. 3 is a schematic diagram illustrating how signals of different channels are encoded separately as individual and independent signals.
  • Fig. 4 is a schematic block diagram illustrating the basic principles of parametric stereo coding.
  • Fig. 5 is a schematic block diagram of a general stereo coder using a parametric prediction and a prediction/parametric residual encoding scheme.
  • Fig. 6 is a scatter plot illustrating the dependencies between channel level difference (CLD) and channel level sums (CLS).
  • Fig. 7 illustrates an example of the encoder operation of the present invention in the form of a flowchart. The overview is valid for embodiments A, B and C.
  • Fig. 8 is a flowchart that describes an example of the stereo synthesis chain in the decoder for embodiment A.
  • Fig. 9A is a schematic block diagram describing an example of the operation of the encoder and decoder for embodiment A.
  • Fig. 9B illustrates an example of the operation of the encoder and decoder which is valid for embodiment B.
  • Fig. 9C illustrates an example of the operation of the encoder and decoder which is valid for embodiment C.
  • Fig. 10 illustrates an example of the decoder stereo synthesis chain valid for embodiments B and C.
  • Fig. 11 is a plot that shows how the channel prediction factors (panning factors) varies with respect to the normalized cross-correlation coefficient .
  • Fig. 12 shows the result from an AB test evaluation of the proposed invention in the form of a histogram of the votes.
  • Fig. 13 illustrates an example of the overall encoder operation for a multichannel encoder in the form of a flowchart.
  • Fig. 14 shows a possible multichannel embodiment of the encoder and decoder processes, where the energy measurement on received signals is performed before the multichannel prediction.
  • Fig. 15 is a flowchart which illustrates an example of the overall decoder operation when the energies of the decoded signal components are estimated before the multichannel prediction.
  • Fig. 16 shows a possible multichannel embodiment of the encoder and decoder processes, where the energy measurement of received signals are performed after the multichannel prediction.
  • Fig. 17 is a flowchart which illustrates an example of the overall decoder operation when the energies of the decoded signal components are estimated after the multichannel prediction.
  • Fig. 18 is a schematic flow diagram illustrating an example of a method for audio encoding.
  • Fig. 19 is a schematic flow diagram illustrating an example of a method for audio decoding.
  • Fig. 20 is a schematic block diagram illustrating an example of an audio encoder device.
  • Fig. 21 is a schematic block diagram illustrating an example of an audio decoder device. Detailed description
  • the invention generally relates to multi-channel (i.e. at least two channels) encoding/decoding techniques in audio applications, and particularly to stereo encoding/decoding in audio transmission systems and/or for audio storage.
  • audio applications include phone conference systems, stereophonic audio transmission in mobile communication systems, various systems for supplying audio services, and multi-channel home cinema systems.
  • the invention may for example be particularly applicable in future standards such as ITU-T WP3/SG16/Q23 SWB/stereo extension for G.729.1 and G.718, but is of course not limited to these standards.
  • a stereo codec for example, the stereo encoding and decoding is normally performed in multiple stages.
  • An overview of the process is depicted in Fig. 5.
  • a down-mix mono signal M is formed from the left and right channels L, R.
  • the mono signal is fed to a mono encoder from which a local synthesis M is extracted.
  • a parametric stereo encoder produces a first approximation to the input channels [L Rf .
  • the prediction residual is calculated and encoded to provide further enhancement.
  • the down-mix is a process of reducing the number of input channels p to a smaller number of down-mix channels q.
  • the down-mix can be any linear or non-linear combination of the input channels, performed in temporal domain or in frequency domain.
  • the down-mix can be adapted to the signal properties.
  • the stereo encoding and decoding is assumed to be done on a frequency band or a group of transform coefficients. This assumes that the processing of the channels is done in frequency bands.
  • An arbitrary down-mix with frequency dependent coefficients can be written as:
  • index b represents the current band and k indexes the samples within that band.
  • more elaborate down-mixing schemes may be used with adaptive and time variant weighting coefficients oc b and ⁇ b .
  • the stereo encoder uses the locally decoded mono signal to produce a stereo signal.
  • the two channels of a stereo signal are often very alike, making it useful to apply prediction techniques in stereo coding. Since the decoded mono channel M will be available at the decoder, the objective of the prediction is to reconstruct the left and right channel pair from this signal together with the transmitted quantized stereo parameters ⁇ .
  • the optimal prediction is obtained by minimizing the error vector [ ⁇ L ⁇ R ] ⁇ .
  • This can be solved in time domain by using a time varying FIR-filter:
  • H L (b,k) and H R (b,k) are the frequency responses of the filters h L and h R for coefficient k of the frequency band b
  • L b (k) , R b (k) and M b (k) are the transformed counterparts of the time signals /( «) , r( ⁇ ) and m(n) .
  • frequency domain processing gives explicit control over the phase, which is relevant to stereo perception [2].
  • phase information is highly relevant but can be discarded in the high frequencies. It can also accommodate a sub-band partitioning that gives a frequency resolution which is perceptually relevant.
  • the drawbacks of frequency domain processing are the complexity and delay requirements for the time/frequency transformations. In cases where these parameters are critical, a time domain approach is desirable.
  • the top layers of the codec are SNR enhancement layers in MDCT domain.
  • the delay requirements for the MDCT are already accounted for in the lower layers and the part of the processing can be reused. For this reason, the MDCT domain is selected for the stereo processing.
  • it has some drawbacks in stereo signal processing since it does not give explicit phase control.
  • the time aliasing property of MDCT may give unexpected results since adjacent frames are inherently dependent.
  • it still gives good flexibility for frequency dependent bit allocation.
  • phase representation a combination of MDCT and MDST could be used. The additional MDST signal representation would however increase the total codec bitrate and processing load. In some cases the MDST can be approximated from the MDCT by using MDCT spectra from multiple frames.
  • the frequency spectrum is preferably divided into processing bands.
  • the processing bands are selected to match the critical bandwidths of human auditory perception. Since the available bitrate is low the selected bands are fewer and wider, but the bandwidths are still proportional to the critical bands. Denoting the band b , the prediction can be written:
  • k denotes the index of the MDCT coefficient in the band b
  • m denotes the time domain frame index.
  • [l b ' R b ' ⁇ represent the prediction obtained with unquantized parameters w b (m) .
  • E[.] denotes the averaging operator and is defined as an example for an arbitrary time frequency variable as an averaging over a predefined time frequency region. For example:
  • each frequency band b is represented with the MDCT bins of the set Band ⁇ ) which has the size BW(b) .
  • the frequency bands may also be overlapping.
  • the use of the coded mono signal M in the derivation of the prediction parameters includes the coding error in the calculation. Although sensible from an MMSE perspective, this may cause instability in the stereo image that is perceptually annoying. For this reason, the prediction parameters are based on the unprocessed mono signal, excluding the mono error from the prediction.
  • the right channel predictor coefficient can be written
  • E[L b (m)L b (m)] and E[R b (m)R b (m)] corresponds to the energies of the left and right channels respectively and E[L b (m)R b (mj)] represents the cross-correlation in band b . Further, the sum of the predictor coefficients can be derived
  • the spatial parameters are preferably encoded with a variable bit rate scheme.
  • the parameter bitrate can go down to a minimum and the saved bits can be used in parts of the codec, e.g. SNR enhancements.
  • the difference between the predicted stereo channels and the input channels will form a prediction residual.
  • the residual signal contains the parts of the input channels which are not correlated with the mono down-mix channel and hence could not be modeled with prediction. Further, the prediction residual depends on the precision of the predictor function since a lower predictor resolution will likely give a larger error. Finally, since the prediction is based on the coded mono down-mix signal, the imperfections of the mono coder will also add to the residual error. The components of the residual error signal show correlation and it is beneficial to exploit this correlation when coding the error, as described in the international patent application PCT/SE2008/000272, which is incorporated herein.
  • the prediction residual often represents the diffuse sound field which cannot be predicted. From a perceptual perspective the inter channel correlation (ICC) [2][3][4] is important. This property can be simulated using the decoded down-mix signal or predicted/upmixed signal together with a system of decorrelating filters. The principles of this invention are applicable to any representation of the prediction residual.
  • ICC inter channel correlation
  • the inventors have made a thorough analysis of the state of the art of audio codecs to gain some useful insights in the function and performance of such codecs.
  • the signals will normally be composed of different components corresponding to the encoder stages.
  • the quality of the decoded components is likely to vary with time due to limited bitrates and changing spatial properties but also the transmission conditions. If the resources are too scarce to represent a signal we can observe an energy loss, which will yield an unstable stereo image when it varies over time.
  • the downmix procedure used in for example MPEG PS [3] compensates for energy loss in the downmix due to channel cancellation, but does not give explicit control over the synthesized channel energies nor the prediction factors.
  • the approach in MPEG Surround [4][5] for example handles the presence of a prediction residual (Wet component) in combination with a parametric part (Dry component).
  • the Wet component may be either 1) the gain adjusted parametric part, 2) the encoded prediction residual or 3) the parametric part passed through decorrelation filters.
  • the solution in 3) can be seen as a parametric representation of the prediction residual.
  • the system does not allow the three to coexist with varying proportion and hence does not offer built-in control of synthesis channel energies in this context.
  • Fig. 18 is a schematic flow diagram illustrating an example of a method for audio encoding.
  • the exemplary audio encoding method is based on an overall encoding procedure operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels.
  • a first encoding process is performed for encoding a first signal representation, including a down-mix signal, of said set of audio input channels.
  • local synthesis is performed in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process.
  • a second encoding process is performed for encoding a second representation of the considered set of audio input channels, using at least the locally decoded down-mix signal as input.
  • input channel energies of the audio input channels are estimated.
  • at least one energy representation of the audio input channels is generated based on the estimated input channel energies of said audio input channels.
  • the generated energy representation(s) is/are encoded.
  • residual error signals from at least one of said encoding processes, including at least the second encoding process are generated.
  • residual encoding of the residual error signals is performed in a third encoding process. In this way, an effective overall encoding of the audio input channels is obtained.
  • the energy representation(s) of the audio input channels enables matching of the energies of output channels at the decoding side with the estimated input channel energies.
  • the output channels are matched with the input channels both in terms of energy and quality.
  • the steps of generating at least one energy representation and encoding the energy representation (s) are performed in the second encoding process, as will be exemplified in greater detail later on.
  • the overall encoding procedure is executed for each of a relatively large number of audio frames. It should however be understood that parts of the overall encoding procedure, such as the estimation and encoding (through a suitable energy representation) of the audio input channel energies, may be performed for a selectable sub-set of frames, and in one or more selectable frequency bands. In effect, this means that, for example, the steps of generating at least one energy representation and encoding the energy representation (s) may be performed for each of a number of frames in at least one frequency band.
  • the first encoding process is a down-mix encoding process
  • the second encoding process is based on channel prediction to generate one or more predicted channels
  • the residual error signals thus includes residual prediction error signals.
  • Channel prediction • based on quantized channel prediction parameters derived from quantized channel energy level differences.
  • Channel prediction • based on quantized channel prediction parameters derived from quantized channel energy level differences and quantized energy- normalized input channel cross-correlation parameters.
  • Fig. 19 is a schematic flow diagram illustrating an example of a method for audio decoding.
  • the exemplary audio decoding method is based on an overall decoding procedure operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels.
  • a first decoding process is performed to produce at least one first decoded channel representation including a decoded down-mix signal based on a first part of said incoming bit stream.
  • a second decoding process is performed to produce at least one second decoded channel representation based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of audio input channels.
  • step S13 input channel energies of audio input channels are estimated based on estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of audio input channels.
  • step S14 residual decoding is performed in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals.
  • step S15 the residual error signals and decoded channel representations from at least one of the first and second decoding processes, including at least the second decoding process, are combined, and channel energy compensation is performed at least partly based on the estimated input channel energies for generating the multi- channel audio signal.
  • the channel energy compensation may be performed to match the energies of output channels of the multi-channel audio signal with the estimated input channel energies.
  • the output channels of the multi-channel audio signal are matched with the corresponding input channels at the encoding side both in terms of energy and quality, wherein higher quality signals may be represented with a larger proportion than lower quality signals to improve the overall quality of the output channels.
  • the channel energy compensation is integrated into the second decoding process when producing one or more second decoded channel representations.
  • the channel energy compensation is performed after combining the residual error signals and decoded channel representations.
  • residual error signals and decoded channel representations from at least one of the first and second decoding processes are combined into a multi-channel synthesis and then energies of the combined multi-channel synthesis are estimated.
  • the channel energy compensation is performed based on the estimated energies of the combined multi-channel synthesis and the estimated input channel energies.
  • the second decoding process to produce at least one second decoded channel representation includes synthesizing predicted channels, and the residual decoding includes generating residual prediction error signals.
  • the second decoding process to produce at least one second decoded channel representation includes deriving one or more one energy representations of the audio input channels from the second part of the incoming bit stream, estimating channel prediction parameters at least partly based on the energy representation(s), and then synthesizing predicted channels based on the decoded down-mix signal and the estimated channel prediction parameters.
  • Estimating input channel energies • based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the normalized energy compensation parameters.
  • Combining • combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis.
  • Fig. 20 is a schematic block diagram illustrating an example of an audio encoder device.
  • the audio encoder device 100 is configured for operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels.
  • the basic encoder device 100 includes a first encoder 130, a second encoder 140, energy estimator 142, an energy representation generator 144 and an energy representation encoder 146, a residual generator 155 and a residual encoder 160.
  • the finally encoded parameters are normally collected by a multiplexer 150 for transfer to the decoding side.
  • the first encoder 130 is configured for receiving and encoding a first representation, including a down-mix signal, of audio input channels in a first encoding process.
  • a down-mix unit 120 may be used for down-mixing a suitable set of the input channels into a down-mix signal.
  • the down-mix-unit 120 may be regarded as an integral part of the basic encoder device 100, or alternatively seen as an "external" support unit.
  • a local synthesizer 132 is arranged for performing local synthesis in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process.
  • the local synthesizer 132 is preferably integrated in the first encoder, but may alternatively be provided as a separate decoder implemented on the encoding side in connection with the first encoder.
  • the second encoder 140 is configured for receiving and encoding a second representation of the considered audio input channels in a second encoding process, using at least the locally decoded down-mix signal as input.
  • the energy estimator 142 is configured for estimating input channel energies of the considered audio input channels
  • the energy representation generator 144 is configured for generating at least one energy representation of the audio input channels based on the estimated input channel energies of the audio input channels.
  • the energy representation encoder 146 is configured for encoding the energy representation(s). In this way, the input channel energies may be estimated and encoded on the encoding side.
  • the energy estimator 142 may be implemented as an integrated part of the second encoder 140, may also be arranged as a dedicated unit outside the second encoder.
  • the energy representation generator 144 and the energy representation encoder 146 are conveniently implemented in the second encoder 140, as will be exemplified in more detail later on. In other embodiments, the energy representation processing may be provided outside the second encoder.
  • the residual generator 155 is configured for generating residual error signals from at least one of the encoding processes, including at least the second encoding process, and the residual encoder 160 is configured for performing residual encoding of the residual error signals in a third encoding process.
  • the energy representation(s) generated by the energy representation generator 144, and subsequently encoded, enables matching of the energies of output channels at the decoding side with the estimated input channel energies.
  • the energy representation (s) enables matching of the output channels with the input channels both in terms of energy and quality.
  • the energy representation generator 144 and the energy representation encoder 146 are preferably configured to generate and encode the energy representation(s) for each of a number of frames in at least one frequency band.
  • the energy estimator 142 may be configured for continuously estimating the input channel energies, or alternatively only for a selected set of frames and/or frequency bands adapted to the activities of the energy representation generator 144 and encoder 146.
  • the first encoder 130 is a down-mix encoder
  • the second encoder 140 is a parametric encoder configured to operate based on channel prediction for generating one or more predicted channels
  • the residual generator 155 is configured for generating residual prediction error signals.
  • the second encoder 140 is preferably configured for jointly representing and encoding estimated input channel energies together with channel prediction parameters.
  • the energy representation generator 144 includes a determiner for determining channel energy level differences, a determiner for determining channel energy level sums, and a determiner for determining so- called delta energy measures based on the channel energy level sums and energy of the locally decoded down-mix signal from the local synthesis in connection with the first encoding process.
  • the energy representation encoder 146 includes a quantizer for quantizing the channel energy level differences, and a quantizer for quantizing the delta energy measures. It may for example be beneficial for the second encoder 140 to perform channel prediction based on unquantized channel prediction parameters.
  • the energy representation generator 144 includes a determiner for determining channel energy level differences, a determiner for determining channel energy level sums, a determiner for determining delta energy measures based on the channel energy level sums and energy of the locally decoded down-mix signal from the local synthesis in connection with the first encoding process, and a determiner for determining so-called normalized energy compensation parameters based on the delta energy measures and energies of the predicted channels normalized by energy of the locally decoded down-mix signal.
  • the energy representation encoder 146 includes a quantizer for quantizing the channel energy level differences, and a quantizer for quantizing the normalized energy compensation parameters.
  • the second encoder 140 may be configured to perform channel prediction based on quantized channel prediction parameters derived from quantized channel energy level differences.
  • the energy representation generator 144 includes a determiner for determining channel energy level differences, and a determiner for determining energy-normalized input channel cross-correlation parameters.
  • the energy representation encoder 146 includes a quantizer for quantizing the channel energy level differences, and a quantizer for quantizing the energy- normalized input channel cross-correlation parameters.
  • the second encoder 140 may be configured to perform channel prediction based on quantized channel prediction parameters derived from quantized channel energy level differences and quantized energy-normalized input channel cross-correlation parameters.
  • Fig. 21 is a schematic block diagram illustrating an example of an audio decoder device.
  • the audio decoder device 200 is configured for operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels.
  • the incoming bitstream is normally received from the encoding side by a bitstream demultiplexer 250, which divides the incoming bitstream into relevant sub-sets or parts of the overall incoming bitstream.
  • the basic audio decoder device 200 comprises a first decoder 230, a second decoder 240, and input channel energy estimator 242, a residual decoder 260, and means 270 for combining and channel energy compensation.
  • the first decoder 230 is configured for producing one or more decoded channel representations including a decoded down-mix signal based on a first part of the incoming bit stream.
  • the second decoder 240 is configured for producing one or more second decoded channel representations based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of the audio input channels.
  • the input channel energy estimator 242 is configured for estimating input channel energies of audio input channels based on estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of the audio input channels.
  • the residual decoder 260 is configured for performing residual decoding in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals.
  • the combining and channel energy compensation means 270 is configured for combining the residual error signals and decoded channel representations from at least one of the first and second decoders/decoding processes, including at least the second decoder/decoding process, and for performing channel energy compensation at least partly based on the estimated input channel energies in order to generate the multi-channel audio signal.
  • the means 270 for combining and performing channel energy compensation may be configured to match the energies of output channels of the multi-channel audio signal with the estimated input channel energies.
  • the means 270 for combining and performing channel energy compensation is configured to match the output channels with the corresponding input channels at the encoding side both in terms of energy and quality, wherein higher quality signals are represented with a larger proportion than lower quality signals to improve the overall quality of the output channels.
  • the overall structure for combining and channel energy compensation can be realized in several different ways.
  • the channel energy compensation may be integrated into the second decoder.
  • the second decoder 240 is preferably configured to operate based on the energy of the decoded down-mix signal and the energies of the residual error signals, implying that the audio decoder device 200 also comprises means for estimating energy of the decoded down- mix signal and energies of the residual error signals.
  • the decoder device includes a combiner for combining the residual error signals and the relevant decoded channel representations into a combined multi-channel synthesis, and a channel energy compensator for applying channel energy compensation on the combined multi-channel synthesis to generate the multi-channel audio signal.
  • the audio decoder device preferably includes an estimator for estimating energies of the combined multi-channel synthesis, and the channel energy compensator is configured for applying channel energy compensation based on the estimated energies of the combined multi-channel synthesis and the estimated input channel energies.
  • the first decoder 230 is a down-mix decoder
  • the second decoder 240 is a parametric decoder configured for synthesizing predicted channels
  • the residual decoder 260 is configured for generating residual prediction error signals.
  • the second decoder 240 may include a deriver 241 (or may otherwise be configured) for deriving the energy representation(s) of the audio input channels from the second part of the incoming bit stream, an estimator for estimating channel prediction parameters at least partly based on the energy representation(s), and a synthesizer for synthesizing predicted channels based on the decoded down- mix signal and the estimated channel prediction parameters.
  • Example A For the exemplary context of down-mix decoding combined with prediction- based decoding and residual decoding, three different exemplary realizations will be summarized below. Further details will be given later on. Example A.
  • the deriver 241 is configured for deriving channel energy level differences and delta energy measures from the second part of the incoming bit stream.
  • the estimator 242 for estimating input channel energies is configured for estimating input channel energies based on estimated energy of the decoded down-mix signal, and the channel energy level differences and delta energy measures.
  • the estimator for estimating channel prediction parameters is preferably configured for estimating channel prediction parameters based on estimated input channel energies, estimated energy of the decoded down-mix signal, and estimated energies of the residual error signals.
  • the deriver 241 is configured for deriving channel energy level differences and normalized energy compensation parameters from the second part of said incoming bit stream.
  • the estimator 242 for estimating input channel energies is configured for estimating input channel energies based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the normalized energy compensation parameters.
  • the estimator for estimating channel prediction parameters is configured for estimating channel prediction parameters based on the channel energy level differences, and the synthesizer for synthesizing predicted channels is configured for synthesizing predicted channels based on the decoded down- mix signal and the estimated channel prediction parameters.
  • the means 270 for combining and for performing channel energy compensation includes a combiner for combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis, and a channel energy compensator.
  • the channel energy compensator includes an estimator for estimating energies of the combined multi-channel synthesis, a determiner for determining an energy correction factor based on estimated input channel energies and estimated energies of the combined multi-channel synthesis, and an energy corrector for applying the energy correction factor to the combined multi-channel synthesis to generate the multi-channel audio signal.
  • the deriver 241 is configured for deriving channel energy level differences and energy-normalized input channel cross-correlation parameters from the second part of the incoming bit stream.
  • the estimator 242 for estimating input channel energies is configured for estimating input channel energies based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the energy-normalized input channel cross-correlation parameters.
  • the estimator for estimating channel prediction parameters is preferably configured for estimating channel prediction parameters based on the channel energy level differences and the energy- normalized input channel cross-correlation parameters.
  • the synthesizer for synthesizing predicted channels is configured for synthesizing predicted channels based on the decoded down-mix signal and the estimated channel prediction parameters.
  • the means 270 for combining and for performing channel energy compensation includes a combiner for combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis, and a channel energy compensator.
  • the channel energy compensator includes an estimator for estimating energies of the combined multi-channel synthesis, a determiner for determining an energy correction factor based on estimated input channel energies and estimated energies of the combined multi-channel synthesis, an energy corrector for applying the energy correction factor to the combined multi-channel synthesis to generate the multi-channel audio signal.
  • the invention aims to solve at least one, and preferably both of the following two problems: to obtain optimal channel prediction and maintain explicit control over the output channel energies.
  • the components of the signal may show individual variations over time in energy and quality, such that a simple adding of the signal components would give an unstable impression in terms of energy and overall quality.
  • the energy and quality variations can have a variety of reasons out of which a few can be mentioned here:
  • a signal component may be lost or degraded due to transmission conditions.
  • Components of the signal could be deliberately attenuated in the encoder, knowing that the lost energy will be recovered in the decoder. Such attenuation may be based on for instance perceptual importance.
  • Parts of the signal may be lost due to limitations in the overall encoder to represent them. Due to for instance limited bitrates or modeling capabilities, parts of the signal may fall outside of the scope of the overall encoder.
  • the individual encoder and related decoder processes each represent a subspace which the true input signal is projected onto.
  • the final residual or coding error is orthogonal to the union of the subspaces which represent the overall encoder and decoder.
  • the final residual cannot be represented with these subspaces, but its energy can be estimated and compensated for if we know or can at least estimate the input energies and the energies of the received subspace components.
  • the invention generally relates to an overall encoding procedure and associated decoding procedure.
  • the encoding procedure involves at least two signal encoding processes operating on signal representations of a set of audio input channels. It also involves a dedicated process to estimate the energies of the input channels.
  • a basic idea of the present invention is to use local synthesis in connection with a first encoding process to generate a locally decoded signal, including a representation of the encoding error of the first encoding process, and apply this locally decoded signal as input to a second encoding process.
  • the sequence of encoding processes can be seen as refinement steps of the overall encoding process, or as capturing different properties of the signal.
  • the first encoding process may be a main encoding process such as a mono encoding process or more generally a down-mix encoder
  • the second encoding process may be an auxiliary encoding process such as a stereo encoding process or a general parametric encoding process.
  • the overall encoding procedure operates on at least two (multiple) audio input channels, including stereophonic encoding as well as more complex multi- channel encoding.
  • Each encoding process is associated with a decoding process.
  • the decoded signals from each encoding process are preferably combined such that the output channels are close to the input channels both in terms of energy and quality.
  • the combination step also adapts to the possible loss of one or more signal representation in part or in whole, such that the energy and quality is optimized with the signals at hand in the decoder.
  • the qualities of the signal components may also be considered so that higher quality signals are represented with a larger proportion than the low quality signals, and thereby improving the overall quality of the output channels.
  • the invention relates to an encoder and an associated decoder.
  • the overall encoder basically comprises at least two encoders for encoding different representations of input channels. Local synthesis in connection with a first encoder generates a locally decoded signal, and this locally decoded signal is applied as input to a second encoder.
  • the overall encoder also generates energy representations of the input channels.
  • the overall decoder includes decoding procedures associated with each encoding procedure in the encoder. It further includes a combination stage where the decoded components are combined with stable energy and quality, facing possible partial or total loss of one or more of the decoded signals.
  • the invention aims to solve at least one, and preferably both of the following two problems: to obtain optimal channel prediction and maintain explicit control over the output channel energies.
  • the components of the signal may show individual variations over time in energy and quality, such that a simple adding of the signal components would give an unstable impression in terms of energy and overall quality.
  • a solution to these and other problems may for example be implemented by means of a joint representation and encoding of both the energies and prediction parameters in a way that is robust to the possible energy and quality variations of the different components.
  • step S21 the encoder performs the down-mix on the input signals and feeds it to the mono encoder, extracting a locally decoded downmix signal in step S22. It further estimates and encodes the input channel energies in step S23.
  • step S24 the channel prediction parameters are derived in step S24.
  • step S25 a local synthesis of the predicted/parametric stereo is created and subtracted from the input signals, forming a prediction/parametric residual which is encoded with suitable methods in step S26. Further iterative refinement steps may be taken if more encoding stages are possible in step S27.
  • step S28 This is executed in step S28 by performing a local synthesis and subtracting the encoded prediction residual from the prediction residual from the previous iteration and encoding the new residual of the current iteration.
  • the example encoder process depicted in Fig. 7 constitutes an overview which is valid for all presented embodiments A, B and C. It should however be noted that the underlying details of the steps outlined in Fig. 7 are different for each presented embodiment, as will be further explained.
  • An example decoder reconstructs the decoded downmix signal which is identical to the locally decoded downmix signal in the encoder.
  • the input channel energies are estimated using the decoded down-mix signal together with encoded energy representation.
  • the channel prediction parameters are derived.
  • the decoder further analyses the energies of the synthesized signals and adjusts the energies to the estimated input channel energies. This step may also be incorporated in the channel prediction step as we shall see in embodiment A. Further, the process of energy adjustment may also consider the qualities of signal components, such that lower quality components may be suppressed in favour of higher quality components.
  • the invention may be regarded as a prediction based upmix which allows multiple components per channel, and further has the energy preserving properties of the energy based upmix.
  • upmix which is commonly used in the context of MPEG Surround, will be used synonymously with the expressions “channel prediction” and “parametric multichannel synthesis”.
  • the encoder and decoder operates on a stereo input and output signals respectively.
  • the encoder of Fig. 9A basically includes a down-mixer that creates a mono signal from the stereo input signals, a mono encoder which encodes the down-mix signal and produces a locally decoded down-mix synthesis. Further, it includes a parametric stereo encoder which creates a first representation of the input stereo channels using the locally decoded down- mix signal and also estimates the input channel energies, creates an energy representation and encodes the representation to be used in the decoder. The encoder also creates a stereo prediction residual which is encoded with the residual encoder.
  • 9A includes a mono decoder which creates a decoded down-mix signal corresponding to the locally decoded down-mix signal of the encoder. It also includes a residual decoder which decodes the encoded stereo prediction residual. Finally, it includes an energy measurement unit and a parametric stereo decoder.
  • Fig. 8 explains the decoder operation in the form of a flowchart.
  • the mono decoding takes place, and the residual decoding is done in step
  • Step S33 includes the energy measurement of the residual signal energies.
  • a parametric stereo synthesis with integrated energy compensation is done in step S34 and the joining of the decoded residuals and the parametric stereo synthesis is done in step S35.
  • the energy encoding and decoding and channel prediction of embodiment A are explained in more detail below.
  • the bandwidth normalization will be equal for all energy parameters in one band and can hence be omitted.
  • CLD channel level differences
  • CLS channel level sums
  • the CLDs D b (m) are preferably quantized in log domain using codebooks which consider perceptual measures for CLD sensitivity.
  • the CLSs S b (m) show strong correlation with the energy of the down-mix signal ⁇ 2 Jm) . Since a decoded down-mix signal is available in the stereo decoder, we form a delta energy measure with respect to this signal
  • S and D are dependent variables as illustrated in Fig. 60.
  • D the distribution of S becomes more narrow and different codebooks may be selected depending on the CLD.
  • the CLS will be dominated by one channel and can be set to a constant using zero bits. For example:
  • the channel energies [ ⁇ b L (m) ⁇ b R (m)f can be expressed using the variables
  • the channel prediction parameters W b (m) used in the encoder are not quantized, thereby ensuring that the prediction residual is minimal.
  • the error from the quantization of the prediction parameters is not transferred to the prediction residual.
  • the channel prediction parameters can be estimated from the energies.
  • the full stereo synthesis can be written
  • Table 1 Variable length codebook for coding the signs of the channel predictor coefficients. It exploits the high probability of two positive signs, as well as the fact that not two signs are negative in the same band.
  • the output channel energies are corrected using the channel prediction factors. If the decoded residual signal is close to the true residual, the channel prediction factors will be close to the optimal prediction factors used in the encoder. If the residual coding energy is lower than the true residua! energy due to e.g. low bitrate encoding, the contribution from the parametric stereo is scaled up to compensate for the energy loss. If the residual coding is zero, the algorithm inherently defaults to intensity stereo coding.
  • the encoder and decoder also operates on stereo signals.
  • the encoder of Fig. 9B basically includes a down-mixer that creates a mono signal from the stereo input signals, a mono encoder which encodes the down- mix signal and produces a locally decoded down-mix synthesis. Further, it includes a parametric stereo encoder which creates a first representation of the input stereo channels using the locally decoded down-mix signal and also estimates the input channel energies, creates an energy representation and encodes the representation to be used in the decoder. The encoder also creates a stereo prediction residual which is encoded with the residual encoder.
  • step S41 The mono decoding is done in step S41 , which is followed by a parametric stereo synthesis in step S42 and a stereo residual decoding in step S43.
  • step S44 the residual and parametric stereo synthesis is joined and the energy of this combined synthesis is done in step S45.
  • step S46 includes the energy adjustment of the combined synthesis.
  • Equation (26) can be solved for C b (m) using either the left or the right channel:
  • Equations (26) and (19) offer two expressions for the input channel energies. Taking the right side of the equality and setting them equal we get
  • the denominator w b 2 L (m) + w b 2 R (m) equals the sum of the energies of the predicted channels normalized by the mono energy.
  • this energy representation is equivalent to the first representation and that it only differs in the normalization of the CLS parameters AS b (m) and C b 2 (m) .
  • the CLD is encoded as in embodiment A.
  • the energy compensation parameters, also referred to as normalized energy compensation parameters, Q 2 Ow) is also quantized in log domain just like AS b (m) , but uses a different codebook (in fact just a different log-value offset) due to the scaling difference.
  • the decoder derives the approximated channel energies from the received parameters C b (m) , b b (m) and measured decoded mono energy ⁇ b 2 ⁇ ⁇ m)
  • the channel predictors used in the encoder are derived from the quantized CLDs
  • the same channel predictors are used in the encoder and decoder. This ensures correct matching between predicted channels and residual coding.
  • coded residual ⁇ differs from ⁇ in equation (20) since different predictors were used in the encoder.
  • the final synthesis is produced by applying an energy correction factor that restores the approximated channel energies
  • the third non-limiting example is also a stereo encoder and decoder embodiment.
  • the encoder of Fig. 9C basically includes a down-mixer that creates a mono signal from the stereo input signals, a mono encoder which encodes the down- mix signal and produces a locally decoded down-mix synthesis. Further, it includes a parametric stereo encoder which creates a first representation of the input stereo channels using the locally decoded down-mix signal and also estimates the input channel energies, creates an energy representation and encodes the representation to be used in the decoder.
  • the encoder also creates a stereo prediction residual which is encoded with the residual encoder.
  • 9C includes a mono decoder which creates a decoded down-mix signal corresponding to the locally decoded down-mix signal of the encoder. It also includes a residual decoder which decodes the encoded stereo prediction residual. Further, it includes a parametric stereo decoder and an energy measurement unit which operates on the combined stereo synthesis and an energy correction unit which modifies the combined stereo synthesis to create a final stereo synthesis. From an overview perspective the decoder operation of embodiment C is similar to the decoder of embodiment B, and Fig. 10 gives an accurate description of the decoder steps for both examples. The energy encoding and decoding and channel prediction of embodiment C are explained in more detail below.
  • Equation (35) we see that the energy estimate decreases with increasing p , which means we can start the search at the value given by equation (33) and perform an incremental search if the initial value does not fulfill ⁇ b 2 ⁇ m)l ⁇ l(m) ⁇ ⁇ lhl .. If there is an energy loss in the mono encoding, we might want to search for decreasing p to minimize ⁇ b 2 (m)- ⁇ b 2 (m) , but this may have an undesired effect on the channel prediction parameters. The effect on the channel prediction with varying p will be further discussed later on.
  • the same channel predictors are used in both encoder and decoder.
  • the difference from embodiment B is that the quantized MMSE optimal channel prediction factors are used. Further, as in embodiment B, the energy relations between the decoded residual and predicted channels are preserved.
  • the output channel energy are corrected after joining the predicted and residual coding components just like in embodiment B.
  • the overall description in the decoder flowchart of Fig. 100 is valid also for embodiment C.
  • the presented exemplary embodiments A, B and C give equal accuracy in representing the CLD in the synthesized stereo sound. They also have equivalent behavior in the case of no residual coding, in which case they all default to an intensity stereo algorithm.
  • a main difference lies in which channel prediction parameters are used in the encoder, and how they are derived in the decoder. The preferred embodiment will be different depending on various parameters, e.g. the available bitrate and the complexity of the input signals with regard to coding and spatial information.
  • the optimal unquantized channel predictors are used in the encoder.
  • the channel predictors used in the decoder will be the same if the bitrate is high and the residual coding approaches perfect reconstruction. For intermediate bitrates, only the predicted part of the stereo is scaled to compensate for energy loss in the residual. If the residual coding is noisier than the predicted stereo component due to e.g. low bitrate residual encoding, using a larger proportion of the predicted stereo is a desirable feature.
  • the quantized channel predictors are used in the encoder.
  • the prediction will not be optimal in the MMSE sense, but it guarantees that the scaling of the predicted signal and the coded residual signal is matched. This is important if the coding error of the mono signal is dominant and the residual mainly corrects this error.
  • embodiment C gives a compact representation of both the channel energies and the channel prediction factors.
  • the parameters show dependencies that can be exploited for encoding. If the mono encoding is not conserving the energy of the mono signal, an additional safeguard for energy increases can be added with a predictable impact on the parametric stereo prediction performance.
  • the invention achieves scalability while maintaining channel energy levels which are important for stereo image perception.
  • the residual coding is nil, the system will default to an intensity stereo algorithm.
  • the synthesized output will scale towards perfect reconstruction while maintaining channel energies and stereo image stability.
  • the exemplary method B was tested.
  • the baseline for comparison was using CLD based channel prediction (intensity stereo) in the range 2.2 kHz to 7.0 kHz.
  • the applied method below 2.2 kHz was identical for tested candidates.
  • Fig. 12 shows a histogram of the votes, indicating a preference for the invention.
  • the audio material consisted of 7 audio clips taken from the AMR-WB+ selection test material.
  • the principles of this invention are also applicable to multi-channel scenarios where the input and output channels are more than two.
  • the properties of the down-mix may create dependencies between the channels of the original multichannel signal and the down-mixed signal which can be exploited to make efficient representations of the channel energies and channel predictors.
  • the multichannel down-mix as such can be performed in multiple stages as have been seen in prior art [5]. If pair-wise channel combinations are performed, principles from the stereo embodiments may apply.
  • the down-mixed signal is fed to a first stage encoder which operates on q channels, and a locally decoded down-mix signal ⁇ is extracted from this process.
  • This signal is used in a multichannel prediction or upmix step, which creates a first approximation
  • X X to the input multichannel signal.
  • the approximation is subtracted from the original input signal, forming a multichannel prediction residual or parametric residual.
  • the residual is fed to a second encoding stage. If desired, a locally decoded residual signal can be extracted and subtracted from the original residual signal to create a second stage residual signal.
  • This encoding process can be repeated to provide further refinements converging towards the original input signal, or to capture different properties of the signal.
  • the encoded prediction, energy and residual parameters are transmitted or stored to be used in a decoder. An overview of an example of the encoding process can be seen in Fig. 13.
  • the overall decoder performs a decoding of the down-mixed signal corresponding to the locally decoded down-mixed signal in the encoder.
  • the encoded residual or residuals are decoded.
  • a first stage multichannel prediction or upmix is performed.
  • the multichannel prediction may be different from the multichannel prediction in the encoder.
  • the decoder measures the energies of the received and decoded signals, such as the decoded down- mixed signal, the predicted multichannel signal and residual signal or signals. An energy estimate of the input channel energies is calculated and is used to combine the decoded signal components into a multichannel output signal.
  • the energies may be measured before the prediction stage, allowing the output energy to be controlled jointly with the prediction as illustrated in Fig. 14 and Fig. 15.
  • the energies may also be measured after the signal components have been joined and adjusted in a final stage on the joined components as illustrated in Fig. 16 and Fig. 17.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

The invention relates to the technical field of audio encoding and/or decoding technologies, and thus concerns an overall encoding procedure and associated decoding procedure. The encoding procedure involves at least two signal encoding processes (S1-S3) operating on signal representations of a set of audio input channels, as well as residual encoding (S7-S8). It also involves a dedicated process (S4-S6) to estimate and encode energies of the audio input channels. Each encoding process is associated with a corresponding decoding process. In the overall decoding procedure the decoded signals from each encoding process are preferably combined such that the output channels are close to the input channels in terms of energy and/or quality. Normally, the combination step also adapts to the possible loss of one or more signal representation in part or in whole, such that the energy and quality is optimized with the signals at hand in the decoder. In this way, the overall quality of the output channels is improved.

Description

Energy conservative multi-channel audio coding
Technical field
The present invention relates to an audio encoding method and a corresponding audio decoding method, as well as an audio encoder and a corresponding audio decoder.
Background
The need for offering telecommunication services over packet switched networks has been dramatically increasing and is today stronger than ever. In parallel there is a growing diversity in the media content to be transmitted, including different bandwidths, mono and stereo sound and both speech and music signals. A lot of efforts at diverse standardization bodies are being mobilized to define flexible and efficient solutions for the delivery of mixed content to the users. Noticeably, two major challenges still await solutions. First, the diversity of deployed networking technologies and user-devices imply that the same service offered for different users may have different user- perceived quality due to the different properties of the transport networks. Hence, improving quality mechanisms is necessary to adapt services to the actual transport characteristics. Second, the communication service must accommodate a wide range of media content. Currently, speech and music transmission still belong to different paradigms and there is a gap to be filled for a service that can provide good quality for all types of audio signals.
Today, scalable audiovisual and in general media content codecs are available, in fact one of the early design guidelines of MPEG was scalability from the beginning. However, although these codecs are attractive due to their functionality, they lack the efficiency to operate at low bitrates, which do not really map to the current mass market wireless devices. With the high penetration of wireless communications more sophisticated scalable-codecs are needed. This fact has been already realized and new codecs are to be expected to appear in the near future.
Despite the tremendous efforts being put on adaptive services and scalable codecs, scalable services will not happen unless more attention is given to the transport issues. Therefore, besides efficient codecs appropriate network architecture and transport framework must be considered as an enabling technology to fully utilize scalability in service delivery. Basically, three scenarios can be considered:
• Adaptation at the end-points. That is, if a lower transmission rate must be chosen the sending side is informed and it performs scaling or codec changes.
• Adaptation at intermediate gateways. If a part of the network becomes congested, or has a different service capability, a dedicated network entity as illustrated in Fig. 1 , performs the transcoding of the service. With scalable codec this could be as simple as dropping or truncating media frames.
• Adaptation inside the network. If a router or wireless interface becomes congested adaptation is performed right at the place of the problem by dropping or truncating packets. This is a desirable solution for transient problems like handling of severe traffic bursts or the channel quality variations of wireless links.
Below, an overview of scalable codecs for speech and audio according to the prior art is given. We also give a general background on stereo coding concepts. Scalable audio coding
Non-conversational, streaming/download
In general the current audio research trend is to improve the compression efficiency at low rates (provide good enough stereo quality at bit rates below 32 kbps). Recent low rate audio improvements are the finalization of the
Parametric Stereo (PS) tool development in MPEG, the standardization of a mixed CELP/and transform codec Extended AMR-WB (a.k.a. AMR-WB+) in
3GPP. There is also an ongoing MPEG standardization activity around Spatial
Audio Coding (Surround/5.1 content), where a first reference model (RMO) has been selected [4].
With respect to scalable audio coding, recent standardization efforts in MPEG have resulted in a scalable to lossless extension tool, MPEG4-SLS. MPEG4- SLS provides progressive enhancements to the core AAC/BSAC all the way up to lossless with granularity step down to 0.4 kbps. An Audio Object Type (AOT) for SLS is yet to be defined. Further within MPEG a Call for Information (CfI) has been issued in January 2005 [1] targeting the area of scalable speech and audio coding, in the CfI the key issues addressed are scalability, consistent performance across content types (e.g. speech and music) and encoding quality at low bit rates (< 24kbps). Later, the scalable part was dropped and the work is now targeting a codec running at a variety of bitrates without embedded scalability. Speech coding (conversational mono)
General
In general speech compression the latest standardization efforts is an extension of the 3GPP2/VMR-WB codec to also support operation at a maximum rate of 8.55 kbps. In ITU-T the Multirate G.722.1 audio/video conferencing codec has previously been updated with two new modes providing super wideband (14 kHz audio bandwidth, 32 kHz sampling) capability operating at 24, 32 and 48 kbps. Further standardization efforts were aiming to add an additional mode that would extend the bandwidth to 48 kHz full-band coding. The end result was the new stand-alone codec G.719, which provides low complex full-band coding from 32 to 128 kbps in steps of 16 kbps.
With respect to scalable conversational speech coding the main standardization effort is taking place in ITU-T, (Working Party 3, Study Group 16). There a scalable extension of G.729 was standardized in May 2006, called G.729.1. This extension is scalable from 8 to 32 kbps with 2 kbps granularity steps from 12 kbps. The main target application for G.729.1 is conversational speech over shared and bandwidth limited xDSL-links, i.e. the scaling is likely to take place in a Digital Residential Gateway that passes the VoIP packets through specific controlled Voice channels (Vc's). ITU-T has also recently (Sept. 2008) approved the recommendation for a completely new scalable conversational codec, G.718. The codec comprises a core rate of 8.0 kbps and a maximum rate of 32 kbps., with scaling steps at 12.0, 16.0 and 24.0 kbps. The G.718 core is a WB speech codec inherited from VMR-WB, but also handles NB input signals by upsampling to the core samplerate. Further a joint extension of the G.718 and G.729.1 codecs that will bring super wideband and stereo capabilities (32 kHz sampling/2 channels) is currently under standardization in ITU-T (Working Party 3, Study Group 16, Question 23). The qualification period ended July 2008.
SNR scalability
The principle of SNR scalability is to increase the SNR with increasing number of bits or layers. The two previously mentioned speech codecs G.729.1 and
G.718 have this feature. Typically this is achieved by stepwise re-encoding of the coding residual from the previous layer. The embedded layered structure is attractive since lower bitrates can be decoded by simply discarding the upper layers. However, the embedded layering may not be optimal when considering the higher bitrates and a layered codec usually performs worse than a fixed bitrate codec at the same bitrate. Other codecs that can be mentioned here is the SNR scalable MPEG4-CELP and G.727 (Embedded ADPCM).
Bandwidth scalability
There are also codecs that can increase bandwidth with increasing amount of bits, e.g. G722 (Sub band ADPCM) but also G.729.1 and G.718. G.729.1 operates with a cascaded CELP codec for the bitrates 8 and 12 kbps, but provides WB signals at 14 kbps using a bandwidth extension to fill the range from 4 kHz to 7 kHz. The bandwidth extension typically creates an excitation signal from the lower band by spectral folding or other mappings, which is further gain adjusted and shaped with a spectral envelope to simulate the higher end frequency spectrum. Although the solution might sound good, the extended spectrum does not generally match the input signal in an MSE sense. For codecs that also SNR scalable, the bandwidth extension used at lower rates is typically replaced with coded content in higher layers. This is the case for G.729.1 where the spectrum is gradually replaced with coded spectrum on a subband basis. G.718 exhibits the same feature and uses bandwidth extension from 6.4 kHz to 7.0 kHz for rates 8, 12 and 16 kbps. For the rates 24 and 32 kbps, the bandwidth extension is disabled and replaced with coded spectrum. Also in addition to being SNR-scalable MPEG4-CELP specifies a bandwidth scalable coding system for 8 and 16 kHz sampled input signals.
Audio Scalability
Basically, audio scalability can be achieved by:
• Changing the quantization of the signal, i.e. SNR-like scalability.
• Extending or tightening the bandwidth of the signal.
• Dropping audio channels (e.g., mono consist of 1 channel, stereo 2 channels, surround 5 channels) - (spatial scalability).
Currently available, fine-grained scalable audio codec is the AAC-BSAC (Advanced Audio Coding - Bit-Sliced Arithmetic Coding). It can be used for both audio and speech coding, it also allows for bit-rate scalability in small increments.
It produces a bit-stream, which can even be decoded if certain parts of the stream are missing. There is a minimum requirement on the amount of data that must be available to permit decoding of the stream. This is referred to as base-layer. The remaining set of bits corresponds to quality enhancements, hence their reference as enhancement-layers. The AAC-BSAC supports enhancement layers of around 1 Kbit/s/channel or smaller for audio signals.
"To obtain such fine grain scalability, a bit-slicing scheme is applied to the quantized spectral data. First the quantized spectral values are grouped into frequency bands, each of these groups containing the quantized spectral values in their binary representation. Then the bits of the group are processed in slices according to their significance and spectral content. Thus, first all most significant bits (MSB) of the quantized values in the group are processed and the bits are processed from lower to higher frequencies within a given slice. These bit-slices are then encoded using a binary arithmetic coding scheme to obtain entropy coding with minimal redundancy." [1]
"With an increasing number of enhancement layers utilized by the decoder, providing more least significant bit (LSB) information refines quantized spectral data. At the same time, providing bit-slices of spectral data in higher frequency bands increases the audio bandwidth. In this way, quasi-continuous scalability is achievable." [1]
In other words, scalability can be achieved in a two-dimensional space. Quality, corresponding to a certain signal bandwidth, can be enhanced by transmitting more LSBs, or the bandwidth of the signal can be extended by providing more bit-slices to the receiver. Moreover, a third dimension of scalability is available by adapting the number of channels available for decoding. For example, a surround audio (5 channels) could be scaled down to stereo (2 channels) which, on the other hand, can be scaled to mono (1 channels) if, e.g., transport conditions make it necessary.
Stereo coding or multi-channel coding
A general example of an audio transmission system using multi-channel (i.e. at least two input channels) coding and decoding is schematically illustrated in Fig. 2. The overall system basically comprises a multi-channel audio encoder 100 and a transmission module 10 on the transmitting side, and a receiving module 20 and a multi-channel audio decoder 200 on the receiving side.
The simplest way of stereophonic or multi-channel coding of audio signals is to encode the signals of the different channels separately as individual and independent signals, as illustrated in Fig. 3. However, this means that the redundancy among the plurality of channels is not removed, and that the bit- rate requirement will be proportional to the number of channels.
Another basic way used in stereo FM radio transmission and which ensures compatibility with legacy mono radio receivers is to transmit a sum signal (mono) and a difference signal (side) of the two involved channels.
State-of-the art audio codecs such as MPEG-1/2 Layer III and MPEG-2/4 AAC make use of so-called joint stereo coding. According to this technique, the signals of the different channels are processed jointly rather than separately and individually. The two most commonly used joint stereo coding techniques are known as 'Mid/Side' (M/S) Stereo and intensity stereo coding which usually are applied on sub-bands of the stereo or multi-channel signals to be encoded.
M/S stereo coding is similar to the described procedure in stereo FM radio, in a sense that it encodes and transmits the sum and difference signals of the channel sub-bands and thereby exploits redundancy between the channel sub-bands. The structure and operation of a coder based on M/S stereo coding is described, e.g., in U. S patent No. 5285498 by J. D. Johnston.
Intensity stereo on the other hand is able to make use of stereo irrelevancy. It transmits the joint intensity of the channels (of the different sub-bands) along with some location information indicating how the intensity is distributed among the channels. Intensity stereo does only provide spectral magnitude information of the channels, while phase information is not conveyed. For this reason and since temporal inter-channel information (more specifically the inter-channel time difference) is of major psycho-acoustical relevancy particularly at lower frequencies, intensity stereo can only be used at high frequencies above e.g. 2 kHz. An intensity stereo coding method is described, e.g., in European Patent 0497413 by R. Veldhuis et al. A recently developed stereo coding method is described, e.g., in a conference paper with title 'Binaural cue coding applied to stereo and multi-channel audio compression', 112th AES convention, May 2002, Munich (Germany) by C. Faller et al. This method is a parametric multi-channel audio coding method. The basic principle of such parametric techniques is that at the encoding side the input signals from the N channels d , c2 cN are combined to one mono signal m. The mono signal is audio encoded using any conventional monophonic audio codec. In parallel, parameters are derived from the channel signals, which describe the multi-channel image. The parameters are encoded and transmitted to the decoder, along with the audio bit stream. The decoder first decodes the mono signal m' and then regenerates the channel signals c1\ c2\ ..., cN', based on the parametric description of the multi-channel image.
The principle of the binaural cue coding (BCC[2]) method is that it transmits the encoded mono signal and so-called BCC parameters. The BCC parameters comprise coded inter-channel level differences and inter-channel time differences for sub-bands of the original multi-channel input signal. The decoder regenerates the different channel signals by applying sub-band-wise level and phase adjustments of the mono signal based on the BCC parameters. The advantage over e.g. M/S or intensity stereo is that stereo information comprising temporal inter-channel information is transmitted at much lower bit rates.
Another technique, described in US Patent No. 5,434,948 by CE. Holt et al. uses the same principle of encoding of the mono signal and side information. In this case, side information consists of predictor filters and optionally a residual signal. The predictor filters, estimated by the LMS algorithm, when applied to the mono signal allow the prediction of the multi-channel audio signals. With this technique one is able to reach very low bit rate encoding of multi-channel audio sources, however at the expense of a quality drop. The basic principles of parametric stereo coding are illustrated in Fig. 4, which displays a layout of a stereo codec, comprising a down-mixing module 120, a core mono codec 130, 230, a bitstream multiplexer/demultiplexer 150, 250 and a parametric stereo side information encoder/decoder 140, 240. The down- mixing transforms the multi-channel (in this case stereo) signal into a mono signal. The objective of the parametric stereo codec is to reproduce a stereo signal at the decoder given the reconstructed mono signal and additional stereo parameters.
In International Patent Application, published as WO 2006/091139, a technique for adaptive bit allocation for multi-channel encoding is described. It utilizes at least two encoders, where the second encoder is a multistage encoder. Encoding bits are adaptively allocated among the different stages of the second multi-stage encoder based on multi-channel audio signal characteristics.
A downmixing technique employed in MPEG Parametric Stereo in explained in [3]. Here the potential energy loss from channel cancellation in the downmix procedure is compensated with a scaling factor.
MPEG Surround [4][5] divides the audio coding into two partitions: one predictive/parametric part called the Dry component and a non- predictable/diffuse part called the Wet component. The Dry component is obtained using channel prediction from a down-mix signal which has been encoded and decoded separately. The Wet component may be either one of the following three: a synthesized diffuse sound signal generated from the prediction and decorrelating filters, a gain adjusted version of the predicted part or simply by the encoded prediction residual. Summary
Although many advances have been made in the field of audio codecs, there is still a general demand for improved audio codec technologies.
It is a general object to provide improved audio encoding and/or decoding technologies.
It is a specific object to provide an improved audio encoding method.
It is also a specific object to provide an improved audio decoding method.
It is another specific object to provide an improved audio encoder device.
It is yet another specific object to provide an improved audio decoder device.
These and other objects are met by the invention as defined by the accompanying patent claims.
In a first aspect, there is provided an audio encoding method based on an overall encoding procedure operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels. According to the audio encoding method, a first encoding process is performed for encoding a first signal representation, including a down-mix signal, of the set of audio input channels. Local synthesis is performed in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process. A second encoding process is performed for encoding a second representation of the set of audio input channels, using at least the locally decoded down-mix signal as input. Input channel energies of the audio input channels are estimated, and at least one energy representation of the audio input channels is generated based on the estimated input channel energies of the audio input channels. The generated energy representation(s) is/are then encoded. Residual error signals from at least one of the encoding processes, including at least the second encoding process, are generated, and residual encoding of the residual error signals is performed in a third encoding process.
In this way, an effective overall encoding of the audio input can be achieved with the possibility of matching the output channels with the input channels in terms of energy and/or quality.
There is also provided a corresponding audio encoder device operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels. Basically, the audio encoder device comprises a first encoder for encoding a first representation, including a down- mix signal, of the set of audio input channels in a first encoding process, a local synthesizer for performing local synthesis in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process, and a second encoder for encoding a second representation of the set of audio input channels in a second encoding process, using at least the locally decoded down-mix signal as input. The audio encoder device further comprises an energy estimator for estimating input channel energies of the audio input channels, an energy representation generator for generating at least one energy representation of the audio input channels based on the estimated input channel energies of the audio input channels, and an energy representation encoder for encoding the energy representation(s). The audio encoder device also comprises a residual generator for generating residual error signals from at least one of the encoding processes, including at least the second encoding process, and a residual encoder for performing residual encoding of the residual error signals in a third encoding process. In a second aspect, there is provided an audio decoding method based on an overall decoding procedure operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels. According to the audio decoding method, a first decoding process is performed to produce at least one first decoded channel representation including a decoded down-mix signal based on a first part of the incoming bit stream. A second decoding process is performed to produce at least one second decoded channel representation based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of audio input channels. Input channel energies of audio input channels are estimated based on the estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of audio input channels. Residual decoding is performed in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals. The residual error signals and decoded channel representations from at least one of the first and second decoding processes, including at least the second decoding process, are then combined, and channel energy compensation is performed at least partly based on the estimated input channel energies for generating the multichannel audio signal.
In this way, it is possible to effectively reconstruct a multi-channel audio signal such that output channels are close to the input channels in terms of energy and/or quality.
There is also provided a corresponding audio decoder device operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels. Basically, the audio decoder device comprises a first decoder for producing at least one first decoded channel representation including a decoded down-mix signal based on a first part of the incoming bit stream, and a second decoder for producing at least one second decoded channel representation based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of audio input channels. The audio decoder device further comprises an estimator for estimating input channel energies of audio input channels based on estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of audio input channels. The audio decoder device also comprises a residual decoder for performing residual decoding in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals. The audio decoder device also includes means for combining the residual error signals and decoded channel representations from at least one of the first and second decoding processes, including at least the second decoding process, and for performing channel energy compensation at least partly based on the estimated input channel energies for generating the multichannel audio signal.
Other advantages offered by the invention will be appreciated when reading the below description of embodiments of the invention.
Brief description of the drawings
The invention, together with further objects and advantages thereof, will be best understood by reference to the following description taken together with the accompanying drawings, in which:
Fig. 1 illustrates an example of a dedicated network entity for media adaptation. Fig. 2 is a schematic block diagram illustrating a general example of an audio transmission system using multi-channel coding and decoding.
Fig. 3 is a schematic diagram illustrating how signals of different channels are encoded separately as individual and independent signals.
Fig. 4 is a schematic block diagram illustrating the basic principles of parametric stereo coding.
Fig. 5 is a schematic block diagram of a general stereo coder using a parametric prediction and a prediction/parametric residual encoding scheme.
Fig. 6 is a scatter plot illustrating the dependencies between channel level difference (CLD) and channel level sums (CLS).
Fig. 7 illustrates an example of the encoder operation of the present invention in the form of a flowchart. The overview is valid for embodiments A, B and C.
Fig. 8 is a flowchart that describes an example of the stereo synthesis chain in the decoder for embodiment A.
Fig. 9A is a schematic block diagram describing an example of the operation of the encoder and decoder for embodiment A.
Fig. 9B illustrates an example of the operation of the encoder and decoder which is valid for embodiment B.
Fig. 9C illustrates an example of the operation of the encoder and decoder which is valid for embodiment C.
Fig. 10 illustrates an example of the decoder stereo synthesis chain valid for embodiments B and C.
Fig. 11 is a plot that shows how the channel prediction factors (panning factors) varies with respect to the normalized cross-correlation coefficient . Fig. 12 shows the result from an AB test evaluation of the proposed invention in the form of a histogram of the votes.
Fig. 13 illustrates an example of the overall encoder operation for a multichannel encoder in the form of a flowchart.
Fig. 14 shows a possible multichannel embodiment of the encoder and decoder processes, where the energy measurement on received signals is performed before the multichannel prediction.
Fig. 15 is a flowchart which illustrates an example of the overall decoder operation when the energies of the decoded signal components are estimated before the multichannel prediction.
Fig. 16 shows a possible multichannel embodiment of the encoder and decoder processes, where the energy measurement of received signals are performed after the multichannel prediction.
Fig. 17 is a flowchart which illustrates an example of the overall decoder operation when the energies of the decoded signal components are estimated after the multichannel prediction.
Fig. 18 is a schematic flow diagram illustrating an example of a method for audio encoding.
Fig. 19 is a schematic flow diagram illustrating an example of a method for audio decoding.
Fig. 20 is a schematic block diagram illustrating an example of an audio encoder device.
Fig. 21 is a schematic block diagram illustrating an example of an audio decoder device. Detailed description
The invention generally relates to multi-channel (i.e. at least two channels) encoding/decoding techniques in audio applications, and particularly to stereo encoding/decoding in audio transmission systems and/or for audio storage. Examples of possible audio applications include phone conference systems, stereophonic audio transmission in mobile communication systems, various systems for supplying audio services, and multi-channel home cinema systems.
The invention may for example be particularly applicable in future standards such as ITU-T WP3/SG16/Q23 SWB/stereo extension for G.729.1 and G.718, but is of course not limited to these standards.
It may be useful to begin with an overview of some concepts of multi-channel and stereo codec techniques.
In a stereo codec for example, the stereo encoding and decoding is normally performed in multiple stages. An overview of the process is depicted in Fig. 5. First, a down-mix mono signal M is formed from the left and right channels L, R. The mono signal is fed to a mono encoder from which a local synthesis M is extracted. Using the signals M , M and [L Rf , a parametric stereo encoder produces a first approximation to the input channels [L Rf . In the final stage, the prediction residual is calculated and encoded to provide further enhancement.
Channel downmix
A standard way of down-mixing is to simply add the signals together:
l(n) + r(n) m m{n) = -^ { ' ) This type of down-mixing is applied directly on the time domain signal indexed by n . In general, the down-mix is a process of reducing the number of input channels p to a smaller number of down-mix channels q. The down-mix can be any linear or non-linear combination of the input channels, performed in temporal domain or in frequency domain. The down-mix can be adapted to the signal properties.
Other types of down-mixing use an arbitrary combination of the Left and Right channels and this combination may also be frequency dependent.
In exemplary embodiments of the invention the stereo encoding and decoding is assumed to be done on a frequency band or a group of transform coefficients. This assumes that the processing of the channels is done in frequency bands. An arbitrary down-mix with frequency dependent coefficients can be written as:
Mb(k) = abLb(k) + βbRb(k) (2)
Here the index b represents the current band and k indexes the samples within that band. Without departing from the spirit of the invention, more elaborate down-mixing schemes may be used with adaptive and time variant weighting coefficients ocb and βb .
Once the mono channel has been produced it is fed to the lower layer mono codec. The stereo encoder then uses the locally decoded mono signal to produce a stereo signal.
Channel prediction
The two channels of a stereo signal are often very alike, making it useful to apply prediction techniques in stereo coding. Since the decoded mono channel M will be available at the decoder, the objective of the prediction is to reconstruct the left and right channel pair from this signal together with the transmitted quantized stereo parameters Ψ .
Figure imgf000021_0001
Subtracting the prediction from the original input signal at the encoder will form an error signal pair:
Figure imgf000021_0002
For an MMSE perspective, the optimal prediction is obtained by minimizing the error vector [εL εR]τ . This can be solved in time domain by using a time varying FIR-filter:
Figure imgf000021_0003
The equivalent operation in frequency domain can be written:
Figure imgf000021_0004
where HL(b,k) and HR(b,k) are the frequency responses of the filters hL and hR for coefficient k of the frequency band b , and Lb(k) , Rb(k) and Mb(k) are the transformed counterparts of the time signals /(«) , r(ή) and m(n) .
Among the advantages of frequency domain processing is that it gives explicit control over the phase, which is relevant to stereo perception [2]. In lower frequency regions, phase information is highly relevant but can be discarded in the high frequencies. It can also accommodate a sub-band partitioning that gives a frequency resolution which is perceptually relevant. The drawbacks of frequency domain processing are the complexity and delay requirements for the time/frequency transformations. In cases where these parameters are critical, a time domain approach is desirable.
For the targeted codec according to this exemplary embodiment of the invention, the top layers of the codec are SNR enhancement layers in MDCT domain. The delay requirements for the MDCT are already accounted for in the lower layers and the part of the processing can be reused. For this reason, the MDCT domain is selected for the stereo processing. Although well suited for transform coding, it has some drawbacks in stereo signal processing since it does not give explicit phase control. Further, the time aliasing property of MDCT may give unexpected results since adjacent frames are inherently dependent. On the other hand, it still gives good flexibility for frequency dependent bit allocation. For accurate phase representation a combination of MDCT and MDST could be used. The additional MDST signal representation would however increase the total codec bitrate and processing load. In some cases the MDST can be approximated from the MDCT by using MDCT spectra from multiple frames.
For the stereo processing, the frequency spectrum is preferably divided into processing bands. In AAC parametric stereo, the processing bands are selected to match the critical bandwidths of human auditory perception. Since the available bitrate is low the selected bands are fewer and wider, but the bandwidths are still proportional to the critical bands. Denoting the band b , the prediction can be written:
Mb (k,m) (7)
Figure imgf000022_0001
Here, k denotes the index of the MDCT coefficient in the band b and m denotes the time domain frame index. Here we let [lb' Rb'\ represent the prediction obtained with unquantized parameters wb(m) .
The solution for wb{m) which is close to [Lb Rb] in the mean square error sense is:
Figure imgf000023_0001
Here E[.] denotes the averaging operator and is defined as an example for an arbitrary time frequency variable as an averaging over a predefined time frequency region. For example:
Figure imgf000023_0002
where each frequency band b is represented with the MDCT bins of the set Bandφ) which has the size BW(b) . Note that the frequency bands may also be overlapping.
The use of the coded mono signal M in the derivation of the prediction parameters includes the coding error in the calculation. Although sensible from an MMSE perspective, this may cause instability in the stereo image that is perceptually annoying. For this reason, the prediction parameters are based on the unprocessed mono signal, excluding the mono error from the prediction.
Figure imgf000023_0003
Using the downmix equation M = (L + R}/2 we can expand this expression, here for the left channel:
Figure imgf000024_0001
Since the signals L , R and M are in MDCT domain they are real valued and the complex conjugate (*) can be omitted.
w _ E[Lb(m)Lb(m)] + E[Lb(m)Rb(m))]
2E[Mb(m)Mb(m)] ^l)
Similarly, the right channel predictor coefficient can be written
Figure imgf000024_0002
The expressions E[Lb(m)Lb(m)] and E[Rb(m)Rb(m)] corresponds to the energies of the left and right channels respectively and E[Lb (m)Rb(mj)] represents the cross-correlation in band b . Further, the sum of the predictor coefficients can be derived
Figure imgf000024_0003
E[Lb(m)Lb(m)]+E[Lb(m)Rb(m))] _ 2E[Mb(m)Mb(m)]
E[Lb(m)Lb(m)] + 2E[Lb(m)Rb(m))] + E[Rb(m)Rb(m))] _ 2E[Mb(m)Mb(m)]
4E[Mb(m)Mb(m)] _ 2 2E\Mb{m)Mb{m)\ The typical range of the channel predictor coefficients is [θ,2], but the values may go beyond these bounds for strong negative cross-correlations. The relation in equation (14) shows that the MMSE channel predictors are connected and can be seen as a single parameter that pans the subband content to the left or right channel. Hence, the channel predictor could also be called a subband panning algorithm.
Since the spatial audio properties of a stereo or multichannel audio signal are likely to change with time, the spatial parameters are preferably encoded with a variable bit rate scheme. For stationary conditions the parameter bitrate can go down to a minimum and the saved bits can be used in parts of the codec, e.g. SNR enhancements.
It may be desirable to represent the channel predictors and the input channel energies in a way that keeps the energies of the synthesized channels stable with varying degree of residual coding. The details are further explained in the exemplary embodiments.
Residual signal encoding
The difference between the predicted stereo channels and the input channels will form a prediction residual.
(15)
Figure imgf000025_0001
The residual signal contains the parts of the input channels which are not correlated with the mono down-mix channel and hence could not be modeled with prediction. Further, the prediction residual depends on the precision of the predictor function since a lower predictor resolution will likely give a larger error. Finally, since the prediction is based on the coded mono down-mix signal, the imperfections of the mono coder will also add to the residual error. The components of the residual error signal show correlation and it is beneficial to exploit this correlation when coding the error, as described in the international patent application PCT/SE2008/000272, which is incorporated herein.
Other means of residual encoding can also be applied. The prediction residual often represents the diffuse sound field which cannot be predicted. From a perceptual perspective the inter channel correlation (ICC) [2][3][4] is important. This property can be simulated using the decoded down-mix signal or predicted/upmixed signal together with a system of decorrelating filters. The principles of this invention are applicable to any representation of the prediction residual.
Problem analysis and non-limiting examples of embodiments
The inventors have made a thorough analysis of the state of the art of audio codecs to gain some useful insights in the function and performance of such codecs. In a multichannel multistage encoder, the signals will normally be composed of different components corresponding to the encoder stages. The quality of the decoded components is likely to vary with time due to limited bitrates and changing spatial properties but also the transmission conditions. If the resources are too scarce to represent a signal we can observe an energy loss, which will yield an unstable stereo image when it varies over time.
The downmix procedure used in for example MPEG PS [3] compensates for energy loss in the downmix due to channel cancellation, but does not give explicit control over the synthesized channel energies nor the prediction factors.
The approach in MPEG Surround [4][5] for example handles the presence of a prediction residual (Wet component) in combination with a parametric part (Dry component). The Wet component may be either 1) the gain adjusted parametric part, 2) the encoded prediction residual or 3) the parametric part passed through decorrelation filters. The solution in 3) can be seen as a parametric representation of the prediction residual. However, the system does not allow the three to coexist with varying proportion and hence does not offer built-in control of synthesis channel energies in this context.
For a better understanding of the invention, it will be useful to introduce concepts of a novel class of audio encoding/decoding technologies with reference to the exemplary flow diagrams of Figs. 18 and 19.
Fig. 18 is a schematic flow diagram illustrating an example of a method for audio encoding. The exemplary audio encoding method is based on an overall encoding procedure operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels. In step S1 , a first encoding process is performed for encoding a first signal representation, including a down-mix signal, of said set of audio input channels. In step S2, local synthesis is performed in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process. In step S3, a second encoding process is performed for encoding a second representation of the considered set of audio input channels, using at least the locally decoded down-mix signal as input. In step S4, input channel energies of the audio input channels are estimated. In step S5, at least one energy representation of the audio input channels is generated based on the estimated input channel energies of said audio input channels. In step S6, the generated energy representation(s) is/are encoded. In step S7, residual error signals from at least one of said encoding processes, including at least the second encoding process, are generated. In step S8, residual encoding of the residual error signals is performed in a third encoding process. In this way, an effective overall encoding of the audio input channels is obtained. The energy representation(s) of the audio input channels enables matching of the energies of output channels at the decoding side with the estimated input channel energies. Preferably, the output channels are matched with the input channels both in terms of energy and quality.
In an exemplary embodiment, the steps of generating at least one energy representation and encoding the energy representation (s) are performed in the second encoding process, as will be exemplified in greater detail later on.
Normally, the overall encoding procedure is executed for each of a relatively large number of audio frames. It should however be understood that parts of the overall encoding procedure, such as the estimation and encoding (through a suitable energy representation) of the audio input channel energies, may be performed for a selectable sub-set of frames, and in one or more selectable frequency bands. In effect, this means that, for example, the steps of generating at least one energy representation and encoding the energy representation (s) may be performed for each of a number of frames in at least one frequency band.
In a particular example, the first encoding process is a down-mix encoding process, the second encoding process is based on channel prediction to generate one or more predicted channels, and the residual error signals thus includes residual prediction error signals. In this exemplary context, it has turned out to be especially advantageous to jointly represent and encode the estimated input channel energies and the prediction parameters of the channel prediction, in the second prediction-based encoding process. Further, in the exemplary context of down-mix encoding combined with prediction-based encoding and residual encoding, there are many different realizations for the energy representation and energy encoding, each having its special advantages. In the following, three different exemplary realizations will be summarized briefly in the tables below, and described in more detail later on:
Example A.
Energy representation: • determining channel energy level differences;
• determining channel energy level sums; and
• determining delta energy measures based on the channel energy level sums and energy of the locally decoded down-mix signal from the local synthesis in connection with the first encoding process.
Energy encoding:
• quantizing the channel energy level differences; and
• quantizing the delta energy measures.
Channel prediction:
• based on unquantized channel prediction parameters.
Example B.
Energy representation: • determining channel energy level differences;
• determining channel energy level sums;
• determining delta energy measures based on the channel energy level sums and energy of the locally decoded down-mix signal from the local synthesis in connection with the first encoding process; and • determining normalized energy compensation parameters based on the delta energy measures and energies of the predicted channels normalized by energy of the locally decoded down-mix signal;
Energy encoding:
• quantizing the channel energy level differences; and
• quantizing the normalized energy compensation parameters.
Channel prediction: • based on quantized channel prediction parameters derived from quantized channel energy level differences.
Example C.
Energy representation: • determining channel energy level differences; and
• determining energy-normalized input channel cross-correlation parameters.
Energy encoding: • quantizing the channel energy level differences; and
• quantizing the energy-normalized input channel cross-correlation parameters.
Channel prediction: • based on quantized channel prediction parameters derived from quantized channel energy level differences and quantized energy- normalized input channel cross-correlation parameters.
Fig. 19 is a schematic flow diagram illustrating an example of a method for audio decoding. The exemplary audio decoding method is based on an overall decoding procedure operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels. In step S11 , a first decoding process is performed to produce at least one first decoded channel representation including a decoded down-mix signal based on a first part of said incoming bit stream. In step S12, a second decoding process is performed to produce at least one second decoded channel representation based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of audio input channels. In step S13, input channel energies of audio input channels are estimated based on estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of audio input channels. In step S14, residual decoding is performed in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals. In step S15, the residual error signals and decoded channel representations from at least one of the first and second decoding processes, including at least the second decoding process, are combined, and channel energy compensation is performed at least partly based on the estimated input channel energies for generating the multi- channel audio signal.
This means that it is possible to effectively reconstruct a multi-channel audio signal such that output channels are close to the input channels in terms of energy and/or quality. In particular, the channel energy compensation may be performed to match the energies of output channels of the multi-channel audio signal with the estimated input channel energies. Preferably, however, the output channels of the multi-channel audio signal are matched with the corresponding input channels at the encoding side both in terms of energy and quality, wherein higher quality signals may be represented with a larger proportion than lower quality signals to improve the overall quality of the output channels.
In an exemplary embodiment, the channel energy compensation is integrated into the second decoding process when producing one or more second decoded channel representations. In this context, it is beneficial to estimate the energy of the decoded down-mix signal and energies of the residual error signals, and perform the second decoding process based on the energy of the decoded down-mix signal and the energies of the residual error signals.
In an alternative exemplary embodiment, the channel energy compensation is performed after combining the residual error signals and decoded channel representations. In this context, residual error signals and decoded channel representations from at least one of the first and second decoding processes are combined into a multi-channel synthesis and then energies of the combined multi-channel synthesis are estimated. Next, the channel energy compensation is performed based on the estimated energies of the combined multi-channel synthesis and the estimated input channel energies.
In a particular example, the second decoding process to produce at least one second decoded channel representation includes synthesizing predicted channels, and the residual decoding includes generating residual prediction error signals. In this exemplary context, the second decoding process to produce at least one second decoded channel representation includes deriving one or more one energy representations of the audio input channels from the second part of the incoming bit stream, estimating channel prediction parameters at least partly based on the energy representation(s), and then synthesizing predicted channels based on the decoded down-mix signal and the estimated channel prediction parameters. In the following, three different exemplary realizations will be summarized briefly in the tables below, and described in more detail later on. The below decoding examples A-C generally correspond to the previously described encoding examples A-C.
Example A.
Deriving energy representation:
• deriving channel energy level differences and delta energy measures from the second part of the incoming bit stream.
Estimating input channel energies:
• based on estimated energy of the decoded down-mix signal, and the channel energy level differences and delta energy measures;
Estimating channel prediction parameters:
9 based on estimated input channel energies, estimated energy of the decoded down-mix signal, and estimated energies of the residual error signals.
Example B.
Deriving energy representation:
• deriving channel energy level differences and normalized energy compensation parameters from the second part of the incoming bit stream.
Estimating input channel energies: • based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the normalized energy compensation parameters.
Estimating channel prediction parameters:
• based on the channel energy level differences.
Synthesizing predicted channels:
• based on the decoded down-mix signal and the estimated channel prediction parameters.
Combining:
• combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis.
Channel energy compensation (after combining):
• estimating energies of the combined multi-channel synthesis,
• determining an energy correction factor based on estimated input channel energies and estimated energies of the combined multi-channel synthesis;
• applying the energy correction factor to the combined multi-channel synthesis to generate the multi-channel audio signal.
Example C. Deriving energy representation:
• deriving channel energy level differences and energy-normalized input channel cross-correlation parameters from the second part of the incoming bit stream. Estimating input channel energies:
• based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the energy-normalized input channel cross-correlation parameters.
Estimating channel prediction parameters:
• based on the channel energy level differences and the energy- normalized input channel cross-correlation parameters.
Synthesizing predicted channels:
• based on the decoded down-mix signal and the estimated channel prediction parameters.
Combining: • combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis.
Channel energy compensation (after combining):
• estimating energies of the combined multi-channel synthesis; • determining an energy correction factor based on estimated input channel energies and estimated energies of the combined multi-channel synthesis;
• applying the energy correction factor to the combined multi-channel synthesis to generate the multi-channel audio signal.
From a structural viewpoint, the invention relates to an audio encoder device and a corresponding audio decoder device, as will be exemplified with reference to the exemplary block diagrams of Figs. 20 and 21. Fig. 20 is a schematic block diagram illustrating an example of an audio encoder device. The audio encoder device 100 is configured for operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels.
The basic encoder device 100 includes a first encoder 130, a second encoder 140, energy estimator 142, an energy representation generator 144 and an energy representation encoder 146, a residual generator 155 and a residual encoder 160. The finally encoded parameters are normally collected by a multiplexer 150 for transfer to the decoding side.
The first encoder 130 is configured for receiving and encoding a first representation, including a down-mix signal, of audio input channels in a first encoding process. A down-mix unit 120 may be used for down-mixing a suitable set of the input channels into a down-mix signal. The down-mix-unit 120 may be regarded as an integral part of the basic encoder device 100, or alternatively seen as an "external" support unit.
Further, a local synthesizer 132 is arranged for performing local synthesis in connection with the first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process. The local synthesizer 132 is preferably integrated in the first encoder, but may alternatively be provided as a separate decoder implemented on the encoding side in connection with the first encoder.
The second encoder 140 is configured for receiving and encoding a second representation of the considered audio input channels in a second encoding process, using at least the locally decoded down-mix signal as input. The energy estimator 142 is configured for estimating input channel energies of the considered audio input channels, and the energy representation generator 144 is configured for generating at least one energy representation of the audio input channels based on the estimated input channel energies of the audio input channels. The energy representation encoder 146 is configured for encoding the energy representation(s). In this way, the input channel energies may be estimated and encoded on the encoding side.
The energy estimator 142 may be implemented as an integrated part of the second encoder 140, may also be arranged as a dedicated unit outside the second encoder. In an exemplary embodiment, the energy representation generator 144 and the energy representation encoder 146 are conveniently implemented in the second encoder 140, as will be exemplified in more detail later on. In other embodiments, the energy representation processing may be provided outside the second encoder.
The residual generator 155 is configured for generating residual error signals from at least one of the encoding processes, including at least the second encoding process, and the residual encoder 160 is configured for performing residual encoding of the residual error signals in a third encoding process.
The energy representation(s) generated by the energy representation generator 144, and subsequently encoded, enables matching of the energies of output channels at the decoding side with the estimated input channel energies. Alternatively, the energy representation (s) enables matching of the output channels with the input channels both in terms of energy and quality.
The energy representation generator 144 and the energy representation encoder 146 are preferably configured to generate and encode the energy representation(s) for each of a number of frames in at least one frequency band. The energy estimator 142 may be configured for continuously estimating the input channel energies, or alternatively only for a selected set of frames and/or frequency bands adapted to the activities of the energy representation generator 144 and encoder 146.
In a particular example, the first encoder 130 is a down-mix encoder, and the second encoder 140 is a parametric encoder configured to operate based on channel prediction for generating one or more predicted channels, and the residual generator 155 is configured for generating residual prediction error signals. In this exemplary context, the second encoder 140 is preferably configured for jointly representing and encoding estimated input channel energies together with channel prediction parameters.
For the exemplary context of down-mix encoding combined with prediction- based encoding and residual encoding, three different exemplary realizations will be summarized below. Further details will be given later on.
Example A. In this example, the energy representation generator 144 includes a determiner for determining channel energy level differences, a determiner for determining channel energy level sums, and a determiner for determining so- called delta energy measures based on the channel energy level sums and energy of the locally decoded down-mix signal from the local synthesis in connection with the first encoding process. The energy representation encoder 146 includes a quantizer for quantizing the channel energy level differences, and a quantizer for quantizing the delta energy measures. It may for example be beneficial for the second encoder 140 to perform channel prediction based on unquantized channel prediction parameters.
Example B. In this example, the energy representation generator 144 includes a determiner for determining channel energy level differences, a determiner for determining channel energy level sums, a determiner for determining delta energy measures based on the channel energy level sums and energy of the locally decoded down-mix signal from the local synthesis in connection with the first encoding process, and a determiner for determining so-called normalized energy compensation parameters based on the delta energy measures and energies of the predicted channels normalized by energy of the locally decoded down-mix signal. The energy representation encoder 146 includes a quantizer for quantizing the channel energy level differences, and a quantizer for quantizing the normalized energy compensation parameters.
For example, the second encoder 140 may be configured to perform channel prediction based on quantized channel prediction parameters derived from quantized channel energy level differences.
Example C.
In this example, the energy representation generator 144 includes a determiner for determining channel energy level differences, and a determiner for determining energy-normalized input channel cross-correlation parameters. The energy representation encoder 146 includes a quantizer for quantizing the channel energy level differences, and a quantizer for quantizing the energy- normalized input channel cross-correlation parameters. For example, the second encoder 140 may be configured to perform channel prediction based on quantized channel prediction parameters derived from quantized channel energy level differences and quantized energy-normalized input channel cross-correlation parameters.
Fig. 21 is a schematic block diagram illustrating an example of an audio decoder device. The audio decoder device 200 is configured for operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels. The incoming bitstream is normally received from the encoding side by a bitstream demultiplexer 250, which divides the incoming bitstream into relevant sub-sets or parts of the overall incoming bitstream.
The basic audio decoder device 200 comprises a first decoder 230, a second decoder 240, and input channel energy estimator 242, a residual decoder 260, and means 270 for combining and channel energy compensation.
The first decoder 230 is configured for producing one or more decoded channel representations including a decoded down-mix signal based on a first part of the incoming bit stream.
The second decoder 240 is configured for producing one or more second decoded channel representations based on estimated energy of the decoded down-mix signal and a second part of the incoming bit stream representative of at least one energy representation of the audio input channels.
The input channel energy estimator 242 is configured for estimating input channel energies of audio input channels based on estimated energy of the decoded down-mix signal and the second part of the incoming bit stream representative of at least one energy representation of the audio input channels. The residual decoder 260 is configured for performing residual decoding in a third decoding process based on a third part of the incoming bit stream representative of residual error signal information to generate residual error signals.
The combining and channel energy compensation means 270 is configured for combining the residual error signals and decoded channel representations from at least one of the first and second decoders/decoding processes, including at least the second decoder/decoding process, and for performing channel energy compensation at least partly based on the estimated input channel energies in order to generate the multi-channel audio signal.
For example, the means 270 for combining and performing channel energy compensation may be configured to match the energies of output channels of the multi-channel audio signal with the estimated input channel energies. Preferably, however, the means 270 for combining and performing channel energy compensation is configured to match the output channels with the corresponding input channels at the encoding side both in terms of energy and quality, wherein higher quality signals are represented with a larger proportion than lower quality signals to improve the overall quality of the output channels.
As will be understood from the exemplary embodiments described later on, the overall structure for combining and channel energy compensation can be realized in several different ways.
For example, the channel energy compensation may be integrated into the second decoder. In this exemplary case, the second decoder 240 is preferably configured to operate based on the energy of the decoded down-mix signal and the energies of the residual error signals, implying that the audio decoder device 200 also comprises means for estimating energy of the decoded down- mix signal and energies of the residual error signals.
Alternatively, the decoder device includes a combiner for combining the residual error signals and the relevant decoded channel representations into a combined multi-channel synthesis, and a channel energy compensator for applying channel energy compensation on the combined multi-channel synthesis to generate the multi-channel audio signal. In this exemplary case, the audio decoder device preferably includes an estimator for estimating energies of the combined multi-channel synthesis, and the channel energy compensator is configured for applying channel energy compensation based on the estimated energies of the combined multi-channel synthesis and the estimated input channel energies.
In a particular example, the first decoder 230 is a down-mix decoder, the second decoder 240 is a parametric decoder configured for synthesizing predicted channels, and the residual decoder 260 is configured for generating residual prediction error signals. In this exemplary context, the second decoder 240 may include a deriver 241 (or may otherwise be configured) for deriving the energy representation(s) of the audio input channels from the second part of the incoming bit stream, an estimator for estimating channel prediction parameters at least partly based on the energy representation(s), and a synthesizer for synthesizing predicted channels based on the decoded down- mix signal and the estimated channel prediction parameters.
For the exemplary context of down-mix decoding combined with prediction- based decoding and residual decoding, three different exemplary realizations will be summarized below. Further details will be given later on. Example A.
In this example, the deriver 241 is configured for deriving channel energy level differences and delta energy measures from the second part of the incoming bit stream. The estimator 242 for estimating input channel energies is configured for estimating input channel energies based on estimated energy of the decoded down-mix signal, and the channel energy level differences and delta energy measures. The estimator for estimating channel prediction parameters is preferably configured for estimating channel prediction parameters based on estimated input channel energies, estimated energy of the decoded down-mix signal, and estimated energies of the residual error signals.
Example B.
In this example, the deriver 241 is configured for deriving channel energy level differences and normalized energy compensation parameters from the second part of said incoming bit stream. The estimator 242 for estimating input channel energies is configured for estimating input channel energies based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the normalized energy compensation parameters. The estimator for estimating channel prediction parameters is configured for estimating channel prediction parameters based on the channel energy level differences, and the synthesizer for synthesizing predicted channels is configured for synthesizing predicted channels based on the decoded down- mix signal and the estimated channel prediction parameters. In this example, the means 270 for combining and for performing channel energy compensation includes a combiner for combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis, and a channel energy compensator. The channel energy compensator includes an estimator for estimating energies of the combined multi-channel synthesis, a determiner for determining an energy correction factor based on estimated input channel energies and estimated energies of the combined multi-channel synthesis, and an energy corrector for applying the energy correction factor to the combined multi-channel synthesis to generate the multi-channel audio signal.
Example C.
In this example, the deriver 241 is configured for deriving channel energy level differences and energy-normalized input channel cross-correlation parameters from the second part of the incoming bit stream. The estimator 242 for estimating input channel energies is configured for estimating input channel energies based on estimated energy of the decoded down-mix signal, and the channel energy level differences and the energy-normalized input channel cross-correlation parameters. The estimator for estimating channel prediction parameters is preferably configured for estimating channel prediction parameters based on the channel energy level differences and the energy- normalized input channel cross-correlation parameters. The synthesizer for synthesizing predicted channels is configured for synthesizing predicted channels based on the decoded down-mix signal and the estimated channel prediction parameters. In this example, the means 270 for combining and for performing channel energy compensation includes a combiner for combining the residual error signals and the synthesized predicted channels into a combined multi-channel synthesis, and a channel energy compensator. In this example, the channel energy compensator includes an estimator for estimating energies of the combined multi-channel synthesis, a determiner for determining an energy correction factor based on estimated input channel energies and estimated energies of the combined multi-channel synthesis, an energy corrector for applying the energy correction factor to the combined multi-channel synthesis to generate the multi-channel audio signal. In a particular example, the invention aims to solve at least one, and preferably both of the following two problems: to obtain optimal channel prediction and maintain explicit control over the output channel energies. The components of the signal may show individual variations over time in energy and quality, such that a simple adding of the signal components would give an unstable impression in terms of energy and overall quality. The energy and quality variations can have a variety of reasons out of which a few can be mentioned here:
• A signal component may be lost or degraded due to transmission conditions.
• Components of the signal could be deliberately attenuated in the encoder, knowing that the lost energy will be recovered in the decoder. Such attenuation may be based on for instance perceptual importance.
• Parts of the signal may be lost due to limitations in the overall encoder to represent them. Due to for instance limited bitrates or modeling capabilities, parts of the signal may fall outside of the scope of the overall encoder. Seen from a general perspective, the individual encoder and related decoder processes each represent a subspace which the true input signal is projected onto. The final residual or coding error is orthogonal to the union of the subspaces which represent the overall encoder and decoder. The final residual cannot be represented with these subspaces, but its energy can be estimated and compensated for if we know or can at least estimate the input energies and the energies of the received subspace components.
An efficient solution to these and other problems may for example be implemented by means of a joint representation and encoding of both the energies and prediction parameters in a way that is robust to the possible energy and quality variations of the different components, as previously mentioned.
The invention generally relates to an overall encoding procedure and associated decoding procedure. The encoding procedure involves at least two signal encoding processes operating on signal representations of a set of audio input channels. It also involves a dedicated process to estimate the energies of the input channels. A basic idea of the present invention is to use local synthesis in connection with a first encoding process to generate a locally decoded signal, including a representation of the encoding error of the first encoding process, and apply this locally decoded signal as input to a second encoding process. The sequence of encoding processes can be seen as refinement steps of the overall encoding process, or as capturing different properties of the signal.
For example, the first encoding process may be a main encoding process such as a mono encoding process or more generally a down-mix encoder, and the second encoding process may be an auxiliary encoding process such as a stereo encoding process or a general parametric encoding process. The overall encoding procedure operates on at least two (multiple) audio input channels, including stereophonic encoding as well as more complex multi- channel encoding.
Each encoding process is associated with a decoding process. In the overall decoding procedure the decoded signals from each encoding process are preferably combined such that the output channels are close to the input channels both in terms of energy and quality. Normally, the combination step also adapts to the possible loss of one or more signal representation in part or in whole, such that the energy and quality is optimized with the signals at hand in the decoder. In the combination step the qualities of the signal components may also be considered so that higher quality signals are represented with a larger proportion than the low quality signals, and thereby improving the overall quality of the output channels.
From a structural or implementational perspective, the invention relates to an encoder and an associated decoder. The overall encoder basically comprises at least two encoders for encoding different representations of input channels. Local synthesis in connection with a first encoder generates a locally decoded signal, and this locally decoded signal is applied as input to a second encoder. The overall encoder also generates energy representations of the input channels. The overall decoder includes decoding procedures associated with each encoding procedure in the encoder. It further includes a combination stage where the decoded components are combined with stable energy and quality, facing possible partial or total loss of one or more of the decoded signals.
• The invention aims to solve at least one, and preferably both of the following two problems: to obtain optimal channel prediction and maintain explicit control over the output channel energies. The components of the signal may show individual variations over time in energy and quality, such that a simple adding of the signal components would give an unstable impression in terms of energy and overall quality.
A solution to these and other problems may for example be implemented by means of a joint representation and encoding of both the energies and prediction parameters in a way that is robust to the possible energy and quality variations of the different components.
In the following, non-limiting examples of different methods of obtaining the energy conservation will be presented, namely embodiments A, B and C. It should be understood that these embodiments are merely examples. For example, they are primarily focusing on stereo applications, and may thus be generalized for applications involving more than two audio channels. Common for these embodiments is that they preserve the synthesis energy with varying resolution on the residual encoding. Some of the differences of the exemplary embodiments are further discussed later on.
An overview of an exemplary stereo case is presented in Fig. 7. In the first step S21 , the encoder performs the down-mix on the input signals and feeds it to the mono encoder, extracting a locally decoded downmix signal in step S22. It further estimates and encodes the input channel energies in step S23. Next, the channel prediction parameters are derived in step S24. In step S25 a local synthesis of the predicted/parametric stereo is created and subtracted from the input signals, forming a prediction/parametric residual which is encoded with suitable methods in step S26. Further iterative refinement steps may be taken if more encoding stages are possible in step S27. This is executed in step S28 by performing a local synthesis and subtracting the encoded prediction residual from the prediction residual from the previous iteration and encoding the new residual of the current iteration. The example encoder process depicted in Fig. 7 constitutes an overview which is valid for all presented embodiments A, B and C. It should however be noted that the underlying details of the steps outlined in Fig. 7 are different for each presented embodiment, as will be further explained.
An example decoder reconstructs the decoded downmix signal which is identical to the locally decoded downmix signal in the encoder. The input channel energies are estimated using the decoded down-mix signal together with encoded energy representation. The channel prediction parameters are derived. The decoder further analyses the energies of the synthesized signals and adjusts the energies to the estimated input channel energies. This step may also be incorporated in the channel prediction step as we shall see in embodiment A. Further, the process of energy adjustment may also consider the qualities of signal components, such that lower quality components may be suppressed in favour of higher quality components.
Expressed in the terms of [5] the invention may be regarded as a prediction based upmix which allows multiple components per channel, and further has the energy preserving properties of the energy based upmix.
The term "upmix", which is commonly used in the context of MPEG Surround, will be used synonymously with the expressions "channel prediction" and "parametric multichannel synthesis".
Although encoding/decoding is often performed on a frame-by-frame basis, it is possible to perform bit allocation and encoding/decoding on variable sized frames, allowing signal adaptive optimized frame processing.
The embodiments described below are merely given as examples, and it should be understood that the present invention is not limited thereto.
Exemplary embodiment A
In this non-limiting example the encoder and decoder operates on a stereo input and output signals respectively. An overview of this embodiment is presented in Fig. 9A. The encoder of Fig. 9A basically includes a down-mixer that creates a mono signal from the stereo input signals, a mono encoder which encodes the down-mix signal and produces a locally decoded down-mix synthesis. Further, it includes a parametric stereo encoder which creates a first representation of the input stereo channels using the locally decoded down- mix signal and also estimates the input channel energies, creates an energy representation and encodes the representation to be used in the decoder. The encoder also creates a stereo prediction residual which is encoded with the residual encoder. The decoder of Fig. 9A includes a mono decoder which creates a decoded down-mix signal corresponding to the locally decoded down-mix signal of the encoder. It also includes a residual decoder which decodes the encoded stereo prediction residual. Finally, it includes an energy measurement unit and a parametric stereo decoder.
Fig. 8 explains the decoder operation in the form of a flowchart. In the first step S31 the mono decoding takes place, and the residual decoding is done in step
S32. Step S33 includes the energy measurement of the residual signal energies. A parametric stereo synthesis with integrated energy compensation is done in step S34 and the joining of the decoded residuals and the parametric stereo synthesis is done in step S35. The energy encoding and decoding and channel prediction of embodiment A are explained in more detail below.
Energy encoding and decoding - Exemplary embodiment A
For the purpose of energy encoding, we will first define the input channel energies. Let σb 2{m) denote the per-sample energy of the input channels for frequency band b of frame index m .
Figure imgf000050_0001
In a practical implementation of the energy measurement, the bandwidth normalization will be equal for all energy parameters in one band and can hence be omitted.
The differences between energies in the left and right channels are perceptually important [2]. To gain explicit control over the energy balance we form the channel level differences (CLD) and channel level sums (CLS)
Figure imgf000051_0001
The CLDs Db(m) are preferably quantized in log domain using codebooks which consider perceptual measures for CLD sensitivity. The CLSs Sb(m) show strong correlation with the energy of the down-mix signal σ2 Jm) . Since a decoded down-mix signal is available in the stereo decoder, we form a delta energy measure with respect to this signal
Figure imgf000051_0002
Further, we note that S and D are dependent variables as illustrated in Fig. 60. For large values of D , the distribution of S becomes more narrow and different codebooks may be selected depending on the CLD. For extreme CLD values the CLS will be dominated by one channel and can be set to a constant using zero bits. For example:
If we assume:
then it follows that
M »r = L + R ∞ — L
2 2
Figure imgf000051_0003
So for large CLDs the CLS will converge to a value of 4, corresponding to the 6 dB level we can observe in Fig. 6. The deviation from the 6 dB value is due to the coding noise in the mono downmix signal. The left channel energy is simply 6 dB lower than the mono energy, due to the downmix factor of 1/2. To exploit this dependency, we encode the CLS with different resolution depending on the quantized CLD. Since the CLS expresses an energy relation, we quantize this parameter in log domain.
The channel energies [σb L(m) σb R(m)f can be expressed using the variables
Figure imgf000052_0001
In the decoder we can use the quantized parameters Db (m) and ΔSb(m) to derive the estimated channel energies ά ~t2
Figure imgf000052_0002
Channel prediction - Exemplary embodiment A
The channel prediction parameters Wb(m) used in the encoder are not quantized, thereby ensuring that the prediction residual is minimal. The error from the quantization of the prediction parameters is not transferred to the prediction residual.
Assuming the energies have been encoded and transmitted to the decoder together with the encoded down-mix signal, the channel prediction parameters can be estimated from the energies. The full stereo synthesis can be written
Figure imgf000053_0001
where [έb L(m,k) έb R(m,k)f are the quantized residual signals for frequency bin k of band b of frame index m , and wb(m) are the channel prediction factors. The corresponding channel energies are
<r(«) wl(m)E Mb(m)Mb{m) + E[έb L(m)έb L(m)]
+ σ 'I:rJm) wlAm)E Mb{m)Mbim) + £.£MO)£MO)]
Figure imgf000053_0002
Under high rate assumptions the prediction error ε will be uncorrelated with the predicted signal, i.e.
Figure imgf000053_0003
Using this assumption and substituting the true synthesis energies [σb 2 Z(m)
Figure imgf000053_0004
the quantized approximation
[σb 2 L(m) σlR(m)J , the equation above can be solved for w:
Figure imgf000054_0001
Note that the sign of the square root is not known at the decoder and would also have to be encoded. However, for the typical input the prediction parameters are within the range [θ,2] and assuming a positive sign will work well for most signals. This truncation can be achieved by limiting one of the prediction factors to [θ,2] and obtaining the other factor using equation (14). If we wish to encode the sign we can exploit the fact that at most one of the channels may have a negative sign, e.g. by using a simple variable length code:
Figure imgf000054_0002
Table 1: Variable length codebook for coding the signs of the channel predictor coefficients. It exploits the high probability of two positive signs, as well as the fact that not two signs are negative in the same band.
Using this embodiment, the output channel energies are corrected using the channel prediction factors. If the decoded residual signal is close to the true residual, the channel prediction factors will be close to the optimal prediction factors used in the encoder. If the residual coding energy is lower than the true residua! energy due to e.g. low bitrate encoding, the contribution from the parametric stereo is scaled up to compensate for the energy loss. If the residual coding is zero, the algorithm inherently defaults to intensity stereo coding.
Exemplary embodiment B
In this second non-limiting example the encoder and decoder also operates on stereo signals. An overview of this embodiment is presented in Fig. 9B, where the encoder of Fig. 9B basically includes a down-mixer that creates a mono signal from the stereo input signals, a mono encoder which encodes the down- mix signal and produces a locally decoded down-mix synthesis. Further, it includes a parametric stereo encoder which creates a first representation of the input stereo channels using the locally decoded down-mix signal and also estimates the input channel energies, creates an energy representation and encodes the representation to be used in the decoder. The encoder also creates a stereo prediction residual which is encoded with the residual encoder. The decoder of Fig. 9B includes a mono decoder which creates a decoded down-mix signal corresponding to the locally decoded down-mix signal of the encoder. It also includes a residual decoder which decodes the encoded stereo prediction residual. Further, it includes a parametric stereo decoder and an energy measurement unit which operates on the combined stereo synthesis and an energy correction unit which modifies the combined stereo synthesis to create a final stereo synthesis. The flowchart of Fig. 10 describes the steps of the decoder operation. The mono decoding is done in step S41 , which is followed by a parametric stereo synthesis in step S42 and a stereo residual decoding in step S43. In step S44 the residual and parametric stereo synthesis is joined and the energy of this combined synthesis is done in step S45. Finally, step S46 includes the energy adjustment of the combined synthesis. The energy encoding and decoding and channel prediction of embodiment B are explained in more detail below.
Energy encoding and decoding - Exemplary embodiment B
An optional strategy for encoding the energies can be derived. The CLDs Db(m) are derived as before. Next, we assume the CLD should be preserved on the predicted stereo contribution without residual encoding which gives us a relation for the channel prediction factors.
Figure imgf000056_0001
Using equation (14) we can calculate the channel prediction factors from the CLDs
Figure imgf000056_0002
We note that a common scaling factor Cb(m) on the synthesized stereo signals will not affect the CLD. Adding this factor to the synthesis we match the synthesized signal energies, again assuming there is no residual coding present.
Figure imgf000057_0001
Equation (26) can be solved for Cb(m) using either the left or the right channel:
Cb(m) = σbJ.(m) \_ < (jn) b,M
Figure imgf000057_0002
These two equations give the same Cb{m) . We choose to use the higher energy channel which should give better numerical precision.
Equations (26) and (19) offer two expressions for the input channel energies. Taking the right side of the equality and setting them equal we get
Figure imgf000057_0003
From this equation we identify
(28)
+ <R(m)
where the denominator wb 2 L(m) + wb 2 R(m) equals the sum of the energies of the predicted channels normalized by the mono energy. We conclude that this energy representation is equivalent to the first representation and that it only differs in the normalization of the CLS parameters ASb(m) and Cb 2(m) . The CLD is encoded as in embodiment A. The energy compensation parameters, also referred to as normalized energy compensation parameters, Q2Ow) is also quantized in log domain just like ASb(m) , but uses a different codebook (in fact just a different log-value offset) due to the scaling difference.
The decoder derives the approximated channel energies
Figure imgf000058_0001
from the received parameters Cb (m) , bb(m) and measured decoded mono energy σb 2 ύ{m)
σu = (29)
Figure imgf000058_0002
Channel prediction - Exemplary embodiment B
In the alternative scheme the channel predictors used in the encoder are derived from the quantized CLDs
Figure imgf000059_0001
In this case the same channel predictors are used in the encoder and decoder. This ensures correct matching between predicted channels and residual coding.
Decoder energy compensation - Exemplary embodiment B
Since σb 2 was derived under the assumption of no residual coding, we must compensate for the residual coding energy if such is present in the decoder. First we synthesize the non-scaled stereo synthesis
Figure imgf000059_0002
Note that the coded residual ε differs from έ in equation (20) since different predictors were used in the encoder. The final synthesis is produced by applying an energy correction factor that restores the approximated channel energies
Lb''(m,k) Lb'(m,kφlL(m)IE U(m,k)J
(32) RX' m,k) R[{m,k)plJm)IE [K(m,k)J
If the residual coding is zero, the energy correction factor will evaluate to 1. This method also compensates for the fact that the high rate assumption may not hold if the available bit rate is limited and the residual coding may show correlation with the predicted channels. Exemplary embodiment C
The third non-limiting example is also a stereo encoder and decoder embodiment. The overview of this embodiment is presented in Fig. 9C, where the encoder of Fig. 9C basically includes a down-mixer that creates a mono signal from the stereo input signals, a mono encoder which encodes the down- mix signal and produces a locally decoded down-mix synthesis. Further, it includes a parametric stereo encoder which creates a first representation of the input stereo channels using the locally decoded down-mix signal and also estimates the input channel energies, creates an energy representation and encodes the representation to be used in the decoder. The encoder also creates a stereo prediction residual which is encoded with the residual encoder. The decoder of Fig. 9C includes a mono decoder which creates a decoded down-mix signal corresponding to the locally decoded down-mix signal of the encoder. It also includes a residual decoder which decodes the encoded stereo prediction residual. Further, it includes a parametric stereo decoder and an energy measurement unit which operates on the combined stereo synthesis and an energy correction unit which modifies the combined stereo synthesis to create a final stereo synthesis. From an overview perspective the decoder operation of embodiment C is similar to the decoder of embodiment B, and Fig. 10 gives an accurate description of the decoder steps for both examples. The energy encoding and decoding and channel prediction of embodiment C are explained in more detail below.
Energy encoding and decoding - Exemplary embodiment C
From equations (12) and (13) we see that the channel predictor coefficients share one term, the normalized cross-correlation, also referred to as energy- normalized input channel cross-correlation, which we define asp E[Lb(m,k)Rb(m,k)]
(33) Hb K E[Mb(m,k)Mb(m,k)]
Using the definition of Db (m) from equation (17) we can form yet an alternative channel energy expression
σu =
Figure imgf000061_0001
This can be rewritten as a straight-line equation which shows that the energy decreases proportionally to an increasing p .
Figure imgf000061_0002
If we assume that the energy is preserved in the mono encoding, i.e. σlM(m) = σ\ (m) . we can express the estimated channel energies in the decoder as
Figure imgf000061_0003
This approach ensures that the quantized CLD DbQn) is preserved, but it may have some energy instability due to the quantization noise in pbQή) and the encoded mono MbQn,k) . Experience shows that sudden energy increases are more perceptually annoying than energy losses. This can be handled by constraining the quantization of p in the encoder such that the energy is never overestimated in the decoder.
Figure imgf000062_0001
{^lAm')lσlΛm') ≤ σthr
We select the pb(jn) as close as possible to pb{m) from equation (33) with the constraint σb 2{m)jσb 2{rή) < σ1hr . We could ensure that the energy is never overestimated on any channel, i.e. fulfill both the lines in equation (37). Another strategy could be to make sure the energy is never overestimated in the lower energy channel, since an energy burst during almost silence is more perceptually annoying. From equation (35) we see that the energy estimate decreases with increasing p , which means we can start the search at the value given by equation (33) and perform an incremental search if the initial value does not fulfill σb 2{m)lσl(m) ≤ σlhl.. If there is an energy loss in the mono encoding, we might want to search for decreasing p to minimize σb 2(m)-σb 2(m) , but this may have an undesired effect on the channel prediction parameters. The effect on the channel prediction with varying p will be further discussed later on.
Channel prediction - Exemplary embodiment C
Using p and D , the MMSE optimal channel prediction factors can be written
Figure imgf000062_0002
We can note that for equal input channel energies D = I the channel prediction coefficients become independent of p . In Fig. 11 we can see that the channel prediction parameters move towards the middle for increasing p . We can conclude that the method outlined in equation (37) is safe with respect to the channel prediction parameters, since a slight increase in p will only yield a prediction that with slightly increased channel leakage, but where the CLDs are still preserved.
Further we can note that for very large negative p , the channel prediction factors become insensitive to D . The dependencies between these variables can be exploited in order to give low distortion at a minimum bitrate.
Given the encoded Db(m) and pb(m) we derive the encoder channel prediction factors as
Figure imgf000063_0001
Like in embodiment B, the same channel predictors are used in both encoder and decoder. The difference from embodiment B is that the quantized MMSE optimal channel prediction factors are used. Further, as in embodiment B, the energy relations between the decoded residual and predicted channels are preserved.
Decoder energy compensation - Exemplary embodiment C
The output channel energy are corrected after joining the predicted and residual coding components just like in embodiment B. Apart from the fact that different parameters are used for channel prediction and energy estimation, the overall description in the decoder flowchart of Fig. 100 is valid also for embodiment C. For embodiment C, reference can also be made to the block diagram of Fig. 9C, as mentioned above.
Differences between exemplary embodiments A-C
The presented exemplary embodiments A, B and C give equal accuracy in representing the CLD in the synthesized stereo sound. They also have equivalent behavior in the case of no residual coding, in which case they all default to an intensity stereo algorithm. A main difference lies in which channel prediction parameters are used in the encoder, and how they are derived in the decoder. The preferred embodiment will be different depending on various parameters, e.g. the available bitrate and the complexity of the input signals with regard to coding and spatial information.
In embodiment A, the optimal unquantized channel predictors are used in the encoder. The channel predictors used in the decoder will be the same if the bitrate is high and the residual coding approaches perfect reconstruction. For intermediate bitrates, only the predicted part of the stereo is scaled to compensate for energy loss in the residual. If the residual coding is noisier than the predicted stereo component due to e.g. low bitrate residual encoding, using a larger proportion of the predicted stereo is a desirable feature.
For embodiment B, the quantized channel predictors are used in the encoder. The prediction will not be optimal in the MMSE sense, but it guarantees that the scaling of the predicted signal and the coded residual signal is matched. This is important if the coding error of the mono signal is dominant and the residual mainly corrects this error.
The benefit of embodiment C is that it gives a compact representation of both the channel energies and the channel prediction factors. The parameters show dependencies that can be exploited for encoding. If the mono encoding is not conserving the energy of the mono signal, an additional safeguard for energy increases can be added with a predictable impact on the parametric stereo prediction performance.
Which one of these strategies is most beneficial may depend on the situation in terms of available bitrate and the typical input signal. For the SWB/stereo extension to G.718, it was found however that embodiment B was giving good results. These methods can also be combined, using different algorithms for different frequency bands. Such combinations could also be made adaptive, in which case the selected strategy would have to be signaled to the decoder. It could also be done without additional signaling if the strategy selection is performed using parameters that are already transmitted to the decoder.
Other encoding schemes could also be combined with the described methods.
The invention achieves scalability while maintaining channel energy levels which are important for stereo image perception. When the residual coding is nil, the system will default to an intensity stereo algorithm. As the residual coding increases, the synthesized output will scale towards perfect reconstruction while maintaining channel energies and stereo image stability.
AB listening test evaluation
As an example, the exemplary method B was tested. The baseline for comparison was using CLD based channel prediction (intensity stereo) in the range 2.2 kHz to 7.0 kHz. The applied method below 2.2 kHz was identical for tested candidates. Fig. 12 shows a histogram of the votes, indicating a preference for the invention.
The audio material consisted of 7 audio clips taken from the AMR-WB+ selection test material. As already mentioned, the principles of this invention are also applicable to multi-channel scenarios where the input and output channels are more than two.
In the following, an overview of an exemplary multi-channel embodiment operating on p input channels will finally be given.
Assume the input signal is a multiple channel signal x = [xx X2 ■•• xp] with P channels. The encoder creates a down-mix signal Y = [Y1 Y2 •■• Yq] with q channels, where p >q . The properties of the down-mix may create dependencies between the channels of the original multichannel signal and the down-mixed signal which can be exploited to make efficient representations of the channel energies and channel predictors. The multichannel down-mix as such can be performed in multiple stages as have been seen in prior art [5]. If pair-wise channel combinations are performed, principles from the stereo embodiments may apply. The down-mixed signal is fed to a first stage encoder which operates on q channels, and a locally decoded down-mix signal Ϋ is extracted from this process. This signal is used in a multichannel prediction or upmix step, which creates a first approximation
X to the input multichannel signal. The approximation is subtracted from the original input signal, forming a multichannel prediction residual or parametric residual. The residual is fed to a second encoding stage. If desired, a locally decoded residual signal can be extracted and subtracted from the original residual signal to create a second stage residual signal. This encoding process can be repeated to provide further refinements converging towards the original input signal, or to capture different properties of the signal. The encoded prediction, energy and residual parameters are transmitted or stored to be used in a decoder. An overview of an example of the encoding process can be seen in Fig. 13. In an exemplary embodiment, the overall decoder performs a decoding of the down-mixed signal corresponding to the locally decoded down-mixed signal in the encoder. The encoded residual or residuals are decoded. Using the transmitted prediction and energy parameters, a first stage multichannel prediction or upmix is performed. The multichannel prediction may be different from the multichannel prediction in the encoder. The decoder measures the energies of the received and decoded signals, such as the decoded down- mixed signal, the predicted multichannel signal and residual signal or signals. An energy estimate of the input channel energies is calculated and is used to combine the decoded signal components into a multichannel output signal. The energies may be measured before the prediction stage, allowing the output energy to be controlled jointly with the prediction as illustrated in Fig. 14 and Fig. 15. The energies may also be measured after the signal components have been joined and adjusted in a final stage on the joined components as illustrated in Fig. 16 and Fig. 17.
The embodiments described above are merely given as examples, and it should be understood that the present invention is not limited thereto. Further modifications, changes and improvements which retain the basic underlying principles disclosed and claimed herein are within the scope of the invention.
Abbreviations
AAC Advanced Audio Coding
AAC-BSAC Advanced Audio Coding - Bit-Sliced Arithmetic Coding
AMR Adaptive Multi-Rate
AMR-WB Adaptive Multi-Rate Wide Band
AOT Audio Object Type
BCC Binaural cue coding [2]
BMLD Binaural masking level difference
CELP Code Excited Linear Prediction
CfI Call for Information
CLD Channel level difference
CLS Channel level sum
EV Embedded VBR (Variable Bit Rate)
ICC Inter-channel correlation
ICP Inter-channel prediction
ITU International Telecommunication Union
LSB Least Significant Bit
MDCT Modified discrete cosine transform
MDST Modified discrete sinusoid transform MMSE Minimum mean squared error
MPEG Moving Picture Experts Group
MPEG-SLS MPEG-Scalable to Lossless
MSB Most Significant Bit
MSE Mean Squared Error
NB Narrow Band (8 kHz samplerate)
SNR Signal-to-noise ratio
SWB Super Wide Band (32 kHz samplerate)
PS Parametric Stereo
VMR-WB Variable Multi Rate-Wide Band
VoIP Voice over Internet Protocol
WB Wide Band (16 kHz samplerate) xDSL x Digital Subscriber Line
References
[1] ISO/IEC JTC 1 , SC 29, WG 11/M11657, "Performance and functionality of existing MPEG-4 technology in the context of CfI on Scalable Speech and Audio Coding", Jan. 2005.
[2] C. Faller and F. Baumgarte, "Binaural cue coding - Part I:
Psychoacoustic fundamentals and design principles", IEEE Trans. Speech Audio Processing, vol. 11 , pp. 509-519, Nov. 2003.
[3] Samsudin et al, "A stereo to mono downmixing scheme for MPEG-4 parametric stereo encoder", ICASSP Proceedings, vol. 5, pp. V - V May 2006.
[4] J. Herre et al, "The Reference Model Architecture for MPEG Spatial
Audio Coding", AES 118th Convention, Paper 6447, May 2005.
[5] ISO/IEC JTC 1 , SC 29, WG 11/N7806, "MPEG audio technologies
— Part 1 : MPEG Surround", pp. 113-114, February 2007.

Claims

1. An audio encoding method based on an overall encoding procedure operating on signal representations of a set of audio input channels of a multi- channel audio signal having at least two channels, wherein said audio encoding method comprises the steps of: performing (S1) a first encoding process for encoding a first signal representation, including a down-mix signal, of said set of audio input channels; - performing (S2) local synthesis in connection with said first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process; performing (S3) a second encoding process for encoding a second representation of said set of audio input channels, using at least said locally decoded down-mix signal as input; estimating (S4) input channel energies of said audio input channels; generating (S5) at least one energy representation of said audio input channels based on the estimated input channel energies of said audio input channels; encoding (S6) said at least one energy representation; and generating (S7) residual error signals from at least one of said encoding processes, including at least said second encoding process; performing (S8) residual encoding of said residual error signals in a third encoding process.
2. The audio encoding method of claim 1 , wherein said at least one energy representation enables matching of the energies of output channels at the decoding side with the estimated input channel energies.
3. The audio encoding method of claim 2, wherein said at least one energy representation enables matching of the output channels with the input channels both in terms of energy and quality.
4. The audio encoding method of claim 1 , wherein said steps (S5, S6) of generating at least one energy representation and encoding said at least one energy representation are performed in said second encoding process.
5. The audio encoding method of claim 4, wherein said steps (S5, S6) of generating at least one energy representation and encoding said at least one energy representation are performed for each of a number of frames in at least one frequency band.
6. The audio encoding method of claim 1 , wherein said first encoding process is a down-mix encoding process, said second encoding process is based on channel prediction for generating at least one predicted channel, and said step (S7) of generating residual error signals includes the step of generating residual prediction error signals.
7. The audio encoding method of claim 6, wherein the estimated input channel energies and prediction parameters of said channel prediction are jointly represented and encoded in said second encoding process.
8. The audio encoding method of claim 6, wherein said step (S5) of generating at least one energy representation includes the steps of: determining channel energy level differences; determining channel energy level sums; and determining delta energy measures based on said channel energy level sums and energy of said locally decoded down-mix signal from said local synthesis in connection with said first encoding process, wherein said step (S6) of encoding said at least one energy representation includes the steps of: quantizing said channel energy level differences; and quantizing said delta energy measures.
9. The audio encoding method of claim 8, wherein said channel prediction is based on unquantized channel prediction parameters.
10. The audio encoding method of claim 6, wherein said step (S5) of generating at least one energy representation includes the steps of: determining channel energy level differences; - determining channel energy level sums; determining delta energy measures based on said channel energy level sums and energy of said locally decoded down-mix signal from said local synthesis in connection with said first encoding process; and determining normalized energy compensation parameters based on said delta energy measures and energies of the predicted channels normalized by energy of said locally decoded down-mix signal; wherein said step (S6) of encoding said at least one energy representation includes the steps of: quantizing said channel energy level differences; and - quantizing said normalized energy compensation parameters.
11. The audio encoding method of claim 10, wherein said channel prediction is based on quantized channel prediction parameters derived from quantized channel energy level differences.
12. The audio encoding method of claim 6, wherein said step (S5) of generating at least one energy representation includes the steps of: determining channel energy level differences; and - determining energy-normalized input channel cross-correlation parameters; wherein said step (S6) of encoding said at least one energy representation includes the steps of: quantizing said channel energy level differences; and - quantizing said energy-normalized input channel cross-correlation parameters.
13. The audio encoding method of claim 12, wherein said channel prediction is based on quantized channel prediction parameters derived from quantized channel energy level differences and quantized energy-normalized input channel cross-correlation parameters.
14. An audio encoder device (100) operating on signal representations of a set of audio input channels of a multi-channel audio signal having at least two channels, wherein said audio encoder device (100) comprises: a first encoder (130) for encoding a first representation, including a down-mix signal, of said set of audio input channels in a first encoding process; a local synthesizer (132) for performing local synthesis in connection with said first encoding process to generate a locally decoded down-mix signal including a representation of the encoding error of the first encoding process; a second encoder (140) for encoding a second representation of said set of audio input channels in a second encoding process, using at least said locally decoded down-mix signal as input; an energy estimator (142) for estimating input channel energies of said audio input channels; an energy representation generator (144) for generating at least one energy representation of said audio input channels based on the estimated input channel energies of said audio input channels; an energy representation encoder (146) for encoding said at least one energy representation; a residual generator (155) for generating residual error signals from at least one of said encoding processes, including at least said second encoding process; and a residual encoder (160) for performing residual encoding of said residual error signals in a third encoding process.
15. The audio encoder device of claim 14, wherein said energy representation generator (144) is configured for generating at least one energy representation to enable matching of the energies of output channels at the decoding side with the estimated input channel energies.
16. The audio encoder device of claim 15, wherein said energy representation generator (144) is configured for generating at least one energy representation to enable matching of the output channels with the input channels both in terms of energy and quality.
17. The audio encoder device of claim 14, wherein said energy representation generator (144) and said energy representation encoder (146) are implemented in said second encoder (140).
18. The audio encoder device of claim 17, wherein said energy representation generator (144) and said energy representation encoder (146) are configured to generate and encode said at least one energy representation for each of a number of frames in at least one frequency band.
19. The audio encoder device of claim 14, wherein said first encoder (130) is a down-mix encoder, said second encoder (140) is a parametric encoder configured to operate based on channel prediction for generating at least one predicted channel, and said residual generator (155) is configured for generating residual prediction error signals.
20. The audio encoder device of claim 19, wherein said second encoder (140) is configured for jointly representing and encoding estimated input channel energies and prediction parameters of said channel prediction.
21. The audio encoder device of claim 19, wherein said energy representation generator (144) includes: a determiner for determining channel energy level differences; - a determiner for determining channel energy level sums; and a determiner for determining delta energy measures based on said channel energy level sums and energy of said locally decoded down-mix signal from said local synthesis in connection with said first encoding process, wherein said energy representation encoder (146) includes: - a quantizer for quantizing said channel energy level differences; a quantizer for quantizing said delta energy measures.
22. The audio encoder device of claim 21 , wherein said second encoder (140) is configured to perform channel prediction based on unquantized channel prediction parameters.
23. The audio encoder device of claim 19, wherein said energy representation generator (144) includes: a determiner for determining channel energy level differences; a determiner for determining channel energy level sums; a determiner for determining delta energy measures based on said channel energy level sums and energy of said locally decoded down-mix signal from said local synthesis in connection with said first encoding process; and a determiner for determining normalized energy compensation parameters based on said delta energy measures and energies of the predicted channels normalized by energy of said locally decoded down-mix signal; wherein said energy representation encoder (146) includes: a quantizer for quantizing said channel energy level differences; a quantizer for quantizing said normalized energy compensation parameters.
24. The audio encoder device of claim 23, wherein said second encoder (140) is configured to perform channel prediction based on quantized channel prediction parameters derived from quantized channel energy level differences.
25. The audio encoder device of claim 19, wherein said energy representation generator (144) includes: a determiner for determining channel energy level differences; a determiner for determining energy-normalized input channel cross-correlation parameters; wherein said energy representation encoder (146) includes: a quantizer for quantizing said channel energy level differences; - a quantizer for quantizing said energy-normalized input channel cross-correlation parameters.
26. The audio encoder device of claim 25, wherein said second encoder (140) is configured to perform channel prediction based on quantized channel prediction parameters derived from quantized channel energy level differences and quantized energy-normalized input channel cross-correlation parameters.
27. An audio decoding method based on an overall decoding procedure operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels, wherein said method comprises the steps of: performing (S11 ) a first decoding process to produce at least one first decoded channel representation including a decoded down-mix signal based on a first part of said incoming bit stream; - performing (S12) a second decoding process to produce at least one second decoded channel representation based on estimated energy of said decoded down-mix signal and a second part of said incoming bit stream representative of at least one energy representation of audio input channels; estimating (S13) input channel energies of audio input channels based on estimated energy of said decoded down-mix signal and said second part of said incoming bit stream representative of at least one energy representation of audio input channels; performing (S14) residual decoding in a third decoding process based on a third part of said incoming bit stream representative of residual error signal information to generate residual error signals; combining said residual error signals and decoded channel representations from at least one of said first and second decoding processes, including at least said second decoding process, and performing channel energy compensation at least partly based on the estimated input channel energies for generating said multi-channel audio signal (S15).
28. The audio decoding method of claim 27, wherein said channel energy compensation is performed to match the energies of output channels of said multi-channel audio signal with the estimated input channel energies.
29. The audio decoding method of claim 28, wherein the output channels of said multi-channel audio signal are matched with the corresponding input channels at the encoding side both in terms of energy and quality, wherein higher quality signals are represented with a larger proportion than lower quality signals to improve the overall quality of the output channels.
30. The audio decoding method of claim 27, wherein said channel energy compensation is integrated into said second decoding process when producing at least one second decoded channel representation.
31. The audio decoding method of claim 30, wherein energy of said decoded down-mix signal and energies of said residual error signals are estimated, and said second decoding process is based on the energy of said decoded down-mix signal and the energies of said residual error signals.
32. The audio decoding method of claim 27, wherein said channel energy compensation is performed after combining said residual error signals and decoded channel representations.
33. The audio decoding method of claim 32, wherein residual error signals and decoded channel representations from at least one of said first and second decoding processes are combined into a multi-channel synthesis and energies of the combined multi-channel synthesis are estimated, and said channel energy compensation is performed based on the estimated energies of the combined multi-channel synthesis and the estimated input channel energies.
34. The audio decoding method of claim 27, wherein said step (S12) of performing a second decoding process to produce at least one second decoded channel representation includes the step of synthesizing predicted channels, and said step (S14) of performing residual decoding includes the step of generating residual prediction error signals.
35. The audio decoding method of claim 34, wherein said step (S12) of performing a second decoding process to produce at least one second decoded channel representation includes the steps of: deriving said at least one energy representation of said audio input channels from said second part of said incoming bit stream; estimating channel prediction parameters at least partly based on said at least one energy representation; and synthesizing predicted channels based on the decoded down-mix signal and the estimated channel prediction parameters.
36. The audio decoding method of claim 35, wherein said step of deriving said at least one energy representation includes the step of deriving channel energy level differences and delta energy measures from said second part of said incoming bit stream; and wherein said step of estimating input channel energies is performed based on estimated energy of said decoded down-mix signal, and said channel energy level differences and delta energy measures; wherein said step of estimating channel prediction parameters is performed based on estimated input channel energies, estimated energy of said decoded down-mix signal, and estimated energies of said residual error signals.
37. The audio decoding method of claim 35, wherein said step of deriving said at least one energy representation includes the step of deriving channel energy level differences and normalized energy compensation parameters from said second part of said incoming bit stream; and wherein said step of estimating input channel energies is performed based on estimated energy of said decoded down-mix signal, and said channel energy level differences and said normalized energy compensation parameters; wherein said step of estimating channel prediction parameters is performed based on said channel energy level differences; wherein said step of synthesizing predicted channels is based on the decoded down-mix signal and the estimated channel prediction parameters; wherein said step of combining said residual error signals and decoded channel representations includes the step of combining said residual error signals and said synthesized predicted channels into a combined multichannel synthesis; wherein said channel energy compensation is performed after said step of combining by: estimating energies of said combined multi-channel synthesis; determining an energy correction factor based on estimated input channel energies and estimated energies of said combined multi-channel synthesis; applying said energy correction factor to said combined multichannel synthesis to generate said multi-channel audio signal.
38. The audio decoding method of claim 35, wherein said step of deriving said at least one energy representation includes the step of deriving channel energy level differences and energy-normalized input channel cross- correlation parameters from said second part of said incoming bit stream; and wherein said step of estimating input channel energies is performed based on estimated energy of said decoded down-mix signal, and said channel energy level differences and said energy-normalized input channel cross-correlation parameters; wherein said step of estimating channel prediction parameters is performed based on said channel energy level differences and said energy- normalized input channel cross-correlation parameters; wherein said step of synthesizing predicted channels is based on the decoded down-mix signal and the estimated channel prediction parameters; wherein said step of combining said residual error signals and decoded channel representations includes the step of combining said residual error signals and said synthesized predicted channels into a combined multichannel synthesis; wherein said channel energy compensation is performed after said step of combining by: estimating energies of said combined multi-channel synthesis; determining an energy correction factor based on estimated input channel energies and estimated energies of said combined multi-channel synthesis; applying said energy correction factor to said combined multi- channel synthesis to generate said multi-channel audio signal.
39. An audio decoder device (200) operating on an incoming bit stream for reconstructing a multi-channel audio signal having at least two channels, wherein said audio decoder device (200) comprises: - a first decoder (230) for producing at least one first decoded channel representation including a decoded down-mix signal based on a first part of said incoming bit stream; a second decoder (240) for producing at least one second decoded channel representation based on estimated energy of said decoded down-mix signal and a second part of said incoming bit stream representative of at least one energy representation of audio input channels; an estimator (242) for estimating input channel energies of audio input channels based on estimated energy of said decoded down-mix signal and said second part of said incoming bit stream representative of at least one energy representation of audio input channels; a residual decoder (260) for performing residual decoding in a third decoding process based on a third part of said incoming bit stream representative of residual error signal information to generate residual error signals; and - means (270) for combining said residual error signals and decoded channel representations from at least one of said first and second decoding processes, including at least said second decoding process, and for performing channel energy compensation at least partly based on the estimated input channel energies for generating said multi-channel audio signal.
40. The audio decoder device of claim 39, wherein said means (270) for combining and for performing channel energy compensation is configured to match the energies of output channels of said multi-channel audio signal with the estimated input channel energies.
41. The audio decoder device of claim 40, wherein said means (270) for combining and for performing channel energy compensation is configured to match the output channels of said multi-channel audio signal with the corresponding input channels at the encoding side both in terms of energy and quality, wherein said means (270) for combining and for performing channel energy compensation is configured to represent higher quality signals with a larger proportion than lower quality signals to improve the overall quality of the output channels.
42. The audio decoder device of claim 39, wherein said channel energy compensation is integrated into said second decoder (240).
43. The audio decoder device of claim 42, wherein said audio decoder device (200) comprises means for estimating energy of said decoded down- mix signal and energies of said residual error signals, and said second decoder (240) is configured to operate based on the energy of said decoded down-mix signal and the energies of said residual error signals.
44. The audio decoder device of claim 39, wherein said means (270) for combining and for performing channel energy compensation includes a combiner for combining said residual error signals and decoded channel representations into a combined multi-channel synthesis, and a channel energy compensator for applying said channel energy compensation on said combined multi-channel synthesis to generate said multi-channel audio signal.
45. The audio decoder device of claim 44, wherein said audio decoder device (200) comprises an estimator for estimating energies of the combined multi-channel synthesis, and said channel energy compensator is configured for applying said channel energy compensation based on the estimated energies of the combined multi-channel synthesis and the estimated input channel energies.
46. The audio decoder device of claim 39, wherein said first decoder (230) is a down-mix decoder, said second decoder (240) is a parametric decoder configured for synthesizing predicted channels, and said residual decoder (260) is configured for generating residual prediction error signals.
47. The audio decoder device of claim 46, wherein said second decoder (240) includes: a deriver (241 ) for deriving said at least one energy representation of said audio input channels from said second part of said incoming bit stream; an estimator for estimating channel prediction parameters at least partly based on said at least one energy representation; and a synthesizer for synthesizing predicted channels based on the decoded down-mix signal and the estimated channel prediction parameters.
48. The audio decoder device of claim 47, wherein said deriver is configured for deriving channel energy level differences and delta energy measures from said second part of said incoming bit stream; and wherein said estimator (242) for estimating input channel energies is configured for estimating input channel energies based on estimated energy of said decoded down-mix signal, and said channel energy level differences and delta energy measures; wherein said estimator for estimating channel prediction parameters is configured for estimating channel prediction parameters based on estimated input channel energies, estimated energy of said decoded down-mix signal, and estimated energies of said residual error signals.
49. The audio decoder device of claim 47, wherein said deriver is configured for deriving channel energy level differences and normalized energy compensation parameters from said second part of said incoming bit stream; and wherein said estimator (242) for estimating input channel energies is configured for estimating input channel energies based on estimated energy of said decoded down-mix signal, and said channel energy level differences and said normalized energy compensation parameters; wherein said estimator for estimating channel prediction parameters is configured for estimating channel prediction parameters based on said channel energy level differences; wherein said synthesizer for synthesizing predicted channels is configured for synthesizing predicted channels based on the decoded down- mix signal and the estimated channel prediction parameters; wherein said means (270) for combining and for performing channel energy compensation includes a combiner for combining said residual error signals and said synthesized predicted channels into a combined multichannel synthesis, and a channel energy compensator including: an estimator for estimating energies of said combined multichannel synthesis; a determiner for determining an energy correction factor based on estimated input channel energies and estimated energies of said combined multi-channel synthesis; an energy corrector for applying said energy correction factor to said combined multi-channel synthesis to generate said multi-channel audio signal.
50. The audio decoder device of claim 47, wherein said deriver is configured for deriving channel energy level differences and energy-normalized input channel cross-correlation parameters from said second part of said incoming bit stream; and wherein said estimator (242) for estimating input channel energies is configured for estimating input channel energies based on estimated energy of said decoded down-mix signal, and said channel energy level differences and said energy-normalized input channel cross-correlation parameters; wherein said estimator for estimating channel prediction parameters is configured for estimating channel prediction parameters based on said channel energy level differences and said energy-normalized input channel cross-correlation parameters; wherein said synthesizer for synthesizing predicted channels is configured for synthesizing predicted channels based on the decoded down- mix signal and the estimated channel prediction parameters; wherein said means (270) for combining and for performing channel energy compensation includes a combiner for combining said residual error signals and said synthesized predicted channels into a combined multichannel synthesis, and a channel energy compensator including: an estimator for estimating energies of said combined multichannel synthesis; a determiner for determining an energy correction factor based on estimated input channel energies and estimated energies of said combined multi-channel synthesis; an energy corrector for applying said energy correction factor to said combined multi-channel synthesis to generate said multi-channel audio signal.
PCT/SE2009/051071 2008-10-10 2009-09-25 Energy conservative multi-channel audio coding WO2010042024A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN2009801401393A CN102177542B (en) 2008-10-10 2009-09-25 Energy conservative multi-channel audio coding
EP09819478.0A EP2345027B1 (en) 2008-10-10 2009-09-25 Energy-conserving multi-channel audio coding and decoding
US13/122,880 US9330671B2 (en) 2008-10-10 2009-09-25 Energy conservative multi-channel audio coding
JP2011530989A JP5608660B2 (en) 2008-10-10 2009-09-25 Energy-conserving multi-channel audio coding

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10440408P 2008-10-10 2008-10-10
US61/104,404 2008-10-10

Publications (1)

Publication Number Publication Date
WO2010042024A1 true WO2010042024A1 (en) 2010-04-15

Family

ID=42100797

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SE2009/051071 WO2010042024A1 (en) 2008-10-10 2009-09-25 Energy conservative multi-channel audio coding

Country Status (5)

Country Link
US (1) US9330671B2 (en)
EP (1) EP2345027B1 (en)
JP (1) JP5608660B2 (en)
CN (1) CN102177542B (en)
WO (1) WO2010042024A1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011153913A1 (en) * 2010-06-10 2011-12-15 华为技术有限公司 Method for generating sideband residual signal and device thereof
WO2012108798A1 (en) * 2011-02-09 2012-08-16 Telefonaktiebolaget L M Ericsson (Publ) Efficient encoding/decoding of audio signals
WO2012121637A1 (en) * 2011-03-04 2012-09-13 Telefonaktiebolaget L M Ericsson (Publ) Post-quantization gain correction in audio coding
JP2013536461A (en) * 2010-07-20 2013-09-19 ファーウェイ テクノロジーズ カンパニー リミテッド Audio signal synthesizer
WO2014116817A3 (en) * 2013-01-24 2014-10-09 Google Inc. Rearrangement and rate allocation for compressing multichannel audio
EP2838086A1 (en) * 2013-07-22 2015-02-18 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. In an reduction of comb filter artifacts in multi-channel downmix with adaptive phase alignment
AU2013301831B2 (en) * 2012-08-10 2016-12-01 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Encoder, decoder, system and method employing a residual concept for parametric audio object coding

Families Citing this family (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5608660B2 (en) * 2008-10-10 2014-10-15 テレフオンアクチーボラゲット エル エム エリクソン（パブル） Energy-conserving multi-channel audio coding
CN102292769B (en) * 2009-02-13 2012-12-19 华为技术有限公司 Stereo encoding method and device
CN105225667B (en) * 2009-03-17 2019-04-05 杜比国际公司 Encoder system, decoder system, coding method and coding/decoding method
US20100324915A1 (en) * 2009-06-23 2010-12-23 Electronic And Telecommunications Research Institute Encoding and decoding apparatuses for high quality multi-channel audio codec
US9838784B2 (en) 2009-12-02 2017-12-05 Knowles Electronics, Llc Directional audio capture
CN102157151B (en) * 2010-02-11 2012-10-03 华为技术有限公司 Encoding method, decoding method, device and system of multichannel signals
BR122020024855B1 (en) * 2010-04-13 2021-03-30 Fraunhofer - Gesellschaft Zur Forderung Der Angewandten Forschung E. V. AUDIO OR VIDEO ENCODER, AUDIO OR VIDEO DECODER AND RELATED METHODS FOR PROCESSING THE AUDIO OR VIDEO SIGNAL OF MULTIPLE CHANNELS USING A VARIABLE FORECAST DIRECTION
US8798290B1 (en) 2010-04-21 2014-08-05 Audience, Inc. Systems and methods for adaptive signal equalization
US9558755B1 (en) 2010-05-20 2017-01-31 Knowles Electronics, Llc Noise suppression assisted automatic speech recognition
US9767822B2 (en) * 2011-02-07 2017-09-19 Qualcomm Incorporated Devices for encoding and decoding a watermarked signal
NO2669468T3 (en) * 2011-05-11 2018-06-02
WO2013188562A2 (en) * 2012-06-12 2013-12-19 Audience, Inc. Bandwidth extension via constrained synthesis
JP6065452B2 (en) 2012-08-14 2017-01-25 富士通株式会社 Data embedding device and method, data extraction device and method, and program
WO2014046916A1 (en) * 2012-09-21 2014-03-27 Dolby Laboratories Licensing Corporation Layered approach to spatial audio coding
EP2757559A1 (en) * 2013-01-22 2014-07-23 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method for spatial audio object coding employing hidden objects for signal mixture manipulation
JP6146069B2 (en) * 2013-03-18 2017-06-14 富士通株式会社 Data embedding device and method, data extraction device and method, and program
US9478224B2 (en) 2013-04-05 2016-10-25 Dolby International Ab Audio processing system
WO2014210284A1 (en) 2013-06-27 2014-12-31 Dolby Laboratories Licensing Corporation Bitstream syntax for spatial voice coding
CN108198564B (en) * 2013-07-01 2021-02-26 华为技术有限公司 Signal encoding and decoding method and apparatus
EP2830051A3 (en) * 2013-07-22 2015-03-04 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Audio encoder, audio decoder, methods and computer program using jointly encoded residual signals
CN117037810A (en) 2013-09-12 2023-11-10 杜比国际公司 Encoding of multichannel audio content
EP2996269A1 (en) 2014-09-09 2016-03-16 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Audio splicing concept
US9978388B2 (en) 2014-09-12 2018-05-22 Knowles Electronics, Llc Systems and methods for restoration of speech components
CN111816194B (en) 2014-10-31 2024-08-09 杜比国际公司 Parametric encoding and decoding of multi-channel audio signals
DE112016000545B4 (en) 2015-01-30 2019-08-22 Knowles Electronics, Llc CONTEXT-RELATED SWITCHING OF MICROPHONES
EP3262638B1 (en) * 2015-02-27 2023-11-08 NewAuro BV Encoding and decoding digital data sets
EP3067886A1 (en) 2015-03-09 2016-09-14 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Audio encoder for encoding a multichannel signal and audio decoder for decoding an encoded audio signal
US12125492B2 (en) * 2015-09-25 2024-10-22 Voiceage Coproration Method and system for decoding left and right channels of a stereo sound signal
EP3208800A1 (en) 2016-02-17 2017-08-23 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method for stereo filing in multichannel coding
CN106023999B (en) * 2016-07-11 2019-06-11 武汉大学 For improving the decoding method and system of three-dimensional audio spatial parameter compression ratio
EP3467824B1 (en) * 2017-10-03 2021-04-21 Dolby Laboratories Licensing Corporation Method and system for inter-channel coding
KR102675420B1 (en) * 2018-04-05 2024-06-17 텔레호낙티에볼라게트 엘엠 에릭슨(피유비엘) Support for generation of comfort noise
WO2020216459A1 (en) * 2019-04-23 2020-10-29 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus, method or computer program for generating an output downmix representation
FR3101741A1 (en) * 2019-10-02 2021-04-09 Orange Determination of corrections to be applied to a multichannel audio signal, associated encoding and decoding
CN111402906B (en) * 2020-03-06 2024-05-14 深圳前海微众银行股份有限公司 Speech decoding method, device, engine and storage medium
JPWO2022009505A1 (en) * 2020-07-07 2022-01-13

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2005059901A1 (en) * 2003-12-19 2005-06-30 Telefonaktiebolaget Lm Ericsson (Publ) Constrained filter encoding of polyphonic signals
WO2005101370A1 (en) * 2004-04-16 2005-10-27 Coding Technologies Ab Apparatus and method for generating a level parameter and apparatus and method for generating a multi-channel representation
WO2006048203A1 (en) * 2004-11-02 2006-05-11 Coding Technologies Ab Methods for improved performance of prediction based multi-channel reconstruction
WO2006108573A1 (en) * 2005-04-15 2006-10-19 Coding Technologies Ab Adaptive residual audio coding
WO2007004830A1 (en) * 2005-06-30 2007-01-11 Lg Electronics Inc. Apparatus for encoding and decoding audio signal and method thereof
WO2007140809A1 (en) * 2006-06-02 2007-12-13 Dolby Sweden Ab Binaural multi-channel decoder in the context of non-energy-conserving upmix rules

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5434948A (en) * 1989-06-15 1995-07-18 British Telecommunications Public Limited Company Polyphonic coding
NL9100173A (en) 1991-02-01 1992-09-01 Philips Nv SUBBAND CODING DEVICE, AND A TRANSMITTER EQUIPPED WITH THE CODING DEVICE.
US5285498A (en) * 1992-03-02 1994-02-08 At&T Bell Laboratories Method and apparatus for coding audio signals based on perceptual model
DE19742655C2 (en) * 1997-09-26 1999-08-05 Fraunhofer Ges Forschung Method and device for coding a discrete-time stereo signal
JP3571890B2 (en) 1997-10-23 2004-09-29 古河電気工業株式会社 Optical fiber core observation device
JP3609623B2 (en) 1998-07-14 2005-01-12 古河電気工業株式会社 Connection loss estimation method of different diameter core fiber connection and connection method of different diameter core fiber
SE0202159D0 (en) * 2001-07-10 2002-07-09 Coding Technologies Sweden Ab Efficientand scalable parametric stereo coding for low bitrate applications
SE0200569L (en) 2002-02-26 2003-10-23 Ericsson Telefon Ab L M Alignment of PM fibers
KR101016251B1 (en) 2002-04-10 2011-02-25 코닌클리케 필립스 일렉트로닉스 엔.브이. Coding of stereo signals
WO2004072956A1 (en) 2003-02-11 2004-08-26 Koninklijke Philips Electronics N.V. Audio coding
US7447317B2 (en) * 2003-10-02 2008-11-04 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V Compatible multi-channel coding/decoding by weighting the downmix channel
DK3561810T3 (en) * 2004-04-05 2023-05-01 Koninklijke Philips Nv METHOD FOR ENCODING LEFT AND RIGHT AUDIO INPUT SIGNALS, CORRESPONDING CODES, DECODERS AND COMPUTER PROGRAM PRODUCT
EP1735774B1 (en) * 2004-04-05 2008-05-14 Koninklijke Philips Electronics N.V. Multi-channel encoder
US7945447B2 (en) * 2004-12-27 2011-05-17 Panasonic Corporation Sound coding device and sound coding method
US7573912B2 (en) 2005-02-22 2009-08-11 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschunng E.V. Near-transparent or transparent multi-channel encoder/decoder scheme
ATE521143T1 (en) * 2005-02-23 2011-09-15 Ericsson Telefon Ab L M ADAPTIVE BIT ALLOCATION FOR MULTI-CHANNEL AUDIO ENCODING
JP5254808B2 (en) * 2006-02-23 2013-08-07 エルジー エレクトロニクス インコーポレイティド Audio signal processing method and apparatus
KR101065704B1 (en) * 2006-09-29 2011-09-19 엘지전자 주식회사 Methods and apparatuses for encoding and decoding object-based audio signals
EP2201566B1 (en) 2007-09-19 2015-11-11 Telefonaktiebolaget LM Ericsson (publ) Joint multi-channel audio encoding/decoding
CN101821799B (en) * 2007-10-17 2012-11-07 弗劳恩霍夫应用研究促进协会 Audio coding using upmix
KR101428487B1 (en) * 2008-07-11 2014-08-08 삼성전자주식회사 Method and apparatus for encoding and decoding multi-channel
EP2144230A1 (en) * 2008-07-11 2010-01-13 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Low bitrate audio encoding/decoding scheme having cascaded switches
JP5608660B2 (en) * 2008-10-10 2014-10-15 テレフオンアクチーボラゲット エル エム エリクソン（パブル） Energy-conserving multi-channel audio coding

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2005059901A1 (en) * 2003-12-19 2005-06-30 Telefonaktiebolaget Lm Ericsson (Publ) Constrained filter encoding of polyphonic signals
WO2005101370A1 (en) * 2004-04-16 2005-10-27 Coding Technologies Ab Apparatus and method for generating a level parameter and apparatus and method for generating a multi-channel representation
WO2006048203A1 (en) * 2004-11-02 2006-05-11 Coding Technologies Ab Methods for improved performance of prediction based multi-channel reconstruction
WO2006108573A1 (en) * 2005-04-15 2006-10-19 Coding Technologies Ab Adaptive residual audio coding
WO2007004830A1 (en) * 2005-06-30 2007-01-11 Lg Electronics Inc. Apparatus for encoding and decoding audio signal and method thereof
WO2007140809A1 (en) * 2006-06-02 2007-12-13 Dolby Sweden Ab Binaural multi-channel decoder in the context of non-energy-conserving upmix rules

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HOTHO G. ET AL: "A Backward-Compatible Multichannel Audio Codec", IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, vol. 16, no. 1, January 2008 (2008-01-01), pages 83 - 93, XP011197126 *

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011153913A1 (en) * 2010-06-10 2011-12-15 华为技术有限公司 Method for generating sideband residual signal and device thereof
US9123329B2 (en) 2010-06-10 2015-09-01 Huawei Technologies Co., Ltd. Method and apparatus for generating sideband residual signal
JP2013536461A (en) * 2010-07-20 2013-09-19 ファーウェイ テクノロジーズ カンパニー リミテッド Audio signal synthesizer
US9082396B2 (en) 2010-07-20 2015-07-14 Huawei Technologies Co., Ltd. Audio signal synthesizer
CN103380455A (en) * 2011-02-09 2013-10-30 瑞典爱立信有限公司 Efficient encoding/decoding of audio signals
CN103380455B (en) * 2011-02-09 2015-06-10 瑞典爱立信有限公司 Efficient encoding/decoding of audio signals
WO2012108798A1 (en) * 2011-02-09 2012-08-16 Telefonaktiebolaget L M Ericsson (Publ) Efficient encoding/decoding of audio signals
US9280980B2 (en) 2011-02-09 2016-03-08 Telefonaktiebolaget L M Ericsson (Publ) Efficient encoding/decoding of audio signals
US10121481B2 (en) 2011-03-04 2018-11-06 Telefonaktiebolaget Lm Ericsson (Publ) Post-quantization gain correction in audio coding
WO2012121637A1 (en) * 2011-03-04 2012-09-13 Telefonaktiebolaget L M Ericsson (Publ) Post-quantization gain correction in audio coding
US11056125B2 (en) 2011-03-04 2021-07-06 Telefonaktiebolaget Lm Ericsson (Publ) Post-quantization gain correction in audio coding
US10460739B2 (en) 2011-03-04 2019-10-29 Telefonaktiebolaget Lm Ericsson (Publ) Post-quantization gain correction in audio coding
US10818301B2 (en) 2012-08-10 2020-10-27 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Encoder, decoder, system and method employing a residual concept for parametric audio object coding
AU2013301831B2 (en) * 2012-08-10 2016-12-01 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Encoder, decoder, system and method employing a residual concept for parametric audio object coding
WO2014116817A3 (en) * 2013-01-24 2014-10-09 Google Inc. Rearrangement and rate allocation for compressing multichannel audio
US10360918B2 (en) 2013-07-22 2019-07-23 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Reduction of comb filter artifacts in multi-channel downmix with adaptive phase alignment
US20160133262A1 (en) * 2013-07-22 2016-05-12 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Reduction of comb filter artifacts in multi-channel downmix with adaptive phase alignment
CN105518775A (en) * 2013-07-22 2016-04-20 弗朗霍夫应用科学研究促进协会 In reduction of comb filter artifacts in multi-channel downmix with adaptive phase alignment
CN111862997A (en) * 2013-07-22 2020-10-30 弗朗霍夫应用科学研究促进协会 Artifact cancellation for multi-channel downmix comb filters using adaptive phase alignment
US10937435B2 (en) 2013-07-22 2021-03-02 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Reduction of comb filter artifacts in multi-channel downmix with adaptive phase alignment
EP2838086A1 (en) * 2013-07-22 2015-02-18 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. In an reduction of comb filter artifacts in multi-channel downmix with adaptive phase alignment

Also Published As

Publication number Publication date
EP2345027A4 (en) 2016-10-12
JP2012505429A (en) 2012-03-01
EP2345027B1 (en) 2018-04-18
CN102177542A (en) 2011-09-07
EP2345027A1 (en) 2011-07-20
US9330671B2 (en) 2016-05-03
US20110224994A1 (en) 2011-09-15
JP5608660B2 (en) 2014-10-15
CN102177542B (en) 2013-01-09

Similar Documents

Publication Publication Date Title
US9330671B2 (en) Energy conservative multi-channel audio coding
JP7140817B2 (en) Method and system using long-term correlation difference between left and right channels for time-domain downmixing of stereo audio signals into primary and secondary channels
US8218775B2 (en) Joint enhancement of multi-channel audio
US8457319B2 (en) Stereo encoding device, stereo decoding device, and stereo encoding method
EP2981956B1 (en) Audio processing system
JP7280306B2 (en) Apparatus and method for MDCT M/S stereo with comprehensive ILD with improved mid/side determination
EP1806737A1 (en) Sound encoder and sound encoding method
US9530422B2 (en) Bitstream syntax for spatial voice coding
US7725324B2 (en) Constrained filter encoding of polyphonic signals
US12125492B2 (en) Method and system for decoding left and right channels of a stereo sound signal

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200980140139.3

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09819478

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 1873/CHENP/2011

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2011530989

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 13122880

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 2009819478

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE