RU2803451C2 - Encoding and decoding parameters - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: processing audio data. The technical result is achieved due to the fact that the target version (CyR) of the covariance information (Cy) of the original signal is restored based on the estimated version () of the original covariance information (Cy), while the estimated version () of the original covariance information (Cy) is reported to the mentioned number of synthesis channels, and the estimated version () of the original covariance information is obtained from the covariance information (Cx) of the downmix signal, with the estimated version () of the original covariance information is obtained by applying to the covariance information (Cx) of the downmix signal an estimation rule (Q), which is or is associated with a prototype rule for calculating the prototype signal.

EFFECT: enabling encoding and decoding of audio signal parameters with low bit rates to describe multi-channel content between encoder and decoder without loss of output sound quality.

74 cl, 27 dwg, 2 tbl

Description

1. Introduction

Several examples of encoding and decoding technology are disclosed herein. In particular, the invention serves to encode and decode multi-channel audio content at low bit rates, for example, using the DirAC infrastructure. This method allows you to obtain a high-quality output signal while using low bit rates. It can be used in a variety of applications including artistic production, communications and virtual reality.

1.1. Documents from the prior art

This section briefly describes the state of the art.

1.1.1. Discrete encoding of multi-channel content

The simplest approach for encoding and transmitting multichannel content is to quantify and encode the waveforms of the multichannel audio signal directly, without prior processing or assumptions. Although this method works perfectly in theory, it has one major drawback, which is the consumption of bits required to encode multi-channel content. Therefore, other methods to be described (as well as the proposed invention) are so-called "parametric approaches" because they use metaparameters to describe and transmit the multi-channel audio signal instead of the original multi-channel audio signal itself.

1.1.2. MPEG surround sound standard

The MPEG surround sound standard is an ISO/MPEG standard, completed in 2006, for parametric encoding of multi-channel audio [1]. This method is based mainly on two sets of parameters:

- inter-channel coherence (ICC), which describes the coherence between each channel of a given multi-channel audio signal.

- channel level difference (CLD), which corresponds to the level difference between two input channels of a multi-channel audio signal.

One feature of the MPEG surround sound standard is the use of so-called “tree structures”, these structures allowing “two input channels to be described by single output channels” (quoted from [1]).

As an example, below is a diagram of a 5.1 multichannel audio encoder using the MPEG surround sound standard. In this drawing, six input channels (labeled "L", "LS", "R", " _RS ", "C" and "LFE" in the drawing) are processed sequentially through a tree structure (labeled "R_OTT" in the drawing "). Each of these tree structures must form the set of parameters, ICC and CLD, specified above, as well as a residual signal, which is processed again through another tree structure and generates another set of parameters. Once the end of the tree is reached, the various parameters previously calculated are passed to the decoder, as is the downmixed signal. These elements are used by the decoder to generate a multi-channel output signal, the decoder processing being essentially the inverse of the tree structure used by the encoder.

The main advantage of the MPEG surround sound standard is based on the use of this structure and the above parameters. However, one of the disadvantages of the MPEG surround sound standard is the lack of flexibility due to the tree structure. Also, due to processing characteristics, a decrease in quality may occur in some specific elements.

See, inter alia, FIG. 7, which shows an overview of the MPEG surround encoder for a 5.1 signal extracted from [1].

1.2. Directional Audio Coding

Directional Audio Coding (abbreviated as "DirAC") [2] is also a parametric method for reproducing spatial audio data, developed by Ville Pulkki of Aalto University in Finland. DirAC is based on frequency band processing, which uses two sets of parameters to describe surround sound systems:

- Direction of Arrival (DoA), which is the angle in degrees that describes the direction of arrival of the predominant sound in an audio signal.

- Scattering, which is a value between 0 and 1 that describes how "scattered" the sound is. If the value is 0, the sound is non-diffuse and can be assimilated as a point source coming from an exact corner; if the value is 1, the sound is completely diffuse and is assumed to come from "every" corner.

To synthesize output signals, DirAC involves decomposing the synthesis into diffuse and non-diffuse parts, diffuse sound synthesis aims to shape the perception of ambient sound, while direct sound synthesis aims to shape the dominant sound.

While DirAC provides good quality output signals, it has one major drawback: it is not designed for multi-channel audio signals. Therefore, the DoA and dispersion parameters are not suitable for describing the multi-channel input audio signal, and as a result, the quality of the output signal is affected.

1.3. Binaural coding by cues

Binaural cue coding (BCC) [3] is a parametric approach developed by Christof Faller. This method is based on a set of parameters similar to those described for the MPEG surround sound standard (see also 1.1.2), namely:

- inter-channel level difference (ICLD), which is a measure of the energy ratio between two channels of a multi-channel input signal.

- inter-channel time difference (ICTD), which is a measure of the delay between two channels of a multi-channel input signal.

- inter-channel correlation (ICC), which is a measure of the correlation between two channels of a multi-channel input signal.

The BCC approach has almost the same characteristics in terms of calculating the parameters to be transmitted compared to the new invention described below, but it lacks the flexibility and scalability of the transmitted parameters.

1.4. Spatial coding of audio objects according to the MPEG standard

What should be noted here is simply the spatial coding of audio objects [4]. It is an MPEG standard for encoding so-called audio objects, which are related to a certain extent to a multi-channel signal. It uses similar parameters to the MPEG surround sound standard.

1.5. Background/disadvantages of the prior art

1.5. Prerequisites

1.5.1.1. Using the DirAC infrastructure

One aspect of the invention that should be noted is that the present invention must conform to the DirAC framework. However, it should also be noted upfront that DirAC parameters are not suitable for multi-channel audio. Some further clarification is needed on this topic.

The original DirAC processing uses either microphone signals or ambiophonic signals. From these signals, parameters, namely direction of arrival (DoA) and scattering, are calculated.

One early approach that has been tested for using DirAC with multi-channel audio signals is to convert multi-channel signals into ambiophonic content using the method proposed by Ville Pulkki described in [5]. Then, after these ambiophonic signals are extracted from the multi-channel audio signals, regular DirAC processing is performed using DoA and diffusion. The result of this first attempt is that the quality and spatial signature of the multi-channel output signal is degraded and does not meet the requirements of the target application.

Therefore, the main rationale behind this new invention is to use a set of parameters that effectively describes a multi-channel signal, and also to use the DirAC infrastructure; Additional explanations are provided in section 1.1.2.

1.5.1.2. Creating a system that operates at low bit rates

One of the objects and purposes of the present invention is to provide an approach that enables low bit rate applications. This requires finding the optimal data set to describe the multi-channel content between the encoder and decoder. This also requires finding the optimal compromise in terms of the number of transmitted parameters and the output quality.

1.5.1.3. Creating a flexible system

Another important object of the present invention is to provide a flexible system that can support any multi-channel audio format for playback on any speaker configuration. The output quality should not be compromised depending on the input configuration.

1.5.2. Disadvantages of the Prior Art

The prior art mentioned above has several disadvantages, which are listed in the table below.

Flaw Considered state of the art A comment Inappropriate bit rates Discrete encoding of multi-channel content Direct encoding of multi-channel content results in bit rates that are too high for the specified requirements and target applications. Inappropriate parameters/descriptors Previously created DirAC The previously created DirAC method uses scattering and DoA as parameter descriptions; it turns out that these parameters are not suitable for describing a multi-channel audio signal Lack of flexibility in approach MPEG surround sound standard
BCC MPEG and BCC surround sound standard are not flexible enough to meet target application requirements

2. Description of the invention

2.1. Disclosure of the Invention

According to an aspect, an audio data synthesis apparatus (encoder) is provided for generating a synthesis signal from a downmix signal, the synthesis signal having a number of synthesis channels, the audio data synthesis apparatus comprising:

- an input interface configured to receive a downmix signal, wherein the downmix signal has a number of downmix channels and auxiliary information, wherein the auxiliary information includes link layer and correlation information of the original signal, wherein the original signal has a number of original channels; And

- a synthesis processor configured to generate, according to at least one mixing rule, a synthesis signal using:

- channel level information and correlation of the original signal; And

- covariance information associated with the downmix signal.

The audio data synthesis device may contain:

- a prototype signal calculation module configured to calculate a prototype signal from the downmix signal, wherein the prototype signal has said number of synthesis channels;

- a mixing rule calculation module configured to calculate at least one mixing rule using:

- channel level information and correlation of the original signal; And

- covariance information associated with the downmix signal;

- wherein the synthesis processor is configured to generate a synthesis signal using a prototype signal and at least one mixing rule.

The audio data synthesis apparatus may be configured to recover target covariance information of the original signal.

The audio data synthesis apparatus may be configured to recover target covariance information adapted to the number of channels of the synthesis signal.

An audio data synthesis apparatus may be configured to recover covariance information adapted to a number of synthesis signal channels by assigning groups of source channels to single synthesis channels, or vice versa, such that the recovered target covariance information is reported to a number of synthesis signal channels.

An audio data synthesis apparatus may be configured to recover covariance information adapted to a number of synthesis signal channels by generating target covariance information for said number of source channels and then applying a downmixing rule or an upmixing rule and energy compensation to achieve the target covariance for the synthesis channels. .

The audio data synthesis apparatus may be configured to recover a target version of the covariance information based on an estimated version of the source covariance information, wherein the estimated version of the source covariance information is reported to a number of synthesis channels or a number of source channels.

The audio data synthesis apparatus may be configured to obtain an estimated version of the original covariance information from the covariance information associated with the downmix signal.

The audio synthesis apparatus may be configured to obtain an estimated version of the original covariance information by applying to the covariance information associated with the downmix signal an evaluation rule associated with a prototype rule for calculating the prototype signal.

The audio data synthesis apparatus may be configured to normalize for at least one pair of channels the estimated version ( ) of the original covariance information (Cy) into the square roots of the channel levels of a pair of channels.

The audio data synthesis apparatus may be configured to interpret a matrix with a normalized estimated version of the original covariance information.

The audio data synthesis apparatus may be configured to populate the matrix by inserting entries obtained in the bit stream auxiliary information.

The audio data synthesis apparatus may be configured to denormalize the matrix by scaling the estimated version of the original covariance information by the square root of the channel levels forming the channel pair.

The audio synthesis apparatus may be configured to perform extraction of the downmix signal from the auxiliary information, wherein the audio synthesis apparatus is further configured to recover a target version of the covariance information by means of an estimated version of the original link layer information and correlation from:

- covariance information for at least one first channel or pair of channels; And

- link layer and correlation information for at least one second channel or pair of channels.

The audio synthesis apparatus may be configured to prefer link layer and correlation information describing a channel or pair of channels obtained from the bit stream auxiliary information rather than covariance information recovered from a downmix signal for the same channel or pair of channels.

The reconstructed target version of the original covariance information can be understood as a description of the energy relationship between a pair of channels based at least in part on the levels associated with each channel of the pair of channels.

An audio data synthesis apparatus may be configured to obtain a frequency domain (FD) version of the downmix signal, the FD version of the downmix signal into frequency bands or groups of frequency bands, with different link layer and correlation information associated with different frequency bands or groups of bands frequencies,

- wherein the audio data synthesis device is configured to operate differently for different frequency bands or groups of frequency bands so as to obtain different mixing rules for different frequency bands or groups of frequency bands.

The downmix signal is divided into intervals, with different link layer and correlation information associated with different intervals, and the audio data synthesis apparatus is configured to operate differently for different intervals so as to obtain different mixing rules for different intervals.

The downmix signal is divided into frames, and each frame is divided into intervals, and the audio data synthesis device is configured to have the presence and position of a transition portion in one frame signaled as being in the same transition interval:

- associating current link layer information and correlation with the transition interval and/or with intervals after the transition interval of the frame; And

- association with the frame interval preceding the transition interval, link layer information and correlation of the previous interval.

The audio data synthesis apparatus may be configured to select a prototype rule configured to calculate a prototype signal based on the number of synthesis channels.

The audio data synthesis apparatus may be configured to select a prototype rule from a plurality of previously stored prototype rules.

The audio data synthesis apparatus may be configured to specify a prototype rule based on manual selection.

A prototype rule may be based on or include a matrix with a first dimension and a second dimension, wherein the first dimension is associated with the number of downmix channels and the second dimension is associated with the number of synthesis channels.

The audio synthesis apparatus may be configured to operate at a bit rate equal to or lower than 160 Kbps.

The audio data synthesis apparatus may further comprise an entropy decoder for obtaining a downmix signal with auxiliary information.

The audio data synthesis device further includes a decorrelation module for reducing the amount of correlation between different channels.

The prototype signal can be directly passed to the synthesis processor without performing decorrelation.

At least one of the link layer information and the original signal correlation of the at least one mixing rule and the covariance information associated with the downmix signal is in the form of a matrix.

Supporting information includes source channel identification data:

wherein the audio data synthesis apparatus may be further configured to compute at least one mixing rule using at least one of link layer information and source signal correlation, covariance information associated with the downmix signal, source channel identification data, and identification data synthesis channels.

The audio data synthesis apparatus may be configured to compute at least one mixing rule via singular value decomposition (SVD).

The downmix signal may be divided into frames, wherein the audio data synthesis device is configured to smooth a received parameter or an estimated or reconstructed value or a mixing matrix using linear combination with a parameter with either an estimated or reconstructed value or a mixing matrix obtained for the previous frame.

The audio data synthesis apparatus may be configured to deactivate smoothing of a received parameter of either an estimated or reconstructed value or a mixing matrix when the presence and/or position of a transition portion in one frame is signaled.

The downmix signal may be divided into frames, and the frames are divided into intervals, wherein link layer and correlation information of the original signal is obtained from the auxiliary information of the bitstream on a frame-by-frame basis, and the audio data synthesis apparatus is configured to use a mixing matrix (or mixing rule) for the current frame. obtained by scaling, the mixing matrix (or mixing rule) calculated for the current frame, by a factor increasing along subsequent intervals of the current frame, and by adding the mixing matrix (or mixing rule) used for the previous frame in a version scaled by a decreasing factor along subsequent intervals of the current frame.

The number of synthesis channels can exceed the number of original channels. The number of synthesis channels may be less than the number of original channels. The number of synthesis channels and the number of source channels can exceed the number of downmix channels.

At least one or all of the number of synthesis channels, the number of source channels, and the number of downmix channels constitute the plural.

The at least one mixing rule may include a first mixing matrix and a second mixing matrix, wherein the audio data synthesis apparatus comprises:

- the first path, including:

- a processing unit for the first mixing matrices, configured to synthesize the first component of the synthesis signal according to the first mixing matrix calculated from:

- a covariance matrix associated with the synthesis signal, wherein the covariance matrix is recovered from link layer and correlation information; And

- covariance matrix associated with the downmix signal,

- a second path for synthesizing a second component of the synthesis signal, wherein the second component is a residual component, wherein the second path includes:

- a prototype signal processing unit configured to upmix a downmix signal from said number of downmix channels into said number of synthesis channels;

- a decorrelator configured to decorrelate an up-mixed prototype signal;

- a second mixing matrix processing unit configured to synthesize a second component of the synthesis signal according to a second mixing matrix from a decorrelated version of the downmixing signal, wherein the second mixing matrix is a residual mixing matrix,

- wherein the audio data synthesis device is configured to estimate the second mixing matrix from:

- residual covariance matrix provided by the processing unit for the first mixing matrices; And

- estimating the covariance matrix of decorrelated prototype signals obtained from the covariance matrix associated with the downmix signal,

- wherein the audio data synthesis device further comprises a summing unit for summing the first component of the synthesis signal with the second component of the synthesis signal.

In accordance with an aspect, an audio data synthesis apparatus may be provided for generating a synthesis signal from a downmix signal having a number of downmix channels, wherein the synthesis signal has a number of synthesis channels, wherein the downmix signal is a downmixed version of the original signal, having a certain number of source channels, and the audio data synthesis device contains:

- the first path, including:

- a processing unit for the first mixing matrices, configured to synthesize the first component of the synthesis signal according to the first mixing matrix calculated from:

- covariance matrix associated with the synthesis signal; And

- a covariance matrix associated with the downmix signal.

- a second path for synthesizing a second component of the synthesis signal, wherein the second component is a residual component, wherein the second path includes:

- a prototype signal processing unit configured to upmix a downmix signal from said number of downmix channels into said number of synthesis channels;

- a decorrelator configured to decorrelate an up-mixed prototype signal;

- a second mixing matrix processing unit configured to synthesize a second component of the synthesis signal according to a second mixing matrix from a decorrelated version of the downmixing signal, wherein the second mixing matrix is a residual mixing matrix,

- wherein the audio data synthesis device is configured to calculate the second mixing matrix from:

- residual covariance matrix provided by the processing unit for the first mixing matrices; And

- estimating the covariance matrix of decorrelated prototype signals obtained from the covariance matrix associated with the downmix signal,

- wherein the audio data synthesis device further comprises a summing unit for summing the first component of the synthesis signal with the second component of the synthesis signal.

The residual covariance matrix is obtained by subtracting, from the covariance matrix associated with the synthesis signal, the matrix obtained by applying the first mixing matrix to the covariance matrix associated with the downmix signal.

The audio data synthesis apparatus may be configured to determine a second mixing matrix from:

- a second matrix, which is obtained by decomposing the residual covariance matrix associated with the synthesis signal;

- a first matrix that is the inverse or regularized inverse of the diagonal matrix obtained from estimating the covariance matrix of the decorrelated prototype signals.

The diagonal matrix can be obtained by applying a square root function to the principal diagonal elements of the covariance matrix of the decorrelated prototype signals.

The second matrix may be obtained by singular value decomposition (SVD) applied to the residual covariance matrix associated with the synthesis signal.

The audio data synthesis apparatus may be configured to determine a second mixing matrix by multiplying the second matrix by the inverse or regularized inverse of a diagonal matrix obtained from estimating the covariance matrix of the decorrelated prototype signals and the third matrix.

The audio data synthesis apparatus may be configured to obtain a third matrix by means of an SVP applied to a matrix obtained from a normalized version of the covariance matrix of the decorrelated prototype signals, the normalization being performed with respect to the main diagonal of the residual covariance matrix and the diagonal matrix and the second matrix.

The audio data synthesis apparatus may be configured to determine a first mixing matrix from a second matrix and an inverse or regularized inversion of the second matrix,

- wherein the second matrix is obtained by decomposing the covariance matrix associated with the downmix signal, and

- the second matrix is obtained by decomposing the reconstructed target covariance matrix associated with the downmix signal.

The audio data synthesis apparatus may be configured to estimate a covariance matrix of decorrelated prototype signals from diagonal entries of the matrix obtained by applying to the covariance matrix associated with the downmix signal a prototype rule used in a prototype block for upmixing a downmix signal from a number of downmix channels. mixing into the mentioned number of synthesis channels.

Frequency bands are aggregated with each other into groups of aggregated frequency bands, wherein information regarding the groups of aggregated frequency bands is carried in the bitstream auxiliary information, wherein link layer and source signal correlation information is provided per each group of frequency bands so as to calculate one and the same at least one mixing matrix for different frequency bands of the same aggregate group of frequency bands.

In accordance with an aspect, an audio encoder may be provided for generating a downmix signal from an original signal, wherein the original signal has a plurality of original channels, wherein the downmix signal has a number of downmix channels, wherein the audio encoder comprises:

- a parameter estimation module configured to evaluate link level information and correlate the original signal, and

- a bitstream recording module for encoding the downmix signal into a bitstream, such that the downmix signal is encoded into the bitstream such that it has auxiliary information including link layer and correlation information of the original signal.

The audio encoder may be configured to provide link layer information and correlation of the original signal as normalized values.

The link layer and source signal correlation information encoded in the auxiliary information represents at least link layer information associated with all source channels.

The link layer and source signal correlation information encoded in the auxiliary information represents at least correlation information describing the power relationships between at least one pair of different source channels, but not all of the source channels.

The link layer and source signal correlation information includes at least one coherence value describing coherence between two channels of a pair of source channels.

The coherence value may normalize. The coherence value can be:

,

- Where is the covariance between channels i and j, while And respectively, are the levels associated with channels i and j.

The link layer and correlation information of the original signal includes at least one inter-channel level difference (ICLD).

At least one ICLD may be provided as a logarithmic value. At least one ICLD may be normalized. The ICLD may be as follows:

- Where:

- X _i is the ICLD for channel i.

- Pi is the power of the current channel i

- P _dmx,i is a linear combination of the covariance information values of the downmix signal.

The audio encoder may be configured to select whether or not to encode at least a portion of the link layer and correlation information of the original signal based on the state information, so as to include in the auxiliary information an increased amount of link layer and correlation information in the case of comparatively less volume of working data.

The audio encoder may be configured to select how much of the link layer and correlation information of the original signal is to be encoded in the ancillary information, based on the per-channel metrics, so as to include the link layer and correlation information associated with the more sensitive metrics in the ancillary information. .

The link layer and source signal correlation information may be in the form of matrix entries.

The matrix may be symmetric or Hermitian, with link layer and correlation information entries provided for all or some of the entries on the diagonal of the matrix and/or for less than half of the non-diagonal elements of the matrix.

The bitstream recording module may be configured to encode identification data of at least one channel.

The original signal or its processed version can be divided into many subsequent frames of equal duration.

The audio encoder may be configured to encode link layer information and correlate the original signal specific to each frame into auxiliary information.

The audio encoder may be configured to encode the same link layer information into the auxiliary information and correlate the original signal collectively associated with a plurality of successive frames.

The audio encoder may be configured to select a number of consecutive frames in which the same link layer and source signal correlation information may be selected such that:

- A comparatively higher bit rate or larger amount of operating data implies an increase in the number of consecutive frames with which the same link layer and correlation information of the source signal is associated, and vice versa.

The audio encoder may be configured to reduce the number of consecutive frames with which the same link layer information is associated and correlate the original signal to detect a transient portion.

Each frame can be divided into an integer number of consecutive intervals.

The audio encoder may be configured to estimate link layer and correlation information for each slot and encode in the auxiliary information the sum or average or other specified linear combination of the link layer information and correlation estimated for different slots.

The audio encoder may be configured to perform transient analysis on the time domain version of the frame to determine the presence of a transient portion in the frame.

The audio decoder may be configured to determine in which frame interval there is a transition portion, and:

- encoding link layer information and correlating the original signal associated with the interval in which there is a transition part, and/or with subsequent intervals in the frame,

- without encoding channel level information and correlating the original signal associated with the intervals preceding the transition part.

The audio encoder may be configured to signal in the ancillary information the presence of a transition portion present in one frame interval.

The audio encoder may be configured to signal in the ancillary information in which frame interval there is a transition portion.

The audio encoder may be configured to estimate link layer and correlation information of an original signal associated with a plurality of frame intervals and sum or average them, or linearly combine them, to obtain link layer and correlation information associated with the frame.

The source signal may be converted to a frequency domain signal, wherein the audio encoder is configured to encode , link layer information and correlate the source signal across frequency bands in auxiliary information.

The audio encoder may be configured to aggregate a number of frequency bands of the original signal into a more reduced number of frequency bands so as to encode link layer information and correlations of the original signal across the aggregated frequency bands in the auxiliary information.

The audio encoder may be configured, if a transient is detected in the frame, to further aggregate frequency bands such that:

- the number of frequency bands is reduced; and/or

- the width of at least one frequency band is increased by aggregation with another frequency band.

The audio encoder may be further configured to encode into the bitstream at least one link layer information and correlation of one frequency band incrementally relative to the previously encoded link layer information and correlation.

The audio encoder may be configured to encode in the bitstream side information a partial version of the link layer information and correlation with respect to the link layer information and the correlation estimated by the estimator.

The audio encoder may be configured to adaptively select, from all the link layer and correlation information estimated by the estimator, the selected information to be encoded in the bit stream auxiliary information, such that the link layer and/or correlation information for the remaining unselected information is estimated via the evaluation module, not coded.

The audio encoder may be configured to recover link layer and correlation information from selected link layer and correlation information, thereby simulating evaluation at the decoder of unselected link layer and correlation information, and calculating error information between:

- unselected link layer information and correlation estimated by the encoder; And

- unselected link layer and correlation information recovered by estimation modeling, in the decoder, uncoded link layer and correlation information; And

- in such a way as to distinguish, based on the calculated error information:

- properly restored link layer and correlation information;

- from data link level and correlation that cannot be properly restored,

- in such a way as to make decisions regarding:

- selection of properly unrecoverable link layer and correlation information to be encoded in the bit stream auxiliary information; And

- failure to select properly recoverable link layer and correlation information, thereby not encoding properly recoverable link layer and correlation information into the bitstream auxiliary information.

The link layer and correlation information may be indexed according to a given ordering, wherein the encoder is configured to signal in the auxiliary information a stream of bits of indices associated with the given ordering, the indices indicating which of the link layer and correlation information is encoded. Indexes are provided via a bitmap. Indexes can be specified according to a combinatorial number system that associates a one-dimensional index with matrix entries.

The audio encoder may be configured to select between:

- adaptive provision of link layer information and correlation, in which indices associated with a given ordering are encoded in the auxiliary information of the bit stream; And

- fixedly providing link layer and correlation information such that the link layer and correlation information that is encoded is specified and ordered according to a given fixed ordering without providing indexes.

The audio encoder may be configured to signal in the bitstream side information whether link layer and correlation information is provided according to adaptive provision or according to fixed provision.

The audio encoder may be further configured to encode current link layer and correlation information into the bitstream as an increment relative to previous link layer and correlation information.

The audio encoder may be further configured to generate a downmix signal according to the static downmix.

According to an aspect, there is provided a method for generating a synthesis signal from a downmix signal, the synthesis signal having a number of synthesis channels, the method comprising:

- receiving a downmix signal, wherein the downmix signal has a number of downmix channels and auxiliary information, wherein the auxiliary information includes:

- link level information and correlation of the source signal, wherein the source signal has a certain number of source channels;

- generating a synthesis signal using channel level information (220) and correlation of the original signal and covariance information associated with the signal.

The method may contain:

- calculating a prototype signal from the downmix signal, wherein the prototype signal has a certain number of synthesis channels;

- calculating a mixing rule using link layer information and correlation of the original signal and covariance information associated with the downmixing signal; And

- generation of a synthesis signal using a prototype signal and mixing rules.

According to an aspect, there is provided a method for generating a downmix signal from an original signal, wherein the original signal has a number of original channels, wherein the downmix signal has a number of downmix channels, the method comprising:

- assessment of channel level information and correlation of the original signal,

- encoding the downmix signal into a bitstream, such that the downmix signal is encoded into the bitstream such that it has auxiliary information including link layer and correlation information of the original signal.

In accordance with an aspect, there is provided a method for generating a synthesis signal from a downmix signal having a number of downmix channels, wherein the synthesis signal has a number of synthesis channels, wherein the downmix signal is a downmixed version of an original signal having a number of original channels. , and the method contains the following phases:

- the first phase, which includes:

- synthesis of the first component of the synthesis signal according to the first mixing matrix calculated from:

- covariance matrix associated with the synthesis signal; And

- a covariance matrix associated with the downmix signal.

- a second phase for synthesizing a second component of the synthesis signal, wherein the second component is a residual component, wherein the second phase includes:

- a prototype signal processing step that upmixes the downmix signal from said number of downmix channels into said number of synthesis channels;

- decorrelation stage, decorrelating the upmixed prototype signal;

- a second mixing matrix processing step that synthesizes a second component of the synthesis signal according to a second mixing matrix from a decorrelated version of the downmixing signal, wherein the second mixing matrix is a residual mixing matrix,

- in this case, the method calculates the second mixing matrix from:

- a residual covariance matrix provided by the first mixing matrix processing step; And

- estimating the covariance matrix of decorrelated prototype signals obtained from the covariance matrix associated with the downmix signal,

- wherein the method further comprises a summing step summing the first component of the synthesis signal with the second component of the synthesis signal, thereby obtaining a synthesis signal.

According to an aspect, an audio data synthesis apparatus is provided for generating a synthesis signal from the downmix signal, the synthesis signal having a number of synthesis channels, the number of synthesis channels being greater than one or greater than two, the audio data synthesis apparatus comprising at least one of:

- an input interface configured to receive a downmix signal, wherein the downmix signal has at least one downmix channel and auxiliary information, wherein the auxiliary information includes at least one of:

- link level information and correlation of the source signal, wherein the source signal has a certain number of source channels, and the number of source channels is more than one or more than two;

- such a part as a prototype signal calculation module (for example, “prototype signal calculation”), configured to calculate a prototype signal from a downmix signal, wherein the prototype signal has a number of synthesis channels;

- such a part as a mixing rules calculation module (for example, “parameter recovery”), configured to calculate one (or more) mixing rules (for example, a mixing matrix) using link level information and correlation of the original signal, covariance information associated with downmix signal; And

- such a part as a synthesis processor, for example, a “synthesis engine”), configured to generate a synthesis signal using a prototype signal and a mixing rule.

The number of synthesis channels can exceed the number of original channels. Alternatively, the number of synthesis channels may be less than the number of source channels.

The audio data synthesis apparatus (and, in particular, in some aspects, the mixing rules computation module) may be configured to reconstruct a target version of the original link layer and correlation information.

The audio synthesis apparatus (and, in particular, in some aspects, the mixing rule calculation module) may be configured to reconstruct a target version of the original link layer information and correlation adapted to the number of channels of the synthesis signal.

The audio data synthesis apparatus (and, in particular, in some aspects, the mixing rule calculation module) may be configured to recover a target version of the original link layer and correlation information based on the estimated version of the original link layer and correlation information.

The audio data synthesis apparatus (and, in particular, in some aspects, the mixing rule calculation module) may be configured to obtain an estimated version of the original link layer information and correlation from the covariance information associated with the downmix signal.

An audio data synthesis apparatus (and, in particular, in some aspects, a mixing rule calculation module) may be configured to obtain an estimated version of the original link layer information and correlation by applying to the covariance information associated with the downmix signal an evaluation rule associated with the prototype the rule used by the prototype signal computation module (for example, "prototype signal computation") to calculate the prototype signal.

The audio synthesis apparatus (and, in particular, in some aspects, the mixing rules computation module) may be configured to extract a downmix signal from the auxiliary information:

- covariance information associated with the downmix signal describing the level of the first channels or the energy relationship between a pair of channels in the downmix signal; And

- link level information and correlation of the original signal describing the level of the first channel or the energy relationship between a pair of channels in the original signal,

- so as to reconstruct the target version of the original link layer and correlation information by using at least one of:

- covariance information of the original channel for at least one first channel or pair of channels; And

- link layer and correlation information describing at least one second channel or pair of channels.

The audio synthesis apparatus (and, in particular, in some aspects, the mixing rules computation module) may be configured to favor link layer and correlation information describing a channel or pair of channels rather than source channel covariance information for the same channel or pair channels.

The reconstructed target version of the original link layer information and correlation describing the energy relationship between a pair of channels is based at least in part on the layers associated with each channel of the channel pair.

The downmix signal may be divided into frequency bands or groups of frequency bands: different link layer and correlation information may be associated with different frequency bands or groups of frequency bands; the synthesizer (the prototype signal calculator and, in particular, in some aspects, at least one of the mixing rule calculation module and the synthesis processor) operates differently for different frequency bands or groups of frequency bands to obtain different mixing rules for different frequency bands or groups of frequency bands.

The downmix signal may be divided into slots, with various link layer and correlation information associated with different slots, and at least one of the synthesizer components (e.g., a prototype signal computer, a mixing rule computer, a synthesis processor, or other synthesizer elements) is operating differently for different intervals to obtain different mixing rules for different intervals.

A synthesizer (eg, a prototype signal computation module) may be configured to select a prototype rule configured to compute a prototype signal based on the number of synthesis channels.

A synthesizer (eg, a prototype signal computation module) may be configured to select a prototype rule from a plurality of pre-stored prototype rules.

A synthesizer (eg, a prototype signal computation module) may be configured to determine a prototype rule based on manual selection.

A synthesizer (eg, a prototype signal processor) may include a matrix of first and second dimensions, wherein the first dimension is associated with the number of downmix channels and the second dimension is associated with the number of synthesis channels.

An audio data synthesis apparatus (eg, a prototype signal computation module) may be configured to operate at a bit rate equal to or less than 64 Kbps or 160 Kbps.

The auxiliary information may include identification data of the original channels (eg, L, R, C, etc.).

An audio data synthesis apparatus (and, in particular, in some aspects, a mixing rule computation module) may be configured to compute (e.g., "parameter recovery") a mixing rule (e.g., a mixing matrix) using link layer information and correlation of the original signal. covariance information associated with the downmix signal; and source channel identities and synthesis channel identities.

An audio synthesis apparatus may select (e.g., by selecting, e.g., manually selecting either through preselection or automatically, e.g., by recognizing the number of speakers), for the synthesis signal, the number of channels independent of at least one of link layer and correlation information the original signal in the auxiliary information.

The audio synthesis apparatus may select different prototype rules for different choices in some examples. The mixing rule calculation module may be configured to calculate the mixing rule.

According to an aspect, there is provided a method for generating a synthesis signal from a downmix signal, wherein the synthesis signal has a number of synthesis channels, wherein the number of synthesis channels is greater than one or greater than two, the method comprising:

- receiving a downmix signal, wherein the downmix signal has at least one downmix channel and auxiliary information, wherein the auxiliary information includes:

- link level information and correlation of the source signal, wherein the source signal has a certain number of source channels, and the number of source channels is more than one or more than two;

- calculating a prototype signal from the downmix signal, wherein the prototype signal has a certain number of synthesis channels;

- calculating a mixing rule using link level information and correlation of the original signal, covariance information associated with the downmixing signal; And

- generation of a synthesis signal using a prototype signal and mixing rules (for example, rules).

In accordance with an aspect, an audio encoder is provided for generating a downmix signal from a source signal (e.g., y), wherein the source signal has at least two channels, wherein the downmix signal has at least one downmix channel, wherein the audio encoder comprises at least one of:

- parameter estimation module, configured to evaluate channel level information and correlate the original signal,

- a bitstream recording module for encoding the downmix signal into a bitstream such that the downmix signal is encoded into the bitstream such that it has auxiliary information including link layer and correlation information of the original signal.

The link layer and source signal correlation information encoded in the auxiliary information represents link layer information associated with not all channels of the source signal.

The link layer and correlation information of the original signal encoded in the auxiliary information represents correlation information describing the power relationships between at least one pair of different channels in the original signal, but not with all channels of the original signal.

The link layer and source signal correlation information may include at least one coherence value describing the coherence between two channels of a pair of channels.

The link layer and correlation information of the original signal may include at least one inter-channel level difference (ICLD) between two channels of a channel pair.

The audio encoder may be configured to select whether or not to encode at least a portion of the link layer and correlation information of the original signal based on the state information, so as to include in the auxiliary information an increased amount of link layer and correlation information in the case of relatively more low overload.

The audio encoder may be configured to select whether to decide what portion of the link layer and correlation information of the original signal should be encoded in the auxiliary information based on the channel-by-channel metrics, so as to include the link layer and correlation information associated with more sensitive metrics (e.g., with measures that are associated with more perceptually significant covariation) into auxiliary information.

The link layer and correlation information of the original signal may be in the form of a matrix.

The bitstream recording module may be configured to encode identification data of at least one channel.

According to an aspect, there is provided a method for generating a downmix signal from an original signal, wherein the original signal has at least two channels, wherein the downmix signal has at least one downmix channel.

The method may contain:

- assessment of channel level information and correlation of the original signal,

- encoding the downmix signal into a bitstream, such that the downmix signal is encoded into the bitstream such that it has auxiliary information including link layer and correlation information of the original signal.

The audio encoder may be decoder agnostic. The audio synthesis device may be decoder agnostic.

In accordance with an aspect, there is provided a system comprising an audio data synthesis apparatus described above or below and an audio encoder described above or below.

In accordance with an aspect, a persistent storage unit is provided storing instructions that, when executed by a processor, cause the processor to carry out a method described above or below.

3. Examples

3.1. Brief description of drawings

Fig. 1 shows a simplified overview of the processing according to the invention.

Fig. 2a shows an audio encoder according to the invention.

Fig. 2b shows another view of an audio encoder according to the invention.

Fig. 2c shows another view of an audio encoder according to the invention.

Fig. 2d shows another view of an audio encoder according to the invention.

Fig. 3a shows an audio synthesis device (decoder) according to the invention.

Fig. 3b shows another view of an audio synthesis device (decoder) according to the invention.

Fig. 3c shows another view of an audio synthesis device (decoder) according to the invention.

Fig. 4a-4d show examples of covariance synthesis.

Fig. 5 shows an example of a filter bank for an audio encoder according to the invention.

Fig. 6a-6c show examples of operation of an audio encoder according to the invention.

Fig. 7 shows an example of the prior art.

Fig. 8a-8c show examples of how covariance information is obtained according to the invention.

Fig. 9a-9d show examples of inter-channel coherence matrices.

Fig. 10a-10b show example frames.

Fig. 11 shows a circuit used by a decoder to obtain a mixing matrix.

3.2. Carrying out the invention

The examples are shown to be based on an encoder downmixing a signal 212 and passing link layer and correlation information 220 to a decoder. The decoder may generate a mixing rule (eg, a mixing matrix) from the link layer and correlation information 220. Information that is important for generating the mixing rule may include covariance information (eg, covariance matrix Cy) of the original signal 212 and covariance information (eg, covariance matrix C _x ) of the downmix signal. Although the covariance matrix C _x can be directly estimated by the decoder by analyzing the downmix signal, the covariance matrix C _y of the original signal 212 is easily estimated by the decoder. The covariance matrix C _y of the source signal 212 is generally a symmetric matrix (for example, a 5x5 matrix in the case of a 5-channel source signal 212): while the matrix represents, on the diagonal, the level of each channel, it represents the covariances between channels in off-diagonal entries. The matrix is diagonal because the covariance between common channels i and j is the same as the covariance between j and i. Therefore, to transmit entire covariance information to the decoder, it is necessary to signal 5 levels in diagonal records and 10 covariances for off-diagonal records to the decoder. However, it has been shown that it is possible to reduce the amount of information that must be encoded.

In addition, it is shown that in some cases, normalized values can be provided instead of levels and covariances. For example, inter-channel coherences (ICC, also indicated by _ξi , _j ) and inter-channel level differences (ICLD, also indicated by Xi ₎ indicating energy values may be provided. ICCs, for example, may be correlation values provided in place of covariances for off-diagonal entries of the matrix C _y . An example of correlation information could be of the form . In some examples, only the ξ _i , _j portion is actually encoded.

In this way, the ICC matrix is formed. The diagonal entries of the ICC matrix should in principle be equal to 1, and as such do not necessarily need to be encoded in the bitstream. However, it should be understood that the encoder may provide an ICLD to the decoder, for example in the form (see also below). In some examples, all X _i's are actually encoded.

Fig. 9a-9d show examples of an ICC matrix 900, with diagonal values "d" which may represent ICLD X _i and off-diagonal values indicated 902, 904, 905, 906, 907 (see below) which may represent ICC ξ _i , _j .

In this document, the product between matrices is indicated by the absence of a symbol. For example, the product between matrix A and matrix B is indicated by AB. The conjugate transposition of a matrix is indicated by an asterisk (*).

When referring to a diagonal, it serves as the main diagonal.

3.3. The present invention

Fig. 1 shows an audio system 100 with an encoder side and a decoder side. The encoder side may be implemented by encoder 200 and may receive audio signal 212, for example, from an audio sensor module (eg, microphones), or may be received from a storage module or from a remote module (eg, via radio transmission). The decoder side may be implemented by an audio decoder 300 (audio data synthesis device), which may transmit audio content to an audio playback module (eg, speakers). Encoder 200 and decoder 300 may communicate with each other, for example, through a communication channel, which may be wired or wireless (eg, radio frequency waves, light or ultrasound, etc.). The encoder and/or decoder may therefore include or couple to communication modules (eg, antennas, transceivers, etc.) to transmit a coded 248 bit stream from encoder 200 to decoder 300. In some cases, the encoder 200 may store the encoded stream of 248 bits in a storage module (eg, random access memory, flash memory, etc.) for future use. Likewise, decoder 300 may read a 248 bit stream stored in the storage module. In some examples, encoder 200 and decoder 300 may be the same device: after encoding and storing a 248 bit stream, the device may need to read it to play audio content.

Fig. 2a, 2b, 2c and 2d show examples of encoders 200. In some examples, the encoders of FIGS. 2a and 2b and 2c and 2d may be the same and differ from each other only due to the absence of certain elements in one and/or in the other drawing.

Audio encoder 200 may be configured to generate a downmix signal 246 from the source signal 212 (wherein the source signal 212 has at least two (e.g., three or more) channels, and the downmix signal 246 has at least one downmix channel).

Audio encoder 200 may include a parameter estimator 218 configured to estimate link layer information 220 and correlate the source signal 212. Audio encoder 200 may include a bitstream recording module 226 for encoding the downmix signal 246 into a bitstream 248. The downmix signal 246 is therefore encoded into a bit stream 248 such that it has side information 228 including link layer and correlation information of the original signal 212.

In particular, input signal 212 may be understood, in some examples, as a time domain audio signal, such as, for example, a time sequence of audio samples. The source signal 212 has at least two channels, which, for example, may correspond to different microphones (for example, for a position in stereo audio data or, for example, a position in multi-channel audio data) or, for example, correspond to different speaker positions of an audio playback module . Input signal 212 may be downmixed in downmixer calculation block 244 to produce a downmixed version 246 (also referred to as x) of original signal 212. This downmixed version of original signal 212 is also referred to as “downmix signal 246.” The downmix signal 246 has at least one downmix channel. The downmix signal 246 has fewer channels than the original signal 212. The downmix signal 212 may be in the time domain.

The downmix signal 246 is encoded into a bitstream 248 by a bitstream writer 226 (e.g., including an entropy encoder or multiplexer, or a base encoder) for a bitstream to be stored or transmitted to a receiving device (e.g., associated with a decoder side) . Encoder 200 may include a parameter estimator 218 (or parameter estimator). A parameter estimator 218 may estimate link layer and correlation information 220 associated with the original signal 212. The link layer and correlation information 220 may be encoded in the bit stream 248 as auxiliary information 228. In examples, link layer and correlation information 220 is encoded by the module 226 recording bit streams. In the examples, even if in FIG. 2b does not show the bitstream recording unit 226 below the downmix calculation unit 235, although the bitstream recording unit 226 may be present. In fig. 2c, it is shown that the bitstream recording module 226 may include a base encoder 247 to encode the downmix signal 246 so as to obtain an encoded version of the downmix signal 246. Fig. 2c also shows that the bitstream recording module 226 may include a multiplexer 249 that encodes in the bitstream 228 both the encoded downmix signal 246 and the link layer and correlation information 220 (eg, as encoded parameters) in the auxiliary information. 228.

As shown by FIG. 2b (but not shown in FIGS. 2a and 2c), the original signal 212 may be processed (e.g., by filter bank 214, see below) to produce a frequency domain version 216 of the original signal 212.

In fig. 6c shows an example of parameter estimation in which parameter estimation module 218 specifies parameters ξ _i , _j and X _i (eg, normalized parameters) to be subsequently encoded in the bit stream. Covariance estimators 502 and 504 estimate the covariance C _x and C _y , respectively, for the downmix signal 246 to be encoded and the input signal 212 . Then, in ICLD block 506 , the ICLD parameters X _i ICLD are calculated and transmitted to bitstream recording module 246 . In covariance-to-coherence conversion block 510, ICC ξ _i , _j (412) are obtained. At block 250, only some ICCs are selected for encoding.

The parameter quantizer 222 (FIG. 2b) may resolve the link layer and correlation information 220 in a quantized version 224.

The link layer and correlation information 220 of the source signal 212 may generally include information regarding the channel energy (or level) of the source signal 212. Additionally or alternatively, the link layer and correlation information 220 of the source signal 212 may include correlation information between pairs channels, for example, the correlation between two different channels. The link layer and correlation information may include information associated with a covariance matrix C _y (e.g., in a normalized form such as correlation or ICC) in which each column and each row is associated with a particular channel of the original signal 212, and in which the channel the layers are described by diagonal elements of the matrix C _y and correlation information, and the correlation information is described by non-diagonal elements of the matrix C _y . The matrix C _y can be such that it is a symmetric matrix (i.e. it is equal to its transpose) or a Hermitian matrix (i.e. it is equal to its conjugate transpose). C _y is, in general, positive semidefinite. In some examples, correlation may be replaced by covariance (and correlation information is replaced by covariance information). It should be understood that it is possible to encode, in the auxiliary information 228 of the bitstream 248, information associated with not all channels of the original signal 212. For example, it is not necessary to provide this link layer and correlation information with respect to all channels or all pairs of channels. For example, only a reduced set of information regarding the correlation between pairs of channels of the downmix signal 212 may be encoded in the bit stream 248, while the remaining information may be evaluated at the decoder side. In general, fewer elements can be encoded than diagonal elements of C _y , and fewer elements can be encoded than elements off the diagonal of C _y .

For example, the link layer and correlation information may include records of the covariance matrix C _y of the original signal 212 (link layer and correlation information of the original signal) and/or the covariance matrix C _x of the downmix signal 246 (covariance information of the downmix signal), for example, in a normalized form. For example, a covariance matrix may associate each line and each column with each channel so as to express the covariances between different channels and, on the diagonal of the matrix, the level of each channel. In some examples, link layer and correlation information 220 of the original signal 212 encoded in the auxiliary information 228 may include only link layer information (e.g., only diagonal values of the correlation matrix Cy) or only correlation information (e.g., only off-diagonal values correlation matrix Cy). The same applies to the covariance information of the downmix signal.

As shown below, the link layer and correlation information 220 may include at least one coherence value (ξ _i , _j ) describing the coherence between two channels i and j of the channel pair i, j . Additionally or alternatively, link layer and correlation information 220 may include at least one inter-channel layer difference (Xi) (ICLD). In particular, a matrix having ICLD values or inter-channel coherence (ICC) values can be specified. Therefore, the above examples regarding the transmission of matrix elements C _y and C _x can be generalized to other values that must be encoded (eg, transmitted) to implement link layer information 220 and correlation and/or downmix channel coherence information.

Input signal 212 may be divided into multiple frames. Different frames may, for example, have the same duration (eg, each may consist, during the elapsed time of one frame, of the same number of time domain samples). The different frames therefore have generally equal durations. In the bit stream 248, the downmix signal 246 (which may be a time domain signal) may be encoded frame by frame (or in any case, its subdivision into frames may be determined by a decoder). Link layer and correlation information 220 encoded as auxiliary information 228 in bit stream 248 may be associated with each frame (eg, link layer and correlation information parameters 220 may be provided for each frame or for multiple consecutive frames). Accordingly, for each frame of the downmix signal 246, associated side information 228 (eg, parameters) may be encoded in the side information 228 of bit stream 248. In some cases, multiple consecutive frames may be associated with the same link layer and correlation information 220 (eg, the same parameters) of the relative information(s) encoded in the auxiliary information 228 of the bit stream 248. Accordingly, one parameter may end up being jointly associated with multiple consecutive frames. This may occur, in some examples, when two consecutive frames have similar properties, or when the bit rate must be reduced (for example, due to the need to reduce operating data). For example:

- in the case of a large amount of operating data, the number of consecutive frames associated with the same specific parameter is increased so as to reduce the number of bits written to the bitstream;

- in the case of a smaller amount of working data, the number of consecutive frames associated with the same specific parameter is reduced in order to increase the quality of the mixing.

In other cases, when the bit rate decreases, the number of consecutive frames associated with the same particular parameter increases so as to decrease the number of bits written to the bit stream, and vice versa.

In some cases, it is possible to smooth parameters (or reconstructed or estimated values, such as covariances) using linear combination with parameters (or reconstructed or estimated values, such as covariances) preceding the current frame, for example, through summation, average, etc. .

In some examples, the frame may be divided between multiple subsequent slots. Fig. 10a shows frame 920 (subdivided into four consecutive slots 921-924), and FIG. 10b shows frame 930 (subdivided into four consecutive slots 931-934). The duration of different intervals may be the same. If the frame length has an interval size of 1.25 ms and 20 ms, there are 16 intervals in one frame (20/1.25=16).

Slot subdivision may be performed in filter banks (eg, 214), explained below.

In the example, the filter bank is a complex-modulated low-latency filter bank (CLDFB), the frame size is 20 ms, and the slot size is 1.25 ms, resulting in 16 filter bank slots per frame and number of frequency bands for each interval, which depends on the input sampling frequency, and the frequency bands are 400 Hz wide. Thus, for example, for an input sample rate of 48 kHz, the frame length per sample is 960, the interval length is 60 samples, and the number of filter bank samples per interval is also 60.

Sampling frequency/
kHz Frame/sample length Interval/sample length Number of filter bank frequency bands 48 960 60 60 32 640 40 40 16 320 20 20 8 160 10 10

Even though each frame (and also each interval) can be encoded in the time domain, analysis across frequency bands can be performed. In the examples, multiple frequency bands are analyzed for each frame (or interval). For example, a filter bank may be applied to the time signal, and the resulting subband signals may be analyzed. In some examples, link layer and correlation information 220 is also provided across frequency bands. For example, for each frequency band of input signal 212 or downmix signal 246, associated link layer and correlation information 220 (eg, C _y or ICC matrix) may be provided. In some examples, the number of frequency bands may be modified based on signal properties and/or the requested bit rate or measurements for the current operating data. In some examples, the more slots that are required, the fewer frequency bands are used to support the same bit rate.

Since the size of the interval is smaller than the size of the frame (in duration), intervals can be suitably used in the case of a transition portion in the original signal 212 detected in the frame: the encoder (and in particular the filter bank 214) can recognize the presence of the transition portion, signal its presence in the bit stream and indicate in the auxiliary information 228 of the bit stream 248 in which frame interval there is a transition part. In addition, the parameters of the link layer and correlation information 220 encoded in the auxiliary information 228 of the bitstream 248, respectively, can be associated only with the slots after the transition portion and/or the slot in which there is a transition portion. The decoder must therefore detect the presence of a transition portion and must associate link layer and correlation information 220 only with the slots after the transition portion and/or the slot in which the transition portion is present (for slots preceding the transition portion, the decoder uses the link layer information 220 and correlations for the previous frame). In fig. 10a, there is no transition portion, and the parameters 220 encoded in the auxiliary information 228 can therefore be understood to be associated with the entire frame 920. In FIG. 10b, the transition portion is present in interval 932: therefore, the parameters 220 encoded in the auxiliary information 228 should mean intervals 932, 933 and 934, while the parameters associated with interval 931 are supposed to be the same for the frame that precedes frame 930.

Given the above, for each frame (or slot) and for each frequency band, specific link layer and correlation information 220 associated with the original signal 212 may be specified. For example, elements of the covariance matrix C _y (eg, covariances and/or levels) can be estimated for each frequency band.

If a transition portion is detected while a plurality of frames are collectively associated with the same parameter, it is then possible to reduce the number of frames collectively associated with the same parameter, so as to improve the mixing quality.

Fig. 10a shows a frame 920 (referred to herein as a "normal frame") for which, in the original signal 212, eight frequency bands are defined (eight frequency bands 1...8 are shown in the ordinate, while intervals 921-924 are shown in the abscissa) . The link layer and correlation information 220 parameters can in theory be encoded in the 248 bit stream auxiliary information 228, across frequency bands (eg, one covariance matrix should be provided for each source frequency band). However, to reduce the amount of side information 228, the encoder may aggregate multiple source bands (eg, successive frequency bands) to obtain at least one aggregate band generated by the plurality of source bands. For example, in FIG. 10a, eight original frequency bands are grouped to obtain four aggregate frequency bands (aggregate frequency band 1 associated with original frequency band 1; aggregate frequency band 2 associated with original frequency band 2; aggregate frequency band 3 grouping original frequency bands 3 and 5; aggregated frequency band 3, grouping the original frequency bands 5...8). Covariance matrices, correlations, ICC, etc. may be associated with each of the aggregated frequency bands. In some examples, what is encoded in the auxiliary information 228 of the bitstream 248 are parameters derived from the sum (or average or other linear combination) of the parameters associated with each aggregate frequency band. Therefore, the size of the auxiliary information 228 of the 248 bit stream is further reduced. Hereinafter, “aggregate frequency band” is also referred to as “parameter frequency band” because it refers to those frequency bands used to determine parameters 220.

Fig. 10b shows a frame 931 (subdivided into four consecutive intervals 931-934 or other integer) in which there is a transition portion. Here, the transition portion is present in the second interval 932 (in the “transition interval”). In this case, the decoder may determine as a guide the parameters of the link layer information 220 and correlations only for the transition slot 932 and/or for subsequent slots 933 and 934. The link layer information 220 and correlations of the previous slot 931 are not provided: it should be understood that the link information level and slot correlation 931 should in principle, in particular, be different from the link layer and slot correlation information, but should likely be more similar to the link layer and slot correlation information of the frame preceding frame 930. Accordingly, the decoder should apply the link layer information and correlation of the frame preceding frame 930 to slot 931, and link layer and correlation information of frame 930 only to slots 932, 933, and 934.

Since the presence and position of transition intervals 931 may be signaled (eg, at 261, as shown below) in the side information 228 of the 248 bit stream, technology has been developed to eliminate or reduce the increase in size of the side information 228: groupings between aggregated frequency bands can change: for example, aggregate frequency band 1 now groups the original frequency bands 1 and 2, and aggregate frequency band 2 groups the original frequency bands 3...8. Therefore, the number of frequency bands is further reduced relative to the case of FIG. 10a, and the parameters need to be provided for only two aggregated frequency bands.

Fig. 6a shows that the parameter estimator 218 (parameter estimator) is capable of extracting a certain amount of link layer and correlation information 220.

Fig. 6a shows that the parameter estimator 218 is capable of extracting a number of parameters (link layer and correlation information 220), which may represent the ICC matrix 900 of FIG. 9a-9d.

However, only a portion of the estimated parameters are actually sent to the bitstream writer 226 to encode the auxiliary information 228. This is because the encoder 200 can be made selectable (in a determination block 250, not shown in FIGS. 1-5) whether or not to encode at least a portion of the link layer and correlation information 220 of the original signal 212.

This is illustrated in FIG. 6a as a plurality of switches 254s that are controlled by selecting 254 (commands) from determination unit 250. If each of the outputs 220 of the parameter estimator 218 is an ICC of the matrix 900 of FIG. 9c, not all of the parameters estimated by the parameter estimator 218 are actually encoded in the 248 bit stream auxiliary information 228: in particular, while the (ICC between channels) entries 908: R and L; C and L; C and R; RS and CS) are actually encoded, records 907 are not encoded (i.e., determination block 250, which may be the same as the block in FIG. 6c, may be considered to have open switches 254s for non-encoded records 907, but closed switches 254s for records 908 that must be encoded in the auxiliary information 228 of the 248 bit stream). It should be noted that information 254' regarding which parameters are selected for encoding (records 908) may be encoded (eg, as a bitmap or other information regarding which records 908 are encoded). In practice, information 254' (which, for example, may be an ICC map) may include indices (schematically illustrated in FIG. 9d) of encoded records 908. Information 254' may be in the form of a bitmap: for example, information 254' may consist from a fixed-length field, each position being associated with an index according to a given ordering, the value of each bit providing information as to whether the parameter associated with that index is actually provided.

In general, determination unit 250 may select whether or not to encode at least a portion of the link layer and correlation information 220 (ie, determine whether an entry of matrix 900 should be encoded), for example, based on state information 252. The state information 252 may be based on the state of the operating data: for example, in the case of a high transmission load, the amount of side information 228 that must be encoded in the bit stream 248 can be reduced. For example, and referring to FIG. 9c:

- in the case of a large volume of working data, the number of entries 908 of the matrix 900 that are actually written to the auxiliary information 228 of the 248 bit stream is reduced;

- in the case of a smaller amount of working data, the number of entries 908 of the matrix 900 that are actually written to the auxiliary information 228 of the 248 bit stream is reduced.

Alternatively or in addition, metrics 252 may be evaluated to determine which parameters 220 should be encoded in auxiliary information 228 (eg, which entries of matrix 900 are intended to be encoded entries 908 and which entries should be discarded). In this case, only parameters 220 (associated with more sensitive metrics, e.g., metrics that are associated with more perceptually significant covariance may be associated with records to be selected as coded records 908) may be encoded in the bitstream.

It should be noted that this process may be repeated for each frame (or for multiple frames, in the case of downsampling) and for each frequency band.

Accordingly, the determination unit 250 can also be controlled, in addition to status indicators, etc., by the parameter estimator 218, via instruction 251 in FIG. 6a.

In some examples (eg, FIG. 6b), the audio encoder may be further configured to encode 248 bits of the current link layer and correlation information 220t as an increment 220k relative to the previous link layer and correlation information 220(t-1). What is encoded by this bitstream recording module 226 in the auxiliary information 228 may be an increment 220k associated with the current frame (or interval) relative to the previous frame. This is shown in FIG. 6b. The current link layer and correlation information 220t is transmitted to the data storage element 270 such that the data storage element 270 stores the value of the current link layer and correlation information 220t for a subsequent frame. Meanwhile, the current link layer and correlation information 220t can be compared with the previously acquired link layer and correlation information 220(t-1). (This is shown in Fig. 6b as subtractor 273). Accordingly, the subtraction result 220Δ may be obtained by the subtractor 273. The difference 220Δ may be used in the scaling unit 220s to obtain a relative increment 220k between the previous link layer and correlation information 220(t-1) and the current link layer and correlation information 220t. For example, if the present link layer and correlation information 220t is 10% larger than the previous link layer and correlation information 220(t-1), then the increment 220 encoded in the auxiliary information 228 by the bitstream writer 226 should indicate the increment information of 10% . In some examples, instead of providing a relative increment of 220k, only a difference of 220Δ may be encoded.

The option of selecting the parameters that should actually be encoded from parameters such as ICC and ICLD, as explained above and below, can be adapted to the specific situation. For example, in some examples:

- for one first frame, only ICC 908 in FIG. 9c are selected to be encoded in the 248 bit stream side information 228, while the ICC 907 are not encoded in the 248 bit stream side information 228;

- for the second frame, various ICCs are selected for encoding, while various unselected ICCs are not encoded.

The same may be valid for intervals and frequency bands (and for various parameters such as ICLD). Therefore, the encoder (and, in particular, block 250) can determine which parameters should be encoded and which should not be encoded, thereby tailoring the selection of parameters that should be encoded to a particular situation (e.g., state, choice, etc.). d.). The "sign of importance" can therefore be analyzed in order to select which parameter should be encoded and which may not be encoded. The importance flag may be metrics associated, for example, with results obtained from modeling operations performed by the decoder. For example, the encoder may model the decoder's recovery of the uncoded covariance parameters 907, and the importance indicator may be scores indicating the absolute error between the uncoded covariance parameters 907 and parameters that are the same as the parameters assumed to be recovered by the decoder. By measuring errors in different simulation scenarios (eg, with each simulation scenario associated with transmitting some encoded covariance parameters 908 and measuring errors affecting the recovery of uncoded covariance parameters 907), it is possible to determine the simulation scenario that is least affected by errors (e.g., scenario simulation for which the metrics are relative to all errors in the reconstruction) in order to distinguish covariance parameters 908 that should be encoded from covariance parameters 907 that should not be encoded, based on the least affected simulation scenario. In the least affected scenario, the unselected parameters 907 represent the parameters that are most easily recoverable, and the selected parameters 908 tend to represent the parameters for which the error-associated metrics should be the largest.

The same can be done, instead of modeling parameters such as ICC and ICLD, by modeling reconstruction or estimation, by decoder, covariance, or by modeling mixing properties or mixing results. In particular, the simulation may be performed on a per-frame or per-slot basis, and may be performed on a per-band or aggregate-band basis.

An example could be a covariance recovery simulation using equation (4) or (6) (see below), starting with the parameters encoded in the auxiliary information 228 of the 248 bit stream.

To summarize, it is possible to recover link layer and correlation information from selected link layer and correlation information, thereby simulating an estimate, in the decoder (300), of unselected link layer and correlation information (220, Cy), and calculate error information between:

- unselected link layer information (220) and correlation estimated by the encoder; And

- unselected link layer and correlation information recovered by estimation modeling in the decoder (300), uncoded link layer and correlation information (220); And

- in order to distinguish, based on the calculated error information:

- properly restored link layer and correlation information; from

- link level and correlation information that cannot be properly restored,

- in order to make decisions on:

- selecting properly unrecoverable link layer and correlation information to be encoded in the auxiliary information (228) of the bit stream (248); And

- not selecting properly recoverable link layer and correlation information, thereby not encoding in the auxiliary information (228) a stream (248) of bits of properly recoverable link layer and correlation information.

In general terms, an encoder can simulate any decoder operation and estimate error metrics from the simulation results.

In some examples, the severity indicator may be different (or contain other different indicators) from the evaluation of indicators associated with errors. In some cases, the importance cue may be associated with a manual selection or an importance-based selection based on psychoacoustic criteria. For example, the most important channel pairs can be selected for coding (908), even without simulation.

The following is some additional explanation to explain how the encoder may signal which parameters 908 are actually encoded in the auxiliary information 220 of the bit stream 248.

Referring to FIG. 9d, the diagonal parameters of the ICC matrix 900 are associated with ordered indices 1...10 (the order being predetermined and known by the decoder). In fig. 9c shows that the selected parameters 908 to be encoded are the ICCs for L-R, L-C, R-C, LS-RS pairs, which are indexed by indices 1, 2, 5, 10, respectively. Accordingly, in the auxiliary information 228 of the bit stream 248, an indicator of indexes 1, 2, 5, 10 is also provided (eg, in information 254' of FIG. 6a). Accordingly, the decoder must understand that the four ICCs provided in the 248 bit stream auxiliary information 228 are L-R, L-C, R-C, LS-RS, based on information regarding indexes 1, 2, 5, 10 also provided by the encoder in the auxiliary information 228. Indexes can be transmitted, for example, through a bitmap, which associates the position of each bit in the bitmap with a given one. For example, to send indices 1, 2, 5, 10 in the service signals, you can write “1100100001” (in field 254' of the auxiliary information 228), since the first, second, fifth and tenth bits mean indices 1, 2, 5, 10 ( other options are available to those skilled in the art). It is a so-called "one-dimensional index", but other indexing strategies are possible. For example, combinatorial number technology, according to which the number N is encoded (in field 254' of the auxiliary information 228), which is uniquely associated with a specific pair of channels (see also https://en.wikipedia.org/wiki/Combinatorial_number_system). A bitmap may also be called an "ICC map" when it refers to ICC.

It should be noted that in some cases non-adaptive (fixed) provision of parameters is used. This means that in the example of FIG. 6a, the selection option 254 from the parameters to be encoded is fixed and there is no need to indicate the selected parameters in the field 254'. Fig. 9b shows an example of fixed parameter provisioning: the selected ICCs are L-C, L-LS, R-C, CRS, and there is no need to signal their indices since the decoder already knows which ICCs are encoded in the 248 bit stream ancillary information 228.

However, in some cases the encoder may choose between fixed parameter provisioning and adaptive parameter provisioning. The encoder may signal the selection in the auxiliary information 228 of the 248 bit stream so that the decoder may know which parameters are actually encoded.

In some cases, at least some parameters may be provided without adaptation. For example:

- ICDLs can be encoded in any case, without the need to specify them in the bitmap; And

- ICCs may be subject to adaptive provisioning.

Explanations are associated with each frame or interval or frequency band. For a subsequent frame or interval or frequency band, various parameters 908 must be transmitted to the decoder, various indices associated with the subsequent frame or interval or frequency band; and different choices can be made (e.g., fixed versus adaptive). Fig. 5 shows an example of a filter bank 214 of the encoder 200 that can be used to process the source signal 212 to produce a frequency domain signal 216. As can be seen from Fig. 5, the time domain (TD) signal 212 may be analyzed by a transient analyzer (transient detector) 258. In addition, conversion to a frequency domain (FD) version 264 of the input signal 212, in a plurality of frequency bands, is provided by a filter 263 (which may implement, for example, a Fourier filter, a short Fourier filter, a quadrature mirror, etc.). The frequency domain version 264 of the input signal 212 may be analyzed, for example, in a frequency band analysis block 267, which can determine (per instruction 268) a particular frequency band grouping to be performed in a segment grouping block 265. Thereafter, the FD signal 216 must be a signal in a reduced number of aggregate frequency bands. Frequency band aggregation is explained above with respect to FIG. 10a and 10b. The segment grouping unit 267 may also be transformed and consistent through transient analysis performed by the transient analysis unit 258 . As explained above, it may be possible to further reduce the number of aggregated frequency bands in the case of a transition portion: therefore, information 260 regarding the transition portion can transform and coordinate the grouping of segments. Additionally or alternatively, information 261 regarding the transition portion is encoded in the auxiliary information 228 of the bit stream 248. The information 261, when encoded in the auxiliary information 228, may include, for example, a flag indicating whether there is a transition portion (for example: "1" meaning "there is a transition portion in the frame" versus "0" meaning: “there is no transition part in the frame”), and/or an indicator of the position of the transition part in the frame (for example, a field indicating in what interval the transition part is observed). In some examples, if the information 261 indicates that there is no transition portion ("0") in the frame, the transition portion position indicator is not encoded in the auxiliary information 228 to reduce the size of the bit stream 248. Information 261 is also called “transition parameter” and is shown in FIG. 2d and 6b as a 246-bit stream encoded in the auxiliary information 228.

In some examples, the grouping of segments in block 265 may also be translated and consistent through external information 260', such as information regarding transmission status (eg, measurements associated with transmissions, error rates, etc.). For example, the larger the amount of operating data (or the higher the error rate), the greater the aggregation (tendentiously fewer aggregated frequency bands that are wider) so as to have less amount of ancillary information 228 that must be encoded in the 248 bit stream. Information 260', in some examples, may be similar to information or metrics 252 of FIG. 6a.

In general, it is not practical to send parameters for every frequency band/slot combination, but filter bank samples are grouped by both the number of slots and the number of frequency bands to reduce the number of parameter sets that are sent per frame. Along the frequency axis, the grouping of frequency bands into parameter frequency bands exploits the non-constant separation within the parameter frequency bands, where the number of frequency bands within the parameter frequency bands is not constant, but often corresponds to the psychoacoustically determined resolution of the parameter frequency bands, i.e. at lower pass bands, parameter bands contain only one or a small number of filter bank frequency bands, and for higher parameter frequency bands, a larger (and increasing) number of filter bank frequency bands are grouped into one parameter band.

Thus, for example, again for an input sample rate of 48 kHz and the number of parameter frequency bands set to 14, the following vector grp14 describes the filter bank indices that provide the frequency band boundaries for the parameter frequency bands (with the index starting at 0):

grp14=

Parameter frequency band j contains filter bank frequency bands .

It should be noted that the grouping of frequency bands for 48 kHz can also be directly used for other possible sampling frequencies by simply truncation, since the grouping both corresponds to a psychoacoustically determined frequency scale and has specific frequency band boundaries corresponding to the number of frequency bands for each sampling frequency ( Table 1).

If the frame is intransitive, or transition processing is not implemented, grouping along the time axis is performed across all intervals in the frame such that one set of parameters is available per parameter bandwidth.

In this case, the number of parameter sets should be large, but the temporal resolution may be lower than 20 millisecond frames (40 ms on average). Thus, to further reduce the number of parameter sets sent per frame, only a subset of the parameter bandwidths is used to determine and encode the parameters to be sent in the bit stream to the decoder. The subsets are fixed and known to both the encoder and the decoder. The particular subset sent in the bitstream is signaled by a field in the bitstream to indicate to the decoder which subset of parameter bands the transmitted parameters belong to, and the decoder then replaces the parameters for that subset by the transmitted ones (ICC, ICLD) and stores parameters from previous frames (ICC, ICLD) for all frequency bands of parameters that are not in the current subset.

In an example, the parameter bands may be divided into two subsets containing approximately half of the total parameter bands, and a contiguous subset for the lower parameter bands and one contiguous subset for the higher parameter bands. Since there are two subsets, the bit stream field to be transmitted in the subset signaling is one bit, and an example for subsets for 48 kHz and 14 parameter bands is:

,

- Where indicates which subset the j frequency band of parameters belongs to.

It should be noted that the downmix signal 246 may actually be encoded, in a bit stream 248, as a time domain signal: simply, the subsequent parameter estimator 218 must estimate the parameters 220 (eg, ξ _i , _j and/or X _i ) in the frequency domain region (and decoder 300 must use parameters 220 to prepare a mixing rule 403 (eg, a mixing matrix), as explained below).

Fig. 2d shows an example of an encoder 200, which may be one of the previous encoders or may include elements of the previously explained encoders. The TD input signal 212 is input to the encoder and a bit stream 248 is output, wherein the bit stream 248 includes a downmix signal 246 (e.g., encoded by the base encoder 247) and correlation and level information 220 encoded in the auxiliary information 228.

As can be seen from Fig. 2d, the filter bank 214 may be turned on (an example of a filter bank is shown in FIG. 5). Frequency domain (FD) conversion is provided in block 263 (frequency domain DMX) to obtain an FD signal 264, which is an FD version of the input signal 212. The FD signal 264 (also indicated with an X) is obtained in a variety of frequency bands. A frequency band/slot grouper 265 may be provided (which may implement the grouper 265 of FIG. 5) to obtain the FD signal 216 in the aggregated frequency bands. The FD signal 216, in some examples, may be a version of the FD signal 264 in fewer frequency bands. Signal 216 may then be passed to parameter estimator 218, which includes covariance estimators 502, 504 (shown here as one separate block) and, below, parameter estimator and encoder 506, 510 (embodiments of elements 502, 504, 506 and 510 are shown in Fig. 6c). The parameter estimator and encoder 506, 510 may also provide parameters 220 to be encoded in the auxiliary information 228 of the bit stream 248. The transition portion detector 258 (which may be implemented by the transient analysis block 258 of FIG. 5) may recognize the transition portions and/or the position of the transition portion in the frame (eg, in which interval the transition portion is identified). Accordingly, information 261 regarding the transition portion (eg, a parameter of the transition portions) may be transmitted to parameter estimation module 218 (eg, to determine which parameters should be encoded). The transition detector 258 may also provide information or commands (268) to block 265 such that grouping is performed based on the presence and/or position of the transition in the frame.

Fig. 3a, 3b, 3c show examples of audio decoders 300 (also called “audio synthesis devices”). In the examples, the decoders of FIGS. 3a, 3b, 3c may be the same decoder, just with some differences to eliminate different elements. In examples, decoder 300 may be the same as the decoders of FIGS. 1 and 4. In the examples, decoder 300 may also be the same device as encoder 200.

Decoder 300 may be configured to generate a synthesis signal (336, 340, yR) from the downmix signal x in the TD (246) or in the FD (314). Audio synthesis apparatus 300 may include an input interface 312 configured to receive a downmix signal 246 (eg, a downmix signal identical to the downmix signal encoded by encoder 200) and side information 228 (eg, encoded in a bit stream 248). Ancillary information 228 may include, as explained above, link layer and correlation information (220, 314), such as at least one of ξ, X, etc. or elements thereof (as explained below) of the original signal, which may be the original input signal 212, y, at the encoder side. In some examples, all ICLD(X) and some (but not all) entries 906 or 908 outside the diagonal of ICC matrix 900 (ICC or ξ values) are obtained by decoder 300.

Decoder 300 may be configured (for example, through a prototype signal calculator or prototype signal calculation module 326) to calculate a prototype signal 328 from the downmix signal (324, 246, x), wherein the prototype signal 328 has a number of (more than one) signal channels. 336 synthesis.

Decoder 300 may be configured (eg, through mixing rule calculation module 402) to calculate mixing rule 403 using at least one of:

- link level information and correlation (for example, 314, C _y , ξ, X or their elements) of the original signal (212, y); And

- covariance information (eg, C _x or elements thereof) associated with the downmix signal (324, 246, x).

Decoder 300 may include a synthesis processor 404 configured to generate a synthesis signal (336, 340, yR) using the prototype signal 328 and the mixing rule 403.

The synthesis processor 404 and the mixing rules calculation module 402 may be combined into a single synthesis engine 334 . In some examples, the mixing rules calculation module 402 may be external to the synthesis engine 334. In some examples, the mixing rule calculation module 402 of FIG. 3a may be integrated with the parameter recovery module 316 of FIG. 3b.

The number of signal synthesis channels (336, 340, yR) of the synthesis is greater than one (and in some cases more than two or more than three) and may be more, less, or equal to the number of original channels compared to the original signal (212, y), which is also greater one (and in some cases more than two or more than three). The number of signal channels (246, 216, x) of the downmix is at least one or two and less than the number of original channels of the original signal (212, y) and the number of synthesis signal channels (336, 340, yR) of the synthesis.

Input interface 312 may read an encoded bit stream of 248 (eg, the same bit stream of 248 encoded by encoder 200). Input interface 312 may be or include a bitstream reader and/or an entropy decoder. The bit stream 248 may encode, as discussed above, the downmix signal (246, x) and side information 228. The side information 228 may comprise, for example, original link layer information and correlations 220 in the form output by either parameter estimator 218 or any of the elements below relative to the parameter estimation module 218 (eg, parameter quantization unit 222, etc.). The auxiliary information 228 may contain either encoded values, indexed values, or both. Even if the input interface 312 is not shown in FIG. 3b for the downmix signal (346, x), although it can also be applied to the downmix signal as shown in FIG. 3a. In some examples, input interface 312 may quantize parameters derived from the 248 bit stream.

The decoder 300 can therefore receive a downmix signal (246, x), which may be in the time domain. As explained above, the downmix signal 246 may be divided into frames and/or intervals (see above). In examples, filter bank 320 may transform the time domain downmix signal 246 to produce a frequency domain version 324 of the downmix signal 246. As explained above, the version frequency bands 324 in the frequency domain of the downmix signal 246 may be grouped into groups of frequency bands. In the examples, the same grouping may be performed as that performed in filter bank 214 (see above). The parameters for grouping (for example, which frequency bands and/or how many frequency bands should be grouped, etc.) can be based, for example, on the transmission of overhead signals through the segment grouping module 265 or the frequency band analysis unit 267, wherein the transmission of overhead signals signals are encoded in auxiliary information 228.

The decoder 300 may include a prototype signal calculation module 326. The prototype signal calculation module 326 may calculate the prototype signal 328 from the downmix signal (eg, one of the versions 324, 246, x), for example, by applying a prototype rule (eg, the Q matrix). The prototype rule may be implemented by a prototype matrix (Q) with a first dimension and a second dimension, wherein the first dimension is associated with the number of downmix channels and the second dimension is associated with the number of synthesis channels. Therefore, the prototype signal has a number of synthesis signal channels 340 that must ultimately be generated.

The prototype signal calculator 326 may apply so-called upmixing to the downmix signal (324, 246, x), in the sense that it simply generates a version of the downmix signal (324, 246, x) on an increased number of channels (the number of channels of the signal synthesis that must be formed), but without the application of significant “intellectuality”. In examples, prototype signal calculator 326 may simply apply a fixed specified prototype matrix (identified as "Q" herein) to FD version 324 of downmix signal 246. In examples, prototype signal calculator 326 may apply different prototype matrices to different frequency bands. The prototype rule (Q) may be selected from a plurality of pre-stored prototype rules, for example, based on a specific number of downmix channels and a specific number of synthesis channels.

The prototype signal 328 may be decorrelated in the decorrelation module 330 so as to obtain a decorrelated version 332 of the prototype signal 328. However, in some examples, preferably the decorrelation module 330 is not present because the invention has been proven to be effective enough to allow its omission.

The prototype signal (in any of its versions 328, 332) may be input to the synthesis engine 334 (and, in particular, the synthesis processor 404). Here, the prototype signal (328, 332) is processed to obtain a synthesis signal (336, yR). The synthesis engine 334 (and, in particular, the synthesis processor 404) may apply a mixing rule 403 (in some examples explained below, the mixing rules are equal to two, for example, one for the main component of the synthesis signal and one for the residual component). The mixing rule 403 may be implemented, for example, by a matrix. The matrix 403 may be generated, for example, by the mixing rules calculation module 402, based on the link layer information and the correlation (314, eg, ξ, X or elements thereof) of the original signal (212, y).

The synthesis signal 336 output by the synthesis engine 334 (and, in particular, the synthesis processor 404) may be filtered by a filter bank 338 if desired. Additionally or alternatively, the synthesis signal 336 may be converted to the time domain in a filter bank 338. A (time domain or filtered) version 340 of the synthesis signal 336 can therefore be used for audio playback (eg, through loudspeakers).

To obtain a mixing rule 403 (eg, a mixing matrix), link layer and correlation information (eg, C _y , C _{y R} , etc.) of the original signal and covariance information (eg, C _x ) associated with the downmix signal may transmitted to the mixing rule calculation module 402. For this purpose, link layer and correlation information 220 encoded in auxiliary information 228 by encoder 200 can be used.

However, in some cases, to reduce the amount of information encoded in the 248 bit stream, not all parameters are encoded by the encoder 200 (e.g., not all of the link layer and correlation information of the original signal 212 and/or not all of the covariance information of the downmixed signal 246). Therefore, some parameters 318 must be estimated in the parameter recovery module 316.

Parameter recovery module 316 may be supplied, for example, with at least one of:

- a version 322 of the downmix signal 246(x), which, for example, may be a filtered version or an FD version of the downmix signal 246; And

- auxiliary information 228 (including link layer and correlation information 228).

The auxiliary information 228 may include (as level and correlation information of the input signal) information associated with the correlation matrix C _y of the original signal (212, y): however, in some case, not all elements of the correlation matrix C _y are actually encoded. Consequently, estimation and recovery technologies are designed to recover the version (C _yR ) of the correlation matrix C _y (for example, through intermediate steps that obtain the estimated version ).

Parameters 314 passed to module 316 may be obtained by entropy decoder 312 (input interface) and, for example, may be quantized.

Fig. 3c shows an example of a decoder 300, which may be an embodiment of one of the decoders of FIG. 1-3b. Here, the decoder 300 includes an input interface 312 provided by a demultiplexer. Decoder 300 outputs a synthesis signal 340, which, for example, may be in the TD (signal 340), which is to be played back through the speakers, or in the FD (signal 336). Decoder 300 of FIG. 3c may include a core decoder 347, which may also be part of an input interface 312. The core decoder 347 may therefore provide a downmix signal x, 246. Filter bank 320 may convert downmix signal 246 from TD to FD. The FD version of the downmix signal x, 246 is indicated by 324. The downmix FD signal 324 may be provided to block 388 for covariance synthesis. The covariance synthesis block 388 may provide a synthesis signal 336 (Y) to the FD. The inverse filter bank 338 may convert the audio signal 314 to its TD version 340. The FD downmix signal 324 may be transmitted to the frequency band/slot grouper 380 . The frequency band/slot grouper 380 may perform the same operation as that performed in the encoder by the segment grouper 265 of FIG. 5 and 2d. Since the frequency bands of the downmix signal 216 of FIG. 5 and 2d, in the encoder, are grouped or aggregated into a small number of frequency bands (with a large width), and the parameters 220 (ICC, ICLD) are associated with groups of aggregated frequency bands, it is now necessary to aggregate the decoded downmix signal in the same way: each aggregated band frequencies with a related parameter. Therefore, number 385 means the XB downmix signal after aggregation. It should be noted that the filter provides a non-aggregated representation of the FD in such a way as to be able to process the parameters, since in the encoder the band/slot grouping in the decoder (380) performs the same aggregation across bands/slots with the encoder to provide aggregated XB downmixing.

Band/slot grouper 380 may also aggregate across different slots in a frame, such that signal 385 is also aggregated into an encoder-like slot dimension. The frequency band/slot grouper 380 may also receive information 261 encoded in the auxiliary information 228 of the bit stream 248 indicating the presence of a transition portion, and in this case, also the position of the transition portion in the frame.

In covariance estimation block 384, the covariance C _x of the downmix signal 246 (324) is estimated. The covariance C _y is obtained in covariance calculation block 386, for example, by using equations (4)-(8) can be used for this purpose. Fig. 3c shows a “multi-channel parameter” which, for example, may be parameters 220 (ICC and ICLD). The covariances C _y and C _x are then passed to a covariance synthesis block 388 to synthesize the synthesis signal 388 . In some examples, blocks 384, 386, and 388 may perform, when considered together, both parameter recovery 316 and mixing calculations 402, and synthesis processor 404, as explained above and below.

4. Explanation

4.1. General overview

The new approach of the present examples aims, among other things, to perform encoding and decoding of multi-channel content at low bit rates (meaning at or below 160 Kbps) while maintaining audio quality as close as possible to the original signal and preserving the spatial properties of multi-channel signal. One characteristic of the new approach is also to fit the DirAC framework mentioned above. The output signal may be rendered by the same speaker configuration as the speaker configuration for input signal 212, or by a different speaker configuration (which may be larger or smaller in terms of speakers). Additionally, the output signal can be prepared by rendering on loudspeakers using binaural rendering.

The current section should provide a comprehensive description of the invention and the various modules that make up it.

The proposed system consists of two main parts:

- Encoder 200, which extracts the necessary parameters 220 from the input signal 212, quantizes them (at 222) and encodes them (at 226). Encoder 200 may also calculate a downmix signal 246, which is encoded into a bit stream 248 (and may be passed to decoder 300).

- A decoder 300 that uses the encoded (e.g., transmitted) parameters and the downmixed signal 246 to produce a multi-channel output signal whose quality is as close as possible to the original signal 212.

Fig. 1 shows an overview of the proposed new approach according to the example. It should be noted that some examples use only a subset of the configuration blocks shown in the overall diagram and exclude certain processing blocks depending on the application situation.

Input signal 212(y) in the invention is a multi-channel audio signal 212 (also referred to as a "multi-channel stream") in the time domain or time-frequency domain (e.g., signal 216), meaning, for example, a set of audio signals that are generated or intended to playback through a set of speakers.

The first part of the processing is the encoding part; from a multi-channel audio signal, a so-called "downmix" signal 246 must be calculated (see also 4.2.6) along with a set of parameters or auxiliary information, 228 (see also 4.2.2 and 4.2.3) that are extracted from the input signal 212 in the time domain or in the frequency domain. These parameters are encoded (see also 4.2.5) and, in this case, transmitted to the decoder 300.

The downmix signal 246 and encoded parameters 228 can then be transmitted to the base encoder and transmission channel that couples the encoder side and the decoder side of the process.

On the decoder side, the downmixed signal is processed (4.3.3 and 4.3.4) and the transmitted parameters are decoded (see also 4.3.2). The decoded parameters are used to synthesize the output signal using covariance synthesis (see also 4.3.5) and this should result in a final multi-channel time domain output signal.

Before going into detail, there are some general characteristics that must be installed, at least one of which is acceptable:

- Processing can be used with any speaker configuration. It should be taken into account that as the number of loudspeakers increases, the complexity of the process and the number of bits required to encode the transmitted parameters must also increase.

- All processing can be done on a frame-by-frame basis, i.e. input signal 212 may be divided into frames that are processed independently. On the encoder side, each frame forms a set of parameters that must be transmitted to the decoder side for processing.

- The frame can also be divided into intervals; these intervals then represent statistical properties that cannot be obtained at the frame scale. The frame may be divided, for example, into eight intervals, and the length of each interval should be 1/8 of the length of the frame.

4.2. Encoder

The encoder's job is to extract the appropriate parameters 220 to describe the multi-channel signal 212, quantize them (at 222), encode them (at 226) as auxiliary information 228, and then, in this case, transmit them to the decoder side. The parameters 220 are described in detail here, as well as how they can be calculated.

A more detailed diagram of encoder 200 is included in FIG. 2a-2d. This general view highlights the two main outputs of the 228 and 246 encoder.

The first output of the encoder 200 is a downmix signal 228, which is calculated from the multi-channel input audio signal 212; downmix signal 228 is a representation of the original multi-channel stream (signal) in fewer channels than the original content (212). Additional information regarding their calculation is contained in paragraph 4.2.6.

The second output of encoder 200 is encoded parameters 220 expressed as side information 228 in a bit stream 248; these parameters 220 represent the key point of the present examples: they are parameters that are used to effectively describe the multi-channel signal at the decoder side. These parameters 220 provide a good compromise between the quality and the number of bits required to encode them into a 248 bit stream. On the encoder side, parameter calculation can be carried out in several stages; The process is described below in the frequency domain, but can also be performed in the time domain. The parameters 220 are first estimated from the multi-channel input signal 212, then they may be quantized in the quantizer 222, and thereafter they may be converted to a digital bit stream 248 as auxiliary information 228. Additional information regarding these steps is contained in paragraphs 4.2.2, 4.2.3 and 4.2.5.

4.2.1. Filter comb and segment grouping

Filter banks are explained for the encoder side (eg, filter bank 214) or for the decoder side (eg, filter banks 320 and/or 338).

The invention may use filter combs at various points during the process. These filter banks can transform the signal either from the time domain to the frequency domain (into so-called aggregate or parameter frequency bands), in which case they are called "analysis filter banks", or from the frequency domain to the time domain (e.g. 338), in which case they are called “synthesis filter banks”.

The filter bank selection must match the desired performance and optimization requirements, but other processing can be performed independently of the specific filter bank selection. For example, you can use a filter bank based on quadrature mirror filters or a filter bank based on the short-time Fourier transform.

Referring to FIG. 5, the output of the filter bank 214 of the encoder 200 should be a frequency domain signal 216 represented for a number of frequency bands (266 versus 264). Doing the rest of the processing for all frequency bands (264) can be understood as providing better quality and better frequency resolution, but also requires more critical bit rates to transmit all the information. Therefore, along with the filter bank process, so-called “segment grouping” (265) is performed, which corresponds to grouping a certain frequency to represent information 266 regarding a smaller set of frequency bands.

For example, the output signal 264 of filter 263 (FIG. 5) may be represented in 128 frequency bands, and grouping the segments into 265 may result in a signal 266 (216) with only 20 frequency bands. There are several ways to group frequency bands, and one obvious way, for example, might be to try to approximate an equivalent rectangular bandwidth. Equivalent Rectangular Bandwidth is a type of psychoacoustically driven frequency band division that attempts to model how the human auditory system processes audio events, i.e. the challenge is to group the filter banks in a way that is suitable for human hearing.

4.2.2. Parameter estimation (e.g., estimation module 218)

Aspect 1. Using covariance matrices to describe and synthesize multichannel content

The parameter estimation at 218 is one of the main points of the invention; they are used on the decoder side to synthesize the output multi-channel audio signal. These parameters 220 (encoded as auxiliary information 228) are chosen because they effectively describe the multi-channel input stream 212 (signal) and they do not require a large amount of data to be transmitted. These parameters 220 are calculated on the encoder side and subsequently used in conjunction with the synthesis engine on the decoder side to calculate the output signal.

Here, covariance matrices can be calculated between the channels of the multi-channel audio signal and the downmixed signal. Namely:

- Cy: covariance matrix of the multi-channel stream (signal), and/or

- C _x : covariance matrix of the downmix stream 246 (signal).

Processing can be performed on a parameter band basis, whereby a parameter band is independent of another parameter band, and equations can be described for a given parameter band without loss of generality.

For a given frequency band of parameters, the covariance matrices are defined as follows:

(1)

- Where:

- denotes the real part operator.

- Instead of the real part, any other operation may be provided that results in a real value that has a relationship with the complex value from which it is derived (for example, an absolute value)

- * denotes the conjugate transpose operator

- B denotes the relationship between the original number of frequency bands and the grouped frequency bands (see 4.2.1 below regarding grouping of segments)

- Y and X are, respectively, the original multi-channel signal 212 and the downmixed frequency domain signal 246

C _y (or its elements or values derived from C _y or its elements) is also indicated as link layer and correlation information of the original signal 212. C _x (or its elements or values derived from C _y or its elements) is also are indicated as covariance information associated with the downmix signal 212.

For a given frame (and frequency band), only one or two covariance matrices C _y and/or C _x may be output, for example, by estimator 218 . Because the process is based on intervals rather than frames, different implementations can be made regarding the relationship between matrices for given intervals and for an entire frame. As an example, one can calculate the covariance matrix(es) for each interval in a frame and sum them to output a matrix for one frame. It should be noted that the definition for calculating the covariance matrices is a mathematical definition, but it is also possible to calculate or at least modify these matrices in advance if it is necessary to obtain an output signal with specific characteristics.

As explained above, it is not necessary that all elements of the matrices C _y and/or C _x are actually encoded in the auxiliary information 228 of the bit stream 248. For C _x , they can simply be estimated from the downmix signal 246 encoded by applying equation (1), and as such, encoder 200 can easily dispense with encoding any element of C _x (or more generally, covariance information associated with the downmix signal). For C _y (or for link layer information and correlation associated with the original signal), at least one of the elements of C _y can be estimated, on the decoder side, using techniques explained below.

Aspect 2a. Transfer of covariance matrices and/or energies to describe and reconstruct a multichannel audio signal

As mentioned above, covariance matrices are used for synthesis. It is possible to pass these covariance matrices (or a subset of them) directly from the encoder to the decoder.

In some examples, the matrix C _x does not necessarily need to be transmitted since it can be recalculated at the decoder side using the downmix signal 246, but depending on the application scenario, this matrix may be required as a transmitted parameter.

From an implementation point of view, not all values in these matrices C _x , C _y need to be encoded or transmitted, for example, to satisfy certain specific bit rate requirements. Non-transmitted values may be evaluated on the decoder side (see also 4.3.2).

Aspect 2b. Transfer of inter-channel coherences and inter-channel level differences for characterization and reconstruction of a multi-channel signal

From the covariance matrices C _x , C _y , an alternative set of parameters may be specified and used to reconstruct the multi-channel signal 212 at the decoder side. These parameters may be, for example, inter-channel coherence (ICC) and/or inter-channel level differences (ICLD).

Inter-channel coherences describe the coherence between each channel of a multi-channel stream. This parameter can be extracted from the covariance matrix C _y and calculated as follows (for a given parameter bandwidth and for two given channels i and j):

(2)

- Where:

- ξ _i , _j is the ICC between channels i and j of input signal 212

- are the values in the covariance matrix (previously specified in equation (1)) of the multi-channel signal between channels i and i of the input signal 212

ICC values may be calculated between each channel of a multi-channel signal, which can result in a large amount of data as the size of the multi-channel signal grows. In practice, a reduced ICC set may be encoded and/or transmitted. The encoded and/or transmitted values must be determined, in some examples, according to a performance requirement.

For example, when solving problems associated with a signal generated by 5.1 (or 5.0) as a certain speaker configuration established by the ITU recommendation “ITU-R BS.2159-4”, you can choose to transmit only four ICCs. The four ICCs may represent the ICC between:

- center and right channel

- central and left channel

- left channel and left surround channel

- right channel and right surround channel

In general, the indices of the ICCs selected from the ICC matrix are described by an ICC map.

In general, for each speaker configuration, a fixed set of ICCs that provide the best quality on average can be selected for encoding and/or transmission to the decoder. The number of ICCs and which ICCs must be transmitted may depend on the speaker configuration and/or the available full bit rate and are available in the encoder and decoder without the need to transmit an ICC map in a 248-bit stream. In other words, a fixed ICC set and/or a corresponding fixed ICC map may be used, for example, depending on the speaker configuration and/or the overall bit rate.

These fixed sets may not be suitable for a particular material and provide, in some cases, significantly worse quality than the average quality for all material using the ICC fixed set. To overcome this, in another example, for each frame (or slot), the optimal ICC set and the corresponding ICC map can be estimated based on the importance indicator of a particular ICC. The ICC map used for the current frame is then explicitly encoded and/or transmitted along with the quantized ICCs in a 248-bit stream.

For example, the ICC importance feature may be determined by generating a covariance estimate or ICC matrix scores using the downmix covariance C _x from equation (1), similar to the decoder using equations (4) and (6) from 4.3.2. Depending on the selected feature, the feature is calculated for each ICC or corresponding entry in the covariance matrix for each frequency band, for which the parameters are transmitted in the current frame and are combined for all frequency bands. This matrix of combined features is then used to determine the most important ICCs and thus the set of ICCs that should be used and the ICC map that should be transmitted.

For example, the ICC importance feature represents the absolute error between records of the estimated covariance and the actual covariance C _y , and the combined feature matrix is the sum for the absolute error for each ICC over all frequency bands to be transmitted in the current frame. From the combined feature matrix, the n entries in which the summed absolute error is the largest are selected, and n constitutes the number of ICCs to be transmitted for the speaker/bit rate combination, and an ICC map is constructed from these entries.

Moreover, in another example, as shown in FIG. 6b, to prevent ICC maps from changing too much between frames, the feature matrix can be underlined for each entry that is in the selected ICC map of the previous parameter frame, for example in the case of absolute covariance error by applying a factor >1 (220k) to the ICC map entries of the previous frame frame.

Additionally, in another example, a flag sent in the 248 bit stream auxiliary information 228 may indicate whether a fixed ICC map or an optimal ICC map is used in the current frame, and if the flag indicates a fixed set, then the ICC map is not sent in the 248 bit stream. bits

The optimal ICC map, for example, is encoded and/or transmitted as a bitmap (for example, the ICC map may implement information 254' of FIG. 6a).

Another example for passing an ICC map is to pass an index to a table of all possible ICC maps, where, for example, the index itself is additionally entropy encoded. For example, a table of all possible ICC maps is not stored in the storage device, but the ICC map indicated by the index is directly calculated from the index.

The second parameter, which may be sent in conjunction with the ICC (or separately), is the ICLD. "ICLD" means inter-channel level difference, and it describes the energy relationships between each channel of the input multi-channel signal 212. There is no unique definition of ICLD; An important aspect of this value is that it describes the relationships of energies in a multichannel flow.

As an example, the conversion from C _y to ICLD could be obtained as follows:

(3)

- Where:

- X _i is the ICLD for channel i.

- Pi is the power of the current channel i, it can be extracted from the diagonal C _y : .

- P _{dmx, i} : depends on channel i, but should always be a linear combination of the values in C _x , it also depends on the original speaker configuration.

In the examples, P _dmx,i is not the same for each channel, but depends on the mapping associated with the downmix matrix (which is a prototype matrix for the decoder), this is mentioned generally in one of the key points in equation (3). Depends on whether channel i is downmixed into only one of the downmix channels or into more than one of them. In other words, P _{dmx, i} may be or include the sum over all diagonal elements of C _x that have a non-zero element in the downmix matrix, so that equation (3) can be rewritten as follows:

- where α _i is a weighting factor associated with the expected proportion of channel energy in the downmix, this weighting factor being fixed for a particular input speaker configuration and known in the encoder and decoder. Below is the concept of matrix Q. Some values of α _i and matrices Q are also given at the end of the document.

In the case of an implementation specifying a mapping for each input channel i, wherein the mapping index is the downmix channel j into which only the input channel i is downmixed, or if the mapping index is greater than the number of downmix channels. Thus, there is a map index m _{ICLD, i} , which is used to determine P _{dmx, i} as follows:

4.2.3. Parameter quantization

Examples of quantizing parameters 220 to obtain quantization parameters 224 may be performed, for example, by the parameter quantization unit 222 of FIG. 2b and 4.

Once the parameter set 220 is calculated, which means either the covariance matrices {C _x , C _y } or the ICC and ICLD ξ, X, they are quantized. The choice of quantizer may represent a trade-off between the quality and amount of data to be transmitted, but there is no restriction on the quantizer used.

As an example, in this case, ICC and ICLD are used; one may be a nonlinear quantizer comprising 10 quantization steps in the interval [-1,1] for ICC, and the other may be a nonlinear quantizer comprising 20 quantization steps in the interval [-30,30] for ICLD.

Additionally, as an implementation optimization, one may choose to downsample the transmitted parameters, which means that the quantized parameters 224 are used every two or more frames in a line.

In an aspect, a subset of the parameters transmitted in the current frame is signaled by the parameter frame index in the bit stream.

4.2.4. Processing of transition parts, downsampled parameters

Some examples explained below can be understood as those shown in FIGS. 5, which in turn may be an example of block 214 of FIG. 1 and 2d.

In the case of downsampled parameter sets (eg, obtained in block 265 in FIG. 5), i.e. the parameter set 220 for a subset of parameter bands may be used for more than one processed frame, transient parts that appear in more than one subset cannot be preserved in terms of localization and coherence. Therefore, it may be advantageous to send parameters for all frequency bands in such a frame. This special type of parameter frame, for example, can be signaled by means of a flag in the bit stream.

In an aspect, the transition portion detection at 258 is used to detect such transition portions in the signal 212. The position of the transition portion in the current frame may also be detected. The degree of temporal granularity may preferably be related to the degree of temporal granularity of the filter bank 214 being used, such that each transition portion position may correspond to an interval or group of intervals of the filter bank 214. The intervals for calculating the covariance matrices C _y and C _x are then selected based on the position of the transition portion, for example, only using intervals from the interval containing the transition portion to the end of the current frame.

The transient detector (or transient analyzer 258) may be a transient detector also used in encoding the downmix signal 212, such as the time domain transient detector of a basic IVAS encoder. Therefore, the example of FIG. 5 may also be applied above with respect to downmix calculation block 244.

In the example, the presence of a transition is encoded using a single bit (for example: "1", which means "there is a transition in the frame" versus "0", which means: "there is no transition in the frame"), and if the transition part is detected, in addition, the position of the transition part is encoded and/or transmitted as a coded field 261 (information regarding the transition part) in the bit stream 248 to enable similar processing in the decoder 300.

If a transition portion is detected and all frequency bands must be transmitted (eg, via signaling), sending parameters 220 using normal segment grouping may result in a burst of data rate required to transmit parameters 220 as ancillary information 228 in stream 248 bits Moreover, time resolution is more important than frequency resolution. Therefore, it may be advantageous, at block 265, to change the grouping of segments for such a frame so as to have fewer frequency bands to transmit (eg, from a large number of frequency bands in signal version 264 to fewer frequency bands in signal version 266 ). The example uses such a different grouping of segments, for example, by combining two adjacent frequency bands across all frequency bands for a normal downsampling factor of 2 for the parameters. In general terms, the presence of a transition part implies that the covariance matrices themselves are expected to differ significantly before and after the transition part. To avoid artifacts for intervals before a transition, only the transition interval itself and all subsequent intervals until the end of the frame can be considered. It is also based on the assumption that, in advance, the signal is sufficiently stationary that the information and mixing rules that are learned for the previous frame can also be used for the intervals preceding the transition portion.

More generally, the encoder may be configured to determine in which frame interval there is a transition portion, and encode link layer information (220) and source signal correlation (212, y) associated with the interval in which the transition portion is present, and/or with subsequent intervals in the frame, without encoding the link layer information (220) and the correlation of the original signal (212, y) associated with the intervals preceding the transition portion.

Likewise, the decoder may (eg, at block 380), when the presence and position of a transition portion in one frame are signaled (261):

- associate current link layer and correlation information (220) with the interval in which there is a transition part, and/or with subsequent intervals in the frame; And

- associate, with the frame interval preceding the interval in which there is a transition part, link layer and correlation information (220) of the previous interval.

Another important aspect of the transition portion is that if it is determined that a transition portion is present in the current frame, smoothing operations are no longer performed on the current frame. In the case of the transition part, no smoothing is performed on C _y and C _x , but C _yR and C _x from the current frame are used when calculating the mixing matrices.

4.2.5. Entropy coding

Entropy encoding module 226 (bitstream recording module) may be the last encoder module; its purpose is to convert the quantized values obtained previously into a binary bit stream, also called "auxiliary information".

The method used to encode the values may be, for example, Huffman encoding [6] or delta encoding. The encoding method is not critical and only affects the final bit rate; one should adapt the encoding method depending on the bit rates it wants to achieve.

Several implementation optimizations can be made to reduce the 248-bit stream size. As an example, a switching mechanism may be implemented that switches from one encoding scheme to another depending on which one is more efficient in terms of bitstream size.

For example, parameters may be delta encoded along the frequency axis for one frame and the resulting sequence of delta indices entropy encoded by a band encoder.

Additionally, in the case of parameter downsampling, also by way of example, a mechanism may be implemented to transmit in each frame only a subset of parameter bands for continuous data transmission.

These examples require overhead bits to signal the decoder-related aspect of encoder-side processing.

4.2.6. Calculation for downmixing

The downmix portion of the processing 244 may be simple, yet, in some examples, extremely important. The downmix used in the invention may be passive, meaning that the way it is calculated remains the same during processing and is independent of the signal or its characteristics at a particular time. However, it should be understood that the calculation for downmixing at 244 can be extended to an active calculation (eg, as described in [7]).

The downmix signal 246 may be calculated at two different locations:

- The first time for parameter estimation (see 4.2.2), on the encoder side, since this may be necessary (in some examples) to calculate the covariance matrix C _x .

- A second time on the encoder side, between encoder 200 and decoder 300 (in the time domain), with the downmixed signal 246 encoded and/or transmitted to decoder 300 and used as a basis for synthesis in module 334.

As an example, in the case of stereo downmix for a 5.1 input signal, the downmix signal can be calculated as follows:

- The downmix left channel is the sum of the left channel, the surround left channel and the center channel.

The right downmix channel is the sum of the right channel, the right surround channel and the center channel. Alternatively, in the case of mono downmix for a 5.1 input signal, the downmix signal is calculated as the sum of each channel of the multi-channel stream.

In examples, each channel of the downmix signal 246 may be obtained as a linear combination of the channels of the original signal 212, for example, with constant parameters, thereby implementing passive downmixing.

The calculation of downmixed signals can be expanded and adapted to additional speaker configurations according to processing needs.

Aspect 3: Low Latency Processing Using Passive Downmixing and Low Latency Filter Banks

The present invention can provide low latency processing through the use of passive downmixing, such as the processing described above for a 5.1 input signal and a low latency filter bank. Using these two elements, it is possible to achieve delays below 5 milliseconds between encoder 200 and decoder 300.

4.3. Decoder

The task of the decoder is to synthesize an output audio signal (336, 340, yR) in a particular speaker configuration by using a coded (eg, transmitted) downmix signal (246, 324) and coded auxiliary information 228. The decoder 300 may render output audio signals (334 , 240, yR) through the same speaker configuration as the speaker configuration used for the input signal (212, y), or through a different speaker configuration. Without loss of generality, the input and output speaker configurations are assumed to be the same (but may differ in the examples). This section further describes the various modules that may comprise the decoder 300.

Fig. 3a and 3b illustrate a detailed overview of possible decoder processing. It is important to note that at least some modules (particularly dashed border modules such as 320, 330, 338) in FIG. 3b may be discarded depending on the needs and requirements for a given application. Decoder 300 may be input by (eg, receiving) two sets of data from encoder 200:

- 228 auxiliary information with coded parameters (as described in 4.2.2)

- A downmixed signal (246, y), which may be in the time domain (as described in 4.2.6).

The encoded parameters 228 may need to be first decoded (eg, by input module 312), eg, using the decoding method previously used. Once this step has been completed, relevant parameters for synthesis, such as covariance matrices, can be recovered. In parallel, the downmix signal (246, x) can be processed through several modules: first, an analysis filter bank 320 (see also 4.2.1) can be used to obtain a frequency domain version 324 of the downmix signal 246. The prototype signal 328 can then be calculated (see also 4.3.3), and an additional decorrelation step (at 330) can be performed (see also 4.3.4). The key point of synthesis is the synthesis engine 334, which uses the covariance matrices (e.g., reconstructed in block 316) and the prototype signal (328 or 332) as an input signal and produces a final signal 336 as an output signal (see also 4.3.5) . Finally, a final step in synthesis filter bank 338 may be performed (eg, if analysis filter bank 320 was previously used), which produces a time domain output signal 340.

4.3.1. Entropy Decoding (e.g. block 312)

Entropy decoding at block 312 (input interface) may provide the ability to obtain the quantized parameters 314 previously obtained in 4. Decoding the 248 bit stream can be understood as a simple operation; the 248 bit stream can be read according to the encoding method used in 4.2.5 and then decoded.

From an implementation point of view, the 248 bit stream may contain overhead bits, which are not data, but which indicate some processing behavior on the encoder side.

For example, the first two bits used may indicate which encoding method is used in case the encoder 200 has the ability to switch between multiple encoding methods. The next bit can also be used to describe which parameter bandwidths are currently being transmitted.

Other information that may be encoded in the auxiliary information of the bit stream 248 may include a flag indicating a transition portion and a field 261 indicating in which frame slot there is a transition portion.

4.3.2. Restoring settings

Restoration of parameters can be performed, for example, by block 316 and/or module 402 for calculating mixing rules.

The purpose of this parameter recovery is to recover the covariance matrices C _x and C _y (or more generally, the covariance information associated with the downmix signal 246 and the level and correlation information of the original signal) from the downmixed signal 246 and/or from auxiliary information 228 (or a version thereof represented by quantized parameters 314). These covariance matrices C _x and C _y may be required for synthesis because they are matrices that effectively describe the multi-channel signal 246.

Restoring parameters in module 316 may be a two-step process:

First, the matrix C _x (or more generally, the covariance information associated with the downmix signal 246) is recalculated from the downmix signal 246 (this step may be omitted in cases in which the covariance information associated with the downmix signal 246 is is actually encoded in the auxiliary information 228 of the 248 bit stream); And

- then, the matrix C _y (or more generally, the level and correlation information of the original signal 212) can be reconstructed, for example, using at least part of the transmitted parameters and C _x or, more generally, covariance information associated with the downmix signal 246 ( this step may be eliminated in cases in which the level and correlation information of the original signal 212 is actually encoded in the auxiliary information 228 of the bit stream 248).

It should be noted that, in some examples, for each frame, the covariance matrix C _x of the current frame can be smoothed using linear combination with the reconstructed covariance matrix preceding the current frame, for example, by summing, averaging, etc. For example, in the t-th frame, the final covariance to be used for equation (4) may take into account the target covariance recovered for the previous frame, for example:

.

However, if it is determined that there is a transition portion in the current frame, smoothing operations are no longer performed on the current frame. In the case of a transition part, no smoothing is performed, C _x from the current frame is used.

Below is a general overview of the process.

Note: As for the encoder, the processing here can be performed based on the parameter frequency bands independently for each frequency band, for clarity, the processing is described below only for one specific frequency band and the notation adapted accordingly.

Aspect 4a. Restoring parameters in case covariance matrices are transmitted

For this aspect, it is assumed that the encoded (eg, transmitted) parameters in the auxiliary information 228 (the covariance matrix associated with the downmix signal 246, and the link layer and correlation information of the original signal 212) are covariance matrices (or a subset thereof) as specified in aspect 2a. However, in some examples, the covariance matrix associated with the downmix signal 246 and/or the link layer and correlation information of the original signal 212 may be implemented by other information.

If the full covariance matrices C _x and C _y are encoded (eg, transmitted), there is no subsequent processing to be processed in block 318 (and block 318 may therefore be eliminated in such examples). If only a subset of at least one of these matrices is encoded (eg, transmitted), missing values must be estimated. The final covariance matrices used in the synthesis engine 334 (or more specifically, in the synthesis processor 404) must consist of encoded (eg, transmitted) values 228 and estimated decoder-side values. For example, if only some elements of the matrix C _y are encoded in the auxiliary information 228 of the 248 bit stream, the remaining elements of C _y are evaluated here.

For the covariance matrix C _x of the downmixed signal 246, missing values can be calculated by using the downmixed signal 246 on the decoder side and apply Equation (1).

In an aspect in which the presence and position of the transition portion is transmitted or encoded, equal intervals are used to calculate the covariance matrix C _x of the downmixed signal 246, similar to the encoder side.

For the covariance matrix C _y , missing values can be calculated, in the first estimation, as follows:

(4),

- Where:

- is an estimate of the covariance matrix of the original signal 212 (it is an example of an estimated version of the original link layer information and correlation)

- Q is the so-called prototype matrix (prototype rule, evaluation rule) which describes the relationship between the downmix and the original signal (see also 4.3.3) (it is an example of a prototype rule),

- C _x is the covariance matrix of the downmix signal (it is an example of the covariance information of the downmix signal 212),

- * denotes conjugate transpose

Once these steps are completed, the covariance matrices are obtained again and can be used for the final synthesis.

Aspect 4b. Restoring parameters if ICC and ICLD are transmitted

For this aspect, it can be assumed that the encoded (eg, transmitted) parameters in the auxiliary information 228 are ICC and ICLD (or a subset thereof), as defined in aspect 2b.

In this case, it may first be necessary to recalculate the covariance matrix C _x . This may be accomplished by using the decoder side downmix signal 212 and applying equation (1).

In an aspect in which the presence and position of the transition portion is transmitted, equal intervals are used to calculate the covariance matrix C _x of the downmixed signal, similar to an encoder. The covariance matrix C _y can then be recomputed from the ICC and ICLD; this operation can be performed as follows:

The energy (also known as level) of each channel of a multi-channel input signal can be obtained. These energies are extracted using the transmitted ICLDs and the following formula:

(5)

- Where:

- where α _i is a weighting factor associated with the expected proportion of channel energy in the downmix, this weighting factor being fixed for a particular input speaker configuration and known in the encoder and decoder. In the case of an implementation specifying a mapping for each input channel i, wherein the mapping index is the downmix channel j into which only the input channel i is downmixed, or if the mapping index is greater than the number of downmix channels. Thus, there is a map index m _{ICLD, i} , which is used to determine P _{dmx, i} as follows:

The notation systems are the same as those used in parameter estimation in 4.2.3.

These energies can be used to normalize the estimated _Cy . In case not all ICCs are transmitted from the encoder side, an estimate of C _y may be calculated for the values not transmitted. Estimated covariance matrix can be obtained using the prototype matrix Q and the covariance matrix C _x using equation (4).

This estimation of the covariance matrix results in an estimation of the ICC matrix, for which the index term (i, j) can be given as follows:

(6)

Thus, the “reconstructed” matrix can be specified as follows:

(7)

- Where:

- The subscript R indicates the reconstructed matrix (which is an example of a reconstructed version of the original level and correlation information)

- Ensemble corresponds to all pairs (i, j) that are decoded (eg, passed from the encoder to the decoder) in the auxiliary information 228.

In examples, ξ _i , _j may be preferred over , due to the fact that is less accurate than the encoded value ξ _i , _j .

Finally, from this reconstructed ICC matrix, the reconstructed covariance matrix may be the derived C _yR . This matrix can be obtained by applying the energies obtained in equation (5) to the reconstructed ICC matrix, thus for indices (i, j):

(8)

In case the complete ICC matrix is transmitted, only equations (5) and (8) are required. The previous paragraphs illustrate one approach for recovering missing parameters, other approaches may be used, and the proposed method is not unique.

From the example in aspect 1b using signal 5.1, it can be noted that the values that are not transmitted are values that must be evaluated at the decoder side.

The covariance matrices C _x and C _yR can now be obtained. It is important to note that the reconstructed matrix C _yR may be an estimate of the covariance matrix C _y of the input signal 212. A compromise of the present invention may be to have an estimate of the decoder side covariance matrix reasonably close to the original, while also transmitting as little as possible number of parameters. These matrices may be required for the final synthesis, which is illustrated in 4.3.5.

It should be noted that, in some examples, for each frame, the reconstructed covariance matrix of the current frame can be smoothed using a linear combination with the reconstructed covariance matrix preceding the current frame, for example, by summing, averaging, etc. For example, at the t-th frame, the final covariance to be used for synthesis may take into account the target covariance recovered from the previous frame, for example:

However, in the case of the transition part, no smoothing is performed, and C _yR for the current frame is used when calculating the mixing matrices.

It should also be noted that in some examples, for each frame, the unsmoothed downmix channel covariance matrix _Cx is used for parameter recovery, while the smoothed covariance matrix _Cx ,t, as described in section 4.2.3, is used for synthesis.

Fig. 8a resumes the operation to obtain the covariance matrices C _x and C _yR in the decoder 300 (eg, performed in blocks 386 or 316...). In the blocks of Fig. 8a, between the brackets, the equation that is adjusted by the specific block is also indicated. As can be seen, the covariance estimator 384, through equation (1), allows the covariance C _x of the downmix signal 324 (or its reduced-bandwidth version 385) to be achieved. The first covariance estimator 384', through the use of equation (4) and a Q rule of the appropriate type, achieves the first estimate covariance C _y . Next, covariance-to-coherence conversion unit 390 obtains coherences ξ by applying equation (6). Thereafter, the replacement ICC block 392, by adjusting equation (7), selects between the estimated ICC (ξ) and the ICC signaled in the auxiliary information 228 of the 348 bit stream. The selected coherences ξ _R are then input to energy application block 394, which applies energy according to ICLD (Xi). The target covariance matrix C _yR is then passed to the mixing rules calculation module 402 or the covariance synthesis block 388 of FIG. 3a or to the mixing rules calculation module of FIG. 3c or synthesis engine 344 of FIG. 3b.

4.3.3. Calculation of prototype signals (block 326)

The purpose of the prototype signal processing module 326 is to generate the downmix signal 212 (or frequency domain version 324 thereof) in such a way that it can be used by the synthesis engine 334 (see 4.3.5). The prototype signal processing module 326 may perform upmixing of the downmixed signal. The calculation of the prototype signal 328 may be performed by the prototype signal processing module 326 by multiplying the downmixed signal 212 (or 324) by the so-called prototype matrix Q:

Yp=XQ (9),

- Where:

- Q is a prototype matrix (which is an example of a prototype rule),

- X is a downmixed signal (212 or 324)

- Yp is a prototype signal (328).

The manner in which the prototype matrix is installed may be processing dependent and may be set to suit the application requirement. The only limitation may be that the number of channels of the prototype signal 328 must be equal to the required number of output channels; this directly limits the size of the prototype matrix. For example, Q may be a matrix having a number of lines that represents the number of channels of the downmix signal (212, 324) and a number of columns that represents the number of channels of the final synthesis output signal (332, 340).

As an example, in the case of 5.1 or 5.0 signals, the prototype matrix could be set up as follows:

It should be noted that the prototype matrix can be pre-defined and fixed. For example, Q may be the same for all frames, but may be different for different frequency bands. In addition, different Qs are provided for different relationships between the number of channels of the downmix signal and the number of channels of the synthesis signal. The Q may be selected from a plurality of pre-stored Qs, for example, based on a particular number of downmix channels and a particular number of synthesis channels.

Aspect 5: Restoring parameters if the output speaker configuration differs from the input speaker configuration

One application of the present invention is to generate an output signal 336 or 340 from a speaker configuration that is different from the original signal 212 (meaning, for example, more or fewer speakers).

To do this, the prototype matrix should be modified accordingly. In this situation, the prototype signal obtained using equation (9) must contain as many channels as the output speaker configuration contains. For example, if there are 5 channel signals as an input signal (on the signal side 212), and it is desired to obtain 7 channel signals as an output signal (on the signal side 336), then the prototype signal should already contain 7 channels.

Thereafter, the covariance matrix estimate in equation (4) still remains and must still be used to estimate covariance parameters for channels that are not present in the input signal 212.

The transmitted parameters 228 between the encoder and decoder are still relevant, and equation (7) can also still be used. More precisely, encoded (eg, transmitted) parameters should be assigned to channel pairs that are as close as possible, in terms of geometry, to the original configuration. Essentially, an adaptation operation is required.

For example, if the encoder side evaluates the ICC value between one speaker on the right and one speaker on the left, this value may be assigned to channel pairs of the output configuration that have the same left and right positions; in case the geometry is different, this value can be assigned to the pair of loudspeakers whose positions are as close as possible to the original pair.

Next, after the target covariance matrix C _y is obtained for the new output configuration, the rest of the processing is unchanged.

Accordingly, to adapt the target covariance matrix (C _yR ) to the number of synthesis channels, you can:

- use a prototype matrix Q, which is converted from the mentioned number of downmixing channels into a certain number of synthesis channels; it can be obtained through:

- adaptation of formula (9), so that the prototype signal has a certain number of synthesis channels;

- adaptation of formula (4), thereby estimating in the number of synthesis channels;

- maintaining formulas (5)-(8), which are therefore obtained in the number of original channels;

- but assigning groups of source channels (for example, pairs of source channels) to single synthesis channels (for example, choosing assignments in terms of geometry) or vice versa.

In fig. 8b is an example which is a version of FIG. 8a, which indicates the number of channels of a certain matrix and vectors. When the ICCs (derived from the 348 bit stream auxiliary information 228) are applied to the ICC matrix at 392, groups of source channels (eg, pairs of source channels) to single synthesis channels (eg, selecting assignments in terms of geometry) or vice versa.

Another possibility for generating a target covariance matrix for a number of output channels that is different from the number of input channels is to first generate a target covariance matrix for the number of input channels (eg, the number of source channels of input signal 212) and then adapt this first target covariance matrix to the number of synthesis channels, obtaining a second target covariance matrix corresponding to the number of output channels. This may be accomplished by applying an up- or down-mixing rule, for example a matrix containing coefficients for the combination of certain input (source) channels with output channels, to the first target covariance matrix C _yR , and in a second step, applying this matrix C _yR to the transmitted powers input channel (ICLD) and obtaining a vector of channel powers for the number of output channels (synthesis) and adjusting the first target covariance matrix according to the vectors to obtain a second target covariance matrix with the requested number of synthesis channels. This adjusted second target covariance matrix can now be used in synthesis. An example of this is shown in Fig. 8c, which is a version of FIG. 8a, in which blocks 390-394 control the reconstruction of the target covariance matrix C _yR so as to have a source signal source channel count of 212. Thereafter, in block 395, the prototype signal QN (which should be converted to a number of synthesis channels) and the ICLD vector can be applied . In particular, block 386 of FIG. 8c is the same as block 386 of FIG. 8a, except for the fact that in FIG. 8c, the number of channels of the recovered target covariance is exactly equal to the number of original channels of the input signal 212 (and in FIG. 8a, for generality, the recovered target covariance has a number of synthesis channels).

4.3.4. Decorrelation

The purpose of the decorrelation module 330 is to reduce the amount of correlation between each channel of the prototype signal. A highly correlated speaker signal can introduce phantom sources and degrade the quality and spatial properties of the multi-channel output signal. This step is optional and may or may not be performed as required by the application. In the present invention, decorrelation is used before the synthesis mechanism. As an example, an all-frequency frequency decorrelator can be used.

It should be noted that regarding the MPEG surround sound standard:

The MPEG surround sound standard, according to the prior art, provides for the use of so-called “mixing matrices” (referred to as M1 and M2 in the standard). The M1 matrix controls how available downmixed signals are input into the decorrelators. The M2 matrix describes how the direct and decorrelated signals must be combined to form the output signal.

While there may be similarities to the prototype matrix defined in 4.3.3, as well as the use of decorrelators described in this section, it is important to note that:

- The Q prototype matrix has a completely different function compared to the matrices used in the MPEG surround sound standard, the aspect of this matrix is to generate the prototype signal. The target of this prototype signal must be input into the synthesis engine.

- The prototype matrix does not intend to prepare downmixed signals for decorrelators and can be adapted depending on the requirements and the target application. For example, the prototype matrix may generate a prototype signal for an output speaker configuration that is larger than the input configuration.

- The use of decorrelators in the proposed invention is not mandatory; processing is based on the use of a covariance matrix in the synthesis mechanism (see also 5.1).

- The proposed invention does not generate an output signal through a combined direct and decorrelated signal.

- The calculation of M1 and M2 is highly dependent on the tree structure, the various coefficients of these matrices are case dependent in terms of structure. This is not the case in the proposed invention, the processing is agnostic with respect to the downmix calculation (see also 5.2), and conceptually the proposed processing aims to consider the relationship between each channel only instead of pairs of channels, as this can be done with a tree structure.

Therefore, the present invention differs from the MPEG surround sound standard of the prior art.

4.3.5. Synthesis mechanism, matrix calculation

The final stage of the decoder includes a synthesis engine 334 or a synthesis processor 402 (and optionally an additional synthesis filter bank 338). The task of the synthesis engine 334 is to generate the final output signal 336 relative to certain constraints. The synthesis engine 334 may calculate an output signal 336 whose characteristics are constrained by the input parameters. In the present invention, the input parameters 318 of the synthesis engine 338, with the exception of the prototype signal 328 (or 332), are covariance matrices C _x and C _y . In particular, C _yR is called the “target covariance matrix” because the characteristics of the output signals should be as close as possible to the characteristics specified by C _y (the estimated version and the pre-constructed version of the target covariance matrix are shown to be explained).

The synthesis engine 334 that may be used is not unique; as an example, covariance synthesis may be used from the prior art source [8], which is incorporated herein by reference. Another synthesis engine 333 that may be used is the synthesis engine described in the DirAC processing in [2].

The output of the synthesis engine 334 may require additional processing through a synthesis filter bank 338.

As a final result, a multi-channel time domain output signal 340 is obtained.

Aspect 6: High Quality Outputs Using “Covariance Synthesis”

As mentioned above, the synthesis engine 334 used is not unique, and any engine that uses the passed parameters or a subset thereof can be used. However, one aspect of the present invention may be to provide high quality output signals 336, for example, through the use of covariance synthesis [8].

This synthesis method aims to calculate an output signal 336, the characteristics of which are specified by the covariance matrix C _yR . To do this, so-called optimal mixing matrices are calculated, and these matrices must combine the prototype signal 328 into the final output signal 336 and must provide a mathematically optimal result, taking into account the target covariance matrix C _yR .

The mixing matrix M is a matrix that converts the prototype signal x _p into an output signal yR (336) through the relationship .

The mixing matrix may also be a matrix that converts the downmix signal x into an output signal through an interconnection . From this relationship we can also deduce .

In the present processing, C _yR and C _x may in some examples already be known (since they respectively represent the target covariance matrix C _yR and the covariance matrix C _x of the downmix signal 246).

One solution from a mathematical point of view is given by , where Ky and represent all matrices obtained by performing singular value decomposition on C _x and C _yR . For P, it represents a free parameter here, but the optimal solution (from a perceptual point of view for the listener) can be found with respect to the constraint prescribed by the prototype matrix Q. Mathematical confirmation of what is stated here is contained in [8].

This synthesis engine 334 provides a high quality output signal 336 because the approach is designed to provide an optimal mathematical solution to the output signal reconstruction problem.

In a non-mathematical sense, it is important to understand that covariance matrices represent the energy relationships between the various channels of a multi-channel audio signal. A matrix C _y for the original multi-channel signal 212 and a matrix C _x for the down-mixed multi-channel signal 246. Each value of these matrices transforms the energy relationship between two channels of the multi-channel stream.

Therefore, the philosophy behind covariance synthesis is to generate a signal whose characteristics are controlled by the target covariance matrix C _yR . This matrix C _yR is calculated in such a way that it describes the original input signal 212 (or the output signal that is desired to be obtained, in case it is different from the input signal). Then, due to the presence of these elements, covariance synthesis should optimally reduce the prototype signal to form the final output signal.

In a further aspect, the mixing matrix used for interval synthesis is a combination of the current frame's mixing matrix M, and the previous frame's mixing matrix Mp, so as to ensure smooth synthesis, eg, linear interpolation based on the index of the interval in the current frame.

In a further aspect, in which the presence and position of the transition portion is transmitted, a previous mixing matrix Mp is used for all intervals before the transition portion position, and a mixing matrix M is used for the interval containing the transition portion position and all subsequent intervals in the current frame. It should be noted that, in some examples, for each frame or interval, the mixing matrix of the current frame or interval can be smoothed using a linear combination with the mixing matrix used for the previous frame or interval, for example, by summing, averaging, etc. Suppose that, for the current frame t, interval s, bandwidth i of the output signal is obtained by , where M _{s, i} is a combination of the Mt-1, _i mixing matrix used for the previous frame, and M _{t, i} , which is the mixing matrix calculated for the current frame, e.g. linear interpolation between them:

,

- where n _s is the number of slots in the frame (eg, 16), and t-1 and t indicate the previous and current frame. More generally, the mixing matrix M _{s, i} associated with each interval can be obtained by scaling along subsequent intervals of the current frame t the mixing matrix M _{t, i} calculated for the current frame by a boosting factor, and by adding, along subsequent intervals of the current frame t, a mixing matrix Mt-1,i, scaled by a reduction factor. The coefficients can be linear.

It may be provided that in the case of a transition part (for example, signaled in information 261), not the current and past mixing matrices are combined, but the previous mixing matrix up to the interval containing the transition part and the current mixing matrix for the interval containing the transition part. part and all subsequent intervals until the end of the frame.

,

- where s is the interval index, i is the frequency band index, t and t-1 indicate the current and previous frame, and s _t is the interval containing the transition part.

Differences from the prior art source [8]

It is also important to note that the proposed invention goes beyond the scope of the method proposed in [8]. Notable differences include, but are not limited to:

- The target covariance matrix C _yR is calculated at the encoder side of the proposed processing.

- The target covariance matrix C _yR can also be calculated in another way (in the proposed invention, the covariance matrix is not the sum of the scattered and direct parts).

- Processing is not performed for each frequency band separately, but is grouped for parameter frequency bands (as mentioned in 0).

- From a more global point of view: covariance synthesis here represents only one block of the entire process and must be used in conjunction with all other elements on the decoder side.

4.3. Preferred aspects as a list

At least one of the following aspects may characterize the invention:

1. On the encoder side

a. Multi-channel audio input 246.

b. Converting signal 212 from time domain to frequency domain (216) using filter bank 214

c. Calculating the downmix signal 246 at block 244

d. From the original signal 212 and/or the downmix signal 246, estimate a first set of parameters to describe the multi-channel stream 246 (signal): covariance matrices C _x and/or Cy

e. Transmitting and/or encoding covariance matrices C _x and/or C _y directly, or calculating ICC and/or ICLD and transmitting them

f. Encoding of 228 transmitted parameters in a 248 bit stream using an appropriate encoding scheme

g. Calculation of a down-mixed signal 246 in the time domain

h. Transmission of auxiliary information (i.e. parameters) and down-mixed signal 246 in the time domain

2. On the decoder side

a. Decoding a 248 bit stream containing side information 228 and a downmix signal 246

b. (if necessary) Applying a filter bank 320 to the downmix signal 246 to obtain a frequency domain version 324 of the downmix signal 246

c. Recovering covariance matrices C _x and C _yR from previously decoded parameters 228 and downmix signal 246

d. Calculation of prototype signal 328 from downmix signal 246 (324)

e. (if necessary) Decorrelation of the prototype signal (in block 330)

f. Applying a synthesis engine 334 to a prototype signal using the reconstructed _Cx and _CyR .

g. (if necessary) Applying synthesis filter bank 338 to covariance synthesis output 336 334

h. Receiving 340 multi-channel output signal

4.5. Covariance synthesis

This section explains some of the technologies that may be implemented in the systems of FIGS. 1-3d. However, these techniques can also be implemented independently: for example, in some examples, the covariance calculation performed for FIGS. 8a-8c and in equations (1)-(8). Therefore, in some examples, if one refers to C _yR (the recovered target covariance), it may also be replaced by C _y (which may also be directly provided without recovery). Regardless, the technologies in this section can advantageously be used in conjunction with the technologies explained above.

Let us now turn to FIG. 4a-4d. Examples of covariance synthesis blocks 388a-388d are explained here. Blocks 388a-388d may implement, for example, block 388 of FIG. 3c to perform covariance synthesis. Blocks 388a-388d, for example, may be part of the synthesis processor 404 and the mixing rules calculation module 402 of the synthesis engine 334 and/or the parameter recovery unit 316 of FIG. 3a. In fig. 4a-4d, the downmix signal 324 is in the frequency domain (FD) (i.e., below the filter bank 320) and is indicated as X, while the synthesis signal 336 is also in the FD and is indicated as Y. However, it is possible to generalize these results, for example, in the time domain. It should be noted that each of the covariance synthesis blocks 388a-388d of FIG. 4a-4d may be referred to as "one distinct frequency band" (eg, after disaggregation at 380), and the covariance matrices C _x and C _yR (or other reconstructed information) may thereby be associated with one particular frequency band. Covariance synthesis may be performed, for example, frame by frame, in which case the covariance matrices C _x and C _yR (or other reconstructed information) are associated with one single frame (or multiple consecutive frames): therefore, covariance synthesis may be performed frame by frame or each set of frames .

In fig. 4a, the covariance synthesis block 388a may consist of a single energy-compensated optimal mixing block 600a in the absence of a correlation block. Essentially, one separate mixing matrix M is discovered, and the only significant operation that is additionally performed is the computation of the energy-compensated mixing matrix M'.

Fig. 4b shows a block 388b for covariance synthesis conditioned by [8]. The covariance synthesis block 388b may be capable of obtaining a synthesis signal 336 as a synthesis signal having a first principal component 336M and a second residual component 336R. Although the principal component 336M may be obtained in the mixing matrix of the optimal principal component 600b, for example, by discovering the mixing matrix MM from the covariance matrices C _x and C _yR and without decorrelators, the residual component 336R may be obtained in a different manner. MR should in principle satisfy the relationships . Typically, the resulting mixing matrix does not fully satisfy this, and the residual target covariance can be detected using . As can be seen, the downmix signal 324 may be extracted into path 610b (path 610b may be referred to as a "second path" parallel to the first path 610b' including block 600b). A prototype version 613b (indicated by Y _pR ) of the downmix signal 324 may be obtained in a prototype signal processing block (upmixer) 612b. For example, an equation such as equation (9) can be used, i.e.:

Examples of Q (prototype matrix or upmix matrix) are provided herein. Below block 612b there is a decorrelator 614b for decorrelating the prototype signal 613b to obtain a decorrelated signal 615b (also indicated by ). From the decorrelated signal 615b covariance matrix decorrelated signal (615b) is evaluated in block 616b. By using the covariance matrix decorrelated signal as equivalent to _Cx of the principal components mixing and Cr as the target covariance in another optimal mixing block, the residual component 336R of the synthesis signal 336 may be obtained in the optimal residual component mixing matrix processing block 618b. The optimal residual component mixing matrix processing unit 618b may be implemented such that a mixing matrix MR is generated to mix the decorrelated signal 615b and obtain a residual component 336R of the synthesis signal 336 (for a particular frequency band). In adder 620b, the residual component 336R is added to the main component 336M (paths 610b and 610b' are therefore combined in adder 620b).

Fig. 4c shows an example of covariance synthesis 388c alternative to covariance synthesis 388b of FIG. 4b. The covariance synthesis block 388c allows the synthesis signal 336 to be obtained as a Y signal having a first principal component 336M' and a second residual component 336R'. Although the principal component 336M' may be obtained in the optimal principal component mixing matrix 600c, for example, by discovering the mixing matrix MM from the covariance matrices C _x and C _yR (or other Cy information 220) and without correlators, the residual component 336R' may be obtained in another manner. The downmix signal 324 may be extracted to path 610c (path 610c may be referred to as a “second path” parallel to the first path 610c' including block 600c). A prototype version 613c of the downmix signal 324 may be obtained in a downmixer 612c (an upmixer) by using a prototype Q matrix (e.g., a matrix that mixes the upmixed signal 234 into a version 613c of the downmixed signal 234 in the number of channels which is the number of synthesis channels). For example, an equation such as equation (9) may be used. Examples of Q are provided herein. A decorrelator 614c may be provided below block 612c. In some examples, the first path does not have a decorrelator, while the second path has a decorrelator.

Decorrelator 614c may provide decorrelated signal 615c (also indicated by ). However, unlike the technology used in covariance synthesis block 388b of FIG. 4b, in the covariance synthesis block 388c of FIG. 4c, covariance matrix decorrelated signal 615c is not estimated from decorrelated signal 615c ( ). On the contrary, the covariance matrix decorrelated signal 615c is obtained (at block 616c) from:

- the covariance matrix C _x of the downmix signal 324 (eg, estimated in block 384 of FIG. 3c and/or using equation (1)); And

- prototype matrix Q.

By using the covariance matrix , estimated from the covariance matrix _Cx of the downmix signal 324, as equivalent to the principal component mixing matrix _Cx and Cr as the target covariance matrix, the residual component 336R' of the synthesis signal 336 is obtained in the optimal residual component mixing matrix processing unit 618c. The optimal residual component mixing matrix processing unit 618c may be implemented such that a residual component mixing matrix MR is generated so as to obtain a residual component 336R' by mixing the decorrelated signal 615c according to the residual component mixing matrix MR. In summing block 620c, the residual component 336R' is summed with the main component 336M' to produce synthesis signal 336 (paths 610c and 610c' are therefore combined in summing block 620c).

In some examples, the residual component 336R or 336R' is not always or necessarily calculated (and path 610b or 610c is not always used). In some examples, although for some frequency bands the covariance synthesis is performed without calculating the residual signal 336R or 336R' for other frequency bands in the same frame, the covariance synthesis is also processed taking into account the residual signal 336R or 336R'. Fig. 4d shows an example of a covariance synthesis block 388d, which may be a specific case of a covariance synthesis block 388b or 388c: here, the frequency band selector 630 may select or deselect (in a manner represented by switch 631) the calculation of the residual signal 336R or 336R'. For example, path 610b or 610c may be selectively activated by selector 630 for some frequency bands and disabled for other frequency bands. In particular, path 610b or 610c may be disabled for frequency bands at a predetermined threshold (eg, a fixed threshold), which may be a threshold (eg, maximum) that distinguishes between frequency bands for which the human ear is phase insensitive ( frequency bands above the threshold), and frequency bands for which the human ear is phase sensitive (frequency bands below the threshold), so that the residual component 336R or 336R' is not calculated for frequency bands below the threshold and is calculated for frequency bands with a frequency above the threshold value.

Example according to FIG. 4d may also be obtained by substituting block 600b or 600c in place of block 600a of FIG. 4a and substituting block 610b or 610c for block 388b for covariance synthesis in FIG. 4b or the covariance synthesis block 388c of FIG. 4c.

Here is some guidance as to how the mixing rule (matrix) can be obtained in any of blocks 338, 402 (or 404), 600a, 600b, 600c, etc. As explained above, there are many methods for obtaining mixing matrices, but some of them are explained in more detail here.

Specifically, first reference is made to the covariance synthesis block 388b of FIG. 4b. In the optimal principal component mixing matrix processing unit 600c, the mixing matrix M for the principal component 336M of the synthesis signal 336 may be obtained, for example, from:

- covariance matrix C _y of the original signal 212 (C _y can be estimated using at least some of the formulas (6)-(8) explained above, see, for example, Fig. 8; it can have the so-called “target version” form C _yR , for example, estimated using formula (8)); And

- covariance matrix C _x of the downmix signal 246, 324 (C _y can be estimated, for example, using formula (1)).

For example, as proposed in [8], it is generally accepted to decompose the covariance matrices C _x and C _y , which are Hermitian and positive semidefinite, according to the following factorization:

Kx and Ky can be obtained, for example, by applying singular value decomposition (SVD) twice from C _x and C _y . For example:

- SVD in C _x can provide a matrix UC _x of singular vectors (for example, left singular vectors); And

- diagonal matrix SC _x singular values;

- so that Kx is obtained by multiplying UC _x by a diagonal matrix having in its entries the square roots of the values in the corresponding SC _x entries.

Additionally, SVD in C _y can provide:

- matrix VC _y of singular vectors (for example, right singular vectors); And

- diagonal matrix SC _y of singular values,

- so that Ky is obtained by multiplying UC _y by a diagonal matrix having, in its entries, the square roots of the values in the corresponding SC _y entries.

After this you can get the matrix a principal component mix that, when applied to the downmix signal 324, should produce a principal component 336M of the synthesis signal 336. Matrix mixing the main components can be obtained as follows:

If Kx is an irreversible matrix, the regularized inverse matrix can be obtained using known techniques and is replaced instead .

The parameter P is generally free, but it can be optimized. To achieve P, one can apply SVD to:

C _x (covariance matrix of downmix signal 324); And

(covariance matrix of prototype signal 613b).

Once the SVDs are executed, P can be obtained as follows:

Λ is a matrix having as many rows as the number of synthesis channels and as many columns as the number of downmix channels. Λ represents the identification data in the first square block and is padded with zeros in the remaining entries. The following explains how V and U are obtained from C _x And , and U are the singular vector matrices obtained from the SVD:

S is a diagonal singular value matrix, typically obtained through SVD. is a diagonal matrix that normalizes the energies per channel of the prototype signal (615b) in the synthesis signal energy y. To receive , first you need to calculate , i.e. covariance matrix of the prototype signal (614b). Then, to achieve from , diagonal values are normalized to the corresponding diagonal values C _y , thereby providing . An example is that diagonal entries are calculated as , Where are the values of the diagonal entries C _y , and are the values of the diagonal entries .

After it turns out that the covariance matrix Cr of the residual component is obtained as follows:

After obtaining Cr, a mixing matrix can be obtained for mixing the decorrelated signal 615b to obtain the residual signal 336R, and in the same optimal mixing, Cr has the same role as the role of C _yR in the main optimal mixing, and the covariance Decorrelated prototypes act as covariance C _x of the input signals in the underlying optimal mixing.

However, it should be understood that, compared to the technology of FIG. 4b, technology according to FIG. 4c presents some advantages. In some examples, the technology of FIG. 4c is the same as the technology of FIG. 4c at least for calculating the main matrix and for generating the main component of the synthesis signal. In contrast, the technology of FIG. 4c differs from the technology of FIG. 4b when calculating the residual mixing matrix and, generally speaking, to generate the residual component of the synthesis signal. Referring now to FIG. 11 in connection with FIG. 4c to calculate the residual mixing matrix. In the example of FIG. 4c, a frequency domain decorrelator 614c is used that decorrelates the prototype signal 613c but preserves the energies of the prototype signal 613b itself.

Moreover, in the example of FIG. 4c assumes (at least by approximation) that the decorrelated channels of decorrelated signal 615c are mutually incoherent, and as such, that all off-diagonal elements of the decorrelated signal covariance matrix are zero. Under both assumptions, one can simply estimate the covariance of the decorrelated prototypes from applying Q to C _x and apply only the main diagonal of this covariance (i.e., the energy of the prototype signals). This technology according to FIG. 4c is more efficient than the example of FIG. 4b, from the decorrelated signal 615b, in which it is necessary to perform the same frequency/slot aggregation that is already performed on C _x . Therefore, in the example of FIG. 4c, one can simply apply matrix multiplication of the already aggregated C _x . Therefore, the same mixing matrix is calculated for all frequency bands of the same aggregate group of frequency bands.

Thus, the covariance is 711 ( ) of the decorrelated signal can be estimated, at 710, using:

as the main diagonal of the matrix with all off-diagonal elements set to zero, which is used as the covariance input signals. In examples in which C _x is smoothed to perform synthesis of the synthesis signal principal component 336M', a technique may be used whereby the version of C _x that is used to calculate P _decorr is the unsmoothed C _x .

Now, the prototype Qr matrix must be used. However, it should be noted that, for the residual signal, Qr is an identity matrix. Knowledge of properties (diagonal matrix) and Qr (identity matrix) leads to further simplification when calculating the mixing matrix (at least one SVD may not be used), see the following technology and Matlab printout.

First, similar to the example in FIG. 4b, the residual target covariance matrix C _x (Hermitian positive semidefinite) from input signal 212 can be decomposed as . The Kr matrix can be obtained via SVD(702): SVD 702 applied to Cr produces:

- matrix UCr of singular vectors (for example, left singular vectors);

- diagonal matrix SCr of singular values;

- so that Kr is obtained (at 706) by multiplying UCr by a diagonal matrix having, in its entries, the square roots of the values in the corresponding SCr entries (the latter obtained at 704).

At this point, it may be theoretically possible to apply another SVD, this time to the covariance decorrelated prototypes.

However, in this example (Fig. 4c), a different path is chosen to reduce computational effort. , estimated from , is a diagonal matrix, and as such no SVD is required (the SVD of a diagonal matrix provides the singular values as a sorted vector of diagonal elements, and the left and right singular vectors simply indicate the sorting index). By calculating (at 712) the square root of each value in the diagonal entries , we get a diagonal matrix . This diagonal matrix is such that , with the advantage that SVD is not required to obtain . From diagonal covariance decorrelated signals, the estimated covariance matrix is calculated decorrelated signal 615c. But since the prototype matrix Qr (i.e. the identity data matrix), one can directly use to formulate as , Where are the values of the diagonal entries Cr, and are the values of the diagonal entries . is a diagonal matrix (obtained at 722) that normalizes the energies per channel of the decorrelated signal (615b) for the required synthesis signal energies y.

At this moment, you can multiply (by 734) on (also the result of 735 multiplied by 734 is called " "). Then (736), Kr is multiplied by to receive (those. = ). From , SVD (738) may be performed to obtain a left singular vector matrix U and a right singular vector matrix V. By multiplying (740) V and U*, the matrix P ( ). Finally (742), a mixing matrix MR for the residual signal can be obtained by applying:

,

- Where (obtained at 745) can be replaced by regularized inversion. MR can therefore be used in block 618c for residual mixing.

Here is the Matlab code to perform covariance synthesis as explained above. It should be noted that in this code, the asterisk (*) means multiplication and the apostrophe (') means Hermitian matrix.

% Calculate the residual mixing matrix

function[m]=ComputeMixingMatrixResidual(C_hat_y, Cr,reg_sx, reg_ghat)

EPS_=single(1e-15);% epsilon to exclude divisions by zero

num_outputs=size(Cr,1);

% Cy Decomposition

[U_Cr, S_Cr]=svd(Cr);

Kr=U_Cr*sqrt(S_Cr);

The %SVD of a diagonal matrix represents the ordered diagonal elements,

% you can skip the ordering and get Kx directly from C _x

K_hat_y=sqrt(diag(C_haty));

limit=max(K_hat_y)*reg_sx+EPS_;

S_hat_y_reg_diag=max(K_hat_y, limit);

% Formulation of regularized Kx

K_hat_y_reg_inverse=1./S_hat_y_reg_diag;

% Formulation of the G-hat normalization matrix

%Q is the identity matrix in the case of the residual/scattered part, so that:

% Q*C _x *Q'=C _x

Cy_hat_diag=diag(C_hat_y);

limit=max(Cy_hat_diag)*reg_ghat+EPS_;

Cy_hat_diag=max(Cy_hat_diag, limit);

G_hat=sqrt(diag(Cr)./Cy_hat_diag);

% Formulation of optimal P

% Kx, G_hat are diagonal matrices, Q is I, etc.

K_hat_y=K_hat_y.*G_hat;

for k=1:num_outputs

Ky_dash(k,:)=Kr(k,:)*K_hat_y(k);

end

[U,~,V]=svd(Ky_dash);

P=V*U';

% Formulation M

M=Kr*P;

for k=1:num_outputs

M(:,k)=M(:,k)*K_hat_y_reg_inverse(k);

end

end

An explanation is given here regarding the covariance synthesis of FIG. 4b and 4c. In some examples, two synthesis methods may be considered for each frequency band, with some frequency bands employing full synthesis including the residual path from FIG. 4b, for frequency bands typically above a certain frequency at which the human ear is phase insensitive, energy compensation is applied to achieve the required channel energies.

Thus, also in the example of FIG. 4b, for frequency bands below a certain (fixed, known to the decoder) frequency band limit (threshold value), full synthesis according to FIG. 4b may be executed (eg in the case of FIG. 4d). In the example of FIG. 4b, covariance decorrelated signal 615b is extracted directly from decorrelated signal 615b. In contrast, in the example of FIG. 4c, a frequency domain decorrelator 614c is used that decorrelates the prototype signal 613c but preserves the energies of the prototype signal 613b itself.

Additional factors:

--- In both examples of FIG. 4b and at 4c: in the first path (610b', 610c'), a mixing matrix MM is generated (at block 600b, 600c) by basing it on the covariance C _y of the original signal 212 and the covariance C _x of the downmix signal 324;

--- In both examples of FIG. 4b and 4c: in the second path (610b, 610c), a decorrelator is provided (614b, 614c), and a mixing matrix MR is generated (at block 618b, 618c) that must account for covariance decorrelated signal (616b, 616c); But

-ooo In the example of FIG. 4b, covariance the decorrelated signal (616b, 616c) is calculated, intuitively, using the decorrelated signal (616b, 616c) and weighted in the energies of the original channel y;

-ooo In the example of FIG. 4c, the covariance of the decorrelated signal (616b, 616c) is calculated, paradoxically, by estimating it from the matrix C _x and weighted by the energies of the original channel y.

It should be noted that the covariance matrix (C _yR ) may be the reconstructed target matrix explained above (eg, obtained from the link layer and correlation information 220 recorded in the auxiliary information 228 of the bit stream 248), and may therefore be considered associated with the covariance source signal 212. In any case, since it is to be used for synthesis signal 336, the covariance matrix (C _yR ) can also be considered to be the covariance associated with the synthesis signal. The same applies to the residual covariance matrix Cr, which can be understood as the residual covariance matrix (Cr) associated with the synthesis signal, and the main covariance matrix, which can be understood as the main covariance matrix associated with the synthesis signal.

5. Benefits

5.1. Reduced use of decorrelation and optimal use of the synthesis engine

Taking into account the proposed technology, as well as the parameters that are used for processing, and the way in which these parameters are combined with the synthesis engine 334, it is explained that the need for strong decorrelation of the audio signal (for example, in its version 328) is reduced, and also that the influence of decorrelation ( such as artifacts or degradation of spatial properties or degradation of signal quality) is reduced, if not eliminated, even in the absence of decorrelation module 330.

More specifically, as stated above, the processing decorrelation portion 330 is optional. In effect, the synthesis engine 334 takes care of the decorrelation of the signal 328 by using the target covariance matrix C _y (or a subset thereof) and ensures that the channels that make up the output signal 336 are properly decorrelated with each other. The values in the covariance matrix C _y represent the energy relationships between the various channels of a multichannel audio signal precisely because they are used as the target for synthesis.

In addition, encoded (eg, transmitted) parameters 228 (eg, in their version 314 or 318) combined with the synthesis engine 334 can provide a high quality output signal 336, given the fact that the synthesis engine 334 uses the target covariance matrix C _y to reproduce a multi-channel output signal 336 whose spatial characteristics and sound quality are as close as possible to the input signal 212.

5.2. Downmix-agnostic processing

In view of the proposed technology, as well as the manner in which the prototype signals 328 are computed and how they are used with the synthesis engine 334, it is explained here that the proposed decoder is agnostic with respect to the manner in which the downmixed signals 212 are computed in the encoder.

This means that the proposed invention in the decoder 300 can be performed independently of the manner in which the downmixed signals 246 are calculated in the encoder, and that the output quality of the signal 336 (or 340) is not based on a particular downmixing method.

5.3. Scalability of parameters

Taking into account the proposed technology, as well as the way in which the parameters (28, 314, 318) are calculated, and the way in which they are used with the synthesis engine 334, as well as the way in which they are evaluated on the decoder side, it is explained that the parameters used for descriptions of multi-channel audio signals are scalable in number and task.

Typically, only a subset of the parameters (eg, a subset of C _y and/or C _x , eg elements) evaluated at the encoder side are encoded (eg transmitted): this allows the bit rates used by the processing to be reduced. Therefore, the number of parameters (eg C _y and/or C _x elements) encoded (eg transmitted) can be scalable, taking into account the fact that non-transmitted parameters are recovered at the decoder side. This makes it possible to scale the entire processing in terms of output quality and bit rates: the more parameters transmitted, the better the output quality, and vice versa.

In addition, these parameters (eg, C _y and/or C _x or elements thereof) are scalable by purpose, meaning that they can be controlled by user input to change the characteristics of the output multi-channel signal. In addition, these parameters can be calculated for each frequency band and therefore provide scalable frequency resolution.

For example, it may be possible to decide to suppress one speaker in the output signal (336, 340), and because of this, it may be possible to directly manipulate parameters on the decoder side in order to achieve such a conversion.

5.4. Output configuration flexibility

Given the proposed technology, as well as the synthesis engine 334 used and the flexibility of the parameters (eg, C _y and/or C _x or elements thereof), it is explained here that the proposed invention allows for a wide range of rendering capabilities relative to the output configuration.

More precisely, the output configuration need not be the same as the input configuration. The reconstructed target covariance matrix that is fed to the synthesis engine can be manipulated to generate an output signal in a speaker configuration that is larger, smaller, or simply has a different geometry than the original geometry. This is possible due to the parameters that are transmitted and also because the proposed system is agnostic with respect to the downmixed signal (see also 5.2).

For these reasons, it is explained that the present invention is flexible in terms of speaker output configuration.

5. Some examples of prototype matrices

Below are the tables already for 5.1, but excluding LFE, since LFE is also included in the processing (with only one ICC for the LFE/C relationship and ICLD for LFE sent only in the smallest parameter bandwidth, and set to 1 and zero, respectively, for all other frequency bands in decoder-side synthesis). The channel naming and orders are in accordance with the CICP contained in ISO/IEC 23091-3 "Information technology - Coding independent code-points - Part 3: Audio", Q is always used both as a prototype matrix in the decoder and as a downmix matrix in coder.

5.1 (CICP6). α _i should be used to calculate ICLD.

7.1. (CICP12)

5.1+4. (CICP16)

7.1+4. (CICP19)

6. Methods

Although the above technologies are mainly explained as components or functional devices, the invention can also be implemented as methods. The blocks and elements explained above can also be understood as steps and/or phases of methods.

For example, a decoding method is provided for generating a synthesis signal from a downmix signal, the synthesis signal having a number of synthesis channels, the method comprising:

- receiving a downmix signal (246, x), wherein the downmix signal (246, x) has a number of downmix channels and auxiliary information (228), wherein the auxiliary information (228) includes:

- information (220) of the link level and correlation of the original signal (212, y), and the original signal (212, y) has a certain number of original channels;

- generating a synthesis signal using channel level information (220) and correlation of the original signal (212, y) and covariance information (C _x ) associated with the signal (246, x).

The decoding method may comprise at least one of the following steps:

- calculating a prototype signal from the downmix signal (246, x), wherein the prototype signal has a certain number of synthesis channels;

- calculating a mixing rule using link layer information and correlation of the original signal (212, y) and covariance information associated with the downmix signal (246, x); And

- generation of a synthesis signal using a prototype signal and mixing rules.

A decoding method is also provided for generating a synthesis signal (336) from a downmix signal (324, x) having a number of downmix channels, wherein the synthesis signal (336) has a number of synthesis channels, wherein the downmix signal (324, x) represents is a downmixed version of the original signal (212) having a number of original channels, the method comprising the following phases:

- the first phase (610c'), including:

- synthesis of the first component (336M') of the synthesis signal according to the first mixing matrix (MM) calculated from:

- covariance matrix (C _yR ) associated with the synthesis signal (for example, the reconstructed target version of the covariance of the original signal); And

- a covariance matrix (C _x ) associated with the downmix signal (324).

- a second phase (610c) for synthesizing a second component (336R') of the synthesis signal, wherein the second component (336R') is a residual component, wherein the second phase (610c) includes:

- a prototype signal processing step (612c) upmixing a downmix signal (324) from said number of downmix channels into said number of synthesis channels;

- a decorrelation stage (614c), decorrelating the upmixed prototype signal (613c);

- a second mixing matrix processing step (618c) synthesizing a second component (336R') of the synthesis signal according to a second mixing matrix (MR) from a decorrelated version (615c) of the downmixing signal (324), wherein the second mixing matrix (MR) is a residual matrix mixing,

- wherein the method calculates the second mixing matrix (MR) from:

- residual covariance matrix (Cr) provided by the first mixing matrix processing step (600c); And

- estimates of the covariance matrix ( ) decorrelated prototype signals obtained from the covariance matrix (C _x ) associated with the downmix signal (324),

- wherein the method further comprises a summing step (620c) summing the first synthesis signal component (336M') with the second synthesis signal component (336R'), thereby obtaining a synthesis signal (336).

In addition, an encoding method is provided for generating a downmix signal (246, x) from a source signal (212, y), wherein the source signal (212, y) has a number of source channels, and wherein the downmix signal (246, x) has a number of number of downmix channels, wherein the method contains:

- assessment (218) of channel level information (220) and correlation of the original signal (212, y),

- encoding (226) the downmix signal (246, x) into a bitstream (248) such that the downmix signal (246, x) is encoded into the bitstream (248) such that it has auxiliary information (228) including includes information (220) of the channel level and correlation of the original signal (12, y).

These methods may be implemented in any of the encoders and decoders explained above.

7. Storage blocks

Moreover, the invention may be implemented in a persistent storage unit storing instructions that, when executed by a processor, cause the processor to carry out the method described above.

Additionally, the invention may be implemented in a persistent storage module storing instructions that, when executed by a processor, cause the processor to control at least one of the encoder or decoder functions.

The storage module, for example, may be part of an encoder 200 or a decoder 300.

8. Other aspects

Although certain aspects are described in the context of an apparatus, it will be appreciated that these aspects also represent a description of the corresponding method, wherein the block or apparatus corresponds to a method step or a feature of a method step. Likewise, aspects described in the context of a method step also provide a description of the corresponding block or element, or feature of the corresponding device. Some or all of the steps of the method may be performed by (or using) a hardware device, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some aspects, some of one or more of the most important steps of the method can be performed by this device.

Depending on certain implementation requirements, aspects of the invention may be implemented in hardware or software. An implementation may be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM, or flash memory, having stored electronically readable control signals that interact (or allow interaction) with a programmable computer system in such a way that the appropriate method is carried out. Therefore, the digital storage medium may be machine readable.

Some aspects of the invention comprise a storage medium having electronically readable control signals that are capable of interacting with a programmable computer system in a manner that implements one of the methods described herein.

In general, aspects of the present invention may be implemented as a computer program product with program code, wherein the program code is configured to implement one of the methods where the computer program product is executed on a computer. The program code, for example, may be stored on a computer-readable medium.

Other aspects comprise a computer program for implementing one of the methods described herein stored on a computer-readable medium.

In other words, an aspect of the inventive method is therefore a computer program having program code for carrying out one of the methods described herein when the computer program runs on a computer.

Therefore, an additional aspect of the inventive methods is a storage medium (digital storage medium or computer readable medium) containing a recorded computer program for implementing one of the methods described herein. The storage medium, digital storage medium, or recorded data medium is typically tangible and/or non-volatile.

Therefore, an additional aspect of the inventive method is a data stream or signal sequence representing a computer program for implementing one of the methods described herein. The data stream or signal sequence, for example, may be configured to be transmitted over a data connection, such as the Internet.

An additional aspect comprises processing means, such as a computer or programmable logic device, configured to implement one of the methods described herein.

An additional aspect comprises a computer having a computer program installed for performing one of the methods described herein.

A further aspect of the invention comprises a device or system configured to transmit (eg, by electronic or optical means) to a receiving device a computer program for performing one of the methods described herein. The receiving device, for example, may be a computer, mobile device, storage device, or the like. The device or system, for example, may include a file server for transmitting a computer program to a receiving device.

In some aspects, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some aspects, a field programmable gate array may interface with a microprocessor to implement one of the methods described herein. In general, the methods are preferably implemented by any hardware device.

The apparatus described herein may be implemented using a hardware device, either using a computer, or using a combination of a hardware device and a computer.

The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

The above-described aspects are merely illustrative of the principles of the present invention. It should be understood that modifications and changes to the configurations and details described herein will be apparent to those skilled in the art. Therefore, limitation is intended only by the scope of the following claims and not by the specific details provided by way of description and explanation of aspects herein.

9. Bibliography and sources

[1] J. Herre, K. Kjörling, J. Breebart, C. Faller, S. Disch, H. Purnhagen, J. Koppens, J. Hilpert, J. Rödén, W. Oomen, K. Linzmeier and K. S. Chong, "MPEG Surround - The ISO/MPEG Standard for Efficient and Compatible Multichannel Audio Coding", Audio English Society, Vol. 56, No. 11, pp. 932-955, 2008.

[2] V. Pulkki, "Spatial Sound Reproduction with Directional Audio Coding", Audio English Society, Vol. 55, No. 6, pp. 503-516, 2007.

[3] C. Faller and F. Baumgarte, “Binaural Cue Coding - Part II: Schemes and Applications,” IEEE Transactions on Speech and Audio Processing, Vol. 11, No. 6, pp. 520-531, 2003.

[4] O. Hellmuth, H. Purnhagen, J. Koppens, J. Herre, J. Engdegård, J. Hilpert, L. Villemoes, L. Terentiv, C. Falch, A. Hölzer, M. L. Valero, B. Resch, H. Mundt and H.-O. Oh, "MPEG Spatial Audio Object Coding - The ISO/MPEG Standard for Efficient Coding of Interactive Audio Scenes", in AES, San Francisco, 2010.

[5] L. Mikko-Ville and V. Pulkki, "Converting 5.1. Audio Recordings to B-Format for Directional Audio Coding Reproduction", in ICASSP, Prague, 2011.

[6] D. A. Huffman, “A Method for the Construction of Minimum-Redundancy Codes,” Proceedings of the IRE, Vol. 40, No. 9, pp. 1098-1101, 1952.

[7] A. Karapetyan, F. Fleischmann and J. Plogsties, “Active Multichannel Audio Downmix,” in 145th Audio Engineering Society, New York, 2018.

[8] J. Vilkamo, T. Bäckström and A. Kuntz, "Optimized Covariance Domain Framework for Time-Frequency Processing of Spatial Audio", Journal of the Audio Engineering Society, Vol. 61, No. 6, pp. 403-411, 2013 year.

Claims (131)

1. An audio data synthesis device (300) for generating a synthesis signal (336, 340, yR) from a downmix signal (246, x), wherein the synthesis signal (336, 340, yR) has a plurality of synthesis channels, wherein the synthesis device (300) audio data contains: - an input interface (312) configured to receive a downmix signal (246, x), wherein the downmix signal (246, x) has a plurality of downmix channels and auxiliary information (228), wherein the auxiliary information (228) includes link layer and correlation information (314, ξ, χ) of the original signal (212, y), wherein the original signal (212, y) has a plurality of original channels; And - a synthesis processor (404), configured to generate a synthesis signal (336, 340, yR) according to at least one mixing rule in the form of a matrix using: - information (220, 314, ξ, χ) of the channel level and correlation of the original signal (212, y); And - covariance information (C _x ) of the downmix signal (324, 246, x), wherein the audio data synthesis device (300) is configured to restore (386) the target version (C _yR ) of the covariance information (Cy) of the source signal, wherein the audio data synthesis device (300) is configured to restore (386) the target version (C _yR ) of the covariance information (Cy) based on the estimated version ( ) of the original covariance information (Cy), with the estimated version ( ) the original covariance information (Cy) is reported to the mentioned number of synthesis channels, wherein the audio data synthesis device (300) is configured to obtain an evaluated version ( ) of the original covariance information from the covariance information (Cx) of the downmix signal (324, 246, x), wherein the audio data synthesis device (300) is configured to obtain an estimated version of ( ) of the original covariance information (220) by applying to the covariance information (C _x ) the downmixing signal (324, 246, x) an estimation rule (Q) that is or associated with a prototype rule for calculating the prototype signal (326). 2. Audio data synthesis device (300) according to claim 1, containing: - a prototype signal calculation module (326), configured to calculate a prototype signal (328) from a downmix signal (324, 246, x), wherein the prototype signal (328) has said number of synthesis channels; - a mixing rule calculation module (402), configured to calculate at least one mixing rule (403) using: - information (314, ξ, 0) of the channel level and correlation of the original signal (212, y); And - covariance information (C _x ) of the downmix signal (324, 246, x); - wherein the synthesis processor (404) is configured to generate a synthesis signal (336, 340, yR) using the prototype signal (328) and the at least one mixing rule (403). 3. An audio data synthesis device according to any of the preceding paragraphs, configured to restore the target version (C _yR ) of the covariance information (Cy) adapted to the number of signal channels (336, 340, yR) of the synthesis. 4. The audio data synthesis device according to claim 3, configured to restore a target version (C _yR ) of covariance information (Cy) adapted to the number of synthesis signal channels (336, 340, yR) by assigning groups of source channels to single synthesis channels, or conversely, such that the reconstructed target version of the covariance information (C _yR ) is reported to said number of signal channels (336, 340, yR) of the synthesis. 5. The audio data synthesis device according to claim 4, configured to restore the target version (C _yR ) of the covariance information (Cy), adapted to the number of signal channels (336, 340, yR) of the synthesis, by generating the target version (C _yR ) of the covariance information for said number of source channels and then applying a downmixing rule or an upmixing rule and energy compensation to achieve the target version (C _yR ) of the covariance for the synthesis channels. 6. An audio data synthesis device according to any of the preceding paragraphs, configured to normalize for at least one pair of channels of the estimated version ( ) of the original covariance information (Cy) into the square roots of the channel levels of a pair of channels. 7. An audio data synthesis device according to claim 6, configured to interpret a matrix with a normalized estimated version ( ) initial covariance information (Cy). 8. The audio data synthesis device of claim 7, configured to populate the matrix by inserting entries (908) obtained in the auxiliary information (228) of the bit stream (248). 9. Audio data synthesis device according to any one of claims. 6-8, configured to denormalize the matrix by scaling the estimated version ( ) of the original covariance information (Cy) by the square root of the channel levels forming a channel pair. 10. An audio data synthesis device according to any one of the preceding paragraphs, configured to extract from the auxiliary information (228) a signal (324, 246, x) downmixing the link layer information (ξ, χ) and correlation, and the audio data synthesis device is further configured to recovering the target version (C _yR ) of the covariance information (Cy) by means of the estimated version ( ) initial information (220) of the link layer and correlation from: - covariance information (C _x ) for at least one pair of channels; And - information (ξ, χ) of the link layer and correlation for at least one second channel and one pair of channels. 11. The audio data synthesis apparatus of claim 10, configured to prefer link layer information (ξ, χ) and correlation describing a channel or pair of channels obtained from the auxiliary information (228) of the bit stream (248) rather than the covariance information (Cy ), reconstructed from the (324, 246, x) downmix signal for the same channel or pair of channels. 12. The audio data synthesis apparatus of any one of the preceding claims, wherein the recovered target version (C _yR ) of the covariance information (Cy) describes the energy relationship between a pair of channels or is based at least in part on the levels associated with each channel of the pair of channels. 13. The audio data synthesis apparatus of any one of the preceding claims, configured to obtain a frequency domain (FD) version (324) of the downmix signal (246, x), wherein the FD version (324) of the downmix signal (246, x) is divided into frequency bands or groups of frequency bands, wherein different link layer and correlation information (220) is associated with different frequency bands or groups of frequency bands, - wherein the audio data synthesis device is configured to operate differently for different frequency bands or groups of frequency bands so as to obtain different mixing rules (403) for different frequency bands or groups of frequency bands. 14. The audio data synthesis apparatus of any one of the preceding claims, wherein the downmix signal (324, 246, x) is divided into intervals, wherein various link layer and correlation information (220) are associated with the various intervals, and the audio data synthesis apparatus is configured to operating differently for different intervals so as to obtain different mixing rules (403) for different intervals. 15. The audio data synthesis apparatus according to any one of the preceding claims, wherein the downmix signal (324, 246, x) is divided into frames, and each frame is divided into intervals, wherein the audio data synthesis apparatus is configured when the presence and position of the transition portion in one frame are transmitted in service signals (261) as being in the same transition interval: - associating and correlating current link layer information (220) with the transition interval and/or with intervals after the transition interval of the frame; And - associating with the frame interval preceding the transition interval, link layer information (220) and correlation of the previous frame. 16. The audio data synthesis apparatus of any one of the preceding claims, configured to select a prototype rule (Q) configured to calculate a prototype signal (328) based on the number of synthesis channels. 17. The audio data synthesis device according to claim 16, configured to select a prototype rule (Q) from a plurality of previously stored prototype rules. 18. An audio data synthesis device according to any one of the preceding claims, configured to determine a prototype rule (Q) based on manual selection. 19. The audio synthesis apparatus of claim 17 or 18, wherein the prior art rule includes a matrix (Q) with a first dimension and a second dimension, wherein the first dimension is associated with the number of downmix channels and the second dimension is associated with the number of synthesis channels. 20. An audio data synthesis device according to any one of the preceding claims, configured to operate at a bit rate equal to or less than 160 Kbps. 21. The audio data synthesis apparatus of any one of the preceding claims, further comprising an entropy decoder (312) for obtaining a downmix signal (246, x) with auxiliary information (314). 22. The audio data synthesis apparatus of any one of the preceding claims, further comprising a decorrelation module (614b, 614c, 330) for reducing the amount of correlation between different channels. 23. Audio data synthesis device according to any one of claims. 1-21, in which the prototype signal (328) is directly transmitted to the synthesis processor (600a, 600b, 404) without performing decorrelation. 24. The audio data synthesis apparatus of any one of the preceding claims, wherein at least one of link layer information (ξ, χ) and correlation information of the original signal (212, y) and covariance information (C _x ) of the downmix signal (246, x) , has the form of a matrix. 25. The audio data synthesis device according to any one of the preceding claims, wherein the auxiliary information (228) includes identification data of source channels; - wherein the audio data synthesis device is further configured to calculate at least one mixing rule (403) using at least one of the link level information (ξ, χ) and the correlation of the original signal (212, y), covariance information (C _x ) downmix signal (246, x), source channel IDs, and synthesis channel IDs. 26. The audio data synthesis apparatus of any one of the preceding claims, configured to compute at least one mixing rule by singular value decomposition (SVD). 27. The audio synthesis apparatus of any one of the preceding claims, wherein the downmix signal is divided into frames, wherein the audio synthesis apparatus is configured to smooth a received parameter or estimated or reconstructed value or mixing matrix using a linear combination with the parameter or estimated or reconstructed value, or the mixing matrix obtained for the previous frame. 28. The audio data synthesis device of claim 27, configured to, when the presence and/or position of a transition portion in one frame is signaled (261), deactivate smoothing of the received parameter or the estimated or reconstructed value or mixing matrix. 29. The audio data synthesis apparatus of any one of the preceding claims, wherein the downmix signal is divided into frames and the frames are divided into intervals, wherein the link layer and correlation information (220, ξ, χ) of the original signal (212, y) is obtained from the auxiliary information (228) of a stream (248) of bits frame by frame, and the audio data synthesis device is configured to use for the current frame a mixing rule obtained by scaling the mixing rule calculated for the current frame by a factor increasing along subsequent intervals of the current frame, and by adding the rule mix used for the previous frame in a version scaled down by a factor along subsequent intervals of the current frame. 30. An audio data synthesis device according to any of the preceding paragraphs, in which the number of synthesis channels is greater than the number of original channels. 31. An audio data synthesis device according to any of the preceding paragraphs, in which the number of synthesis channels is less than the number of source channels. 32. An audio encoder (200) for generating a downmix signal (246, x) from a source signal (212, y), wherein the source signal (212, y) has a plurality of source channels, wherein the downmix signal (246, x) has a plurality of channels downmixing, wherein the audio encoder (200) comprises: - parameter estimation module (218), configured to evaluate link level information (220) and correlate the original signal (212, y), and - a bitstream recording module (226) for encoding the downmix signal (246, x) into a bit stream (248) such that the downmix signal (246, x) is encoded into the bit stream (248) such that it has an auxiliary information (228) including link layer and correlation information (220) of the original signal (212, y), wherein the link layer and correlation information (220) of the original signal (212, y) includes at least one inter-channel level difference (ICLD), wherein the link layer and correlation information (220) of the original signal (212, y) encoded in the auxiliary information (228) includes at least correlation information (220, 908) describing the energy relationships between at least one pair of different original channels, but not all original channels. 33. The audio encoder of claim 32, configured to provide link layer information (220) and correlation of the original signal (212, y) as normalized values. 34. The audio encoder of claim 32 or 33, wherein the link layer and correlation information (212, y) of the original signal (212, y) encoded in the auxiliary information (228) includes or represents at least link layer information associated with all source channels. 35. Audio encoder according to any one of paragraphs. 32-34, in which the link layer and correlation information (220) of the source signal (212, y) includes at least one coherence value (ξ _i,j ) describing the coherence between two channels of a pair of source channels. 36. The audio encoder of claim 35, wherein the coherence value is normalized. 37. Audio encoder according to any one of paragraphs. 35, 36, in which the coherence value is as follows: - Where is the covariance between channels i and j, while And respectively, are the levels associated with channels i and j. 38. Audio encoder according to any one of paragraphs. 32-37, wherein the at least one ICLD is provided as a logarithmic value. 39. Audio encoder according to claims. 32-38, in which at least one ICLD is normalized. 40. The audio encoder of claim 39, wherein the ICLD is as follows: - Where: - X _i is the ICLD for channel i, - Pi is the power of the current channel i, - P _dmx,i is a linear combination of the covariance information values of the downmix signal. 41. Audio encoder according to any one of paragraphs. 32-40, configured to select (250) whether or not to encode at least a portion of the link layer and correlation information (220) of the original signal (212, y), based on the state information (252) so that include in the auxiliary information (228) an increased amount of link layer information (220) and correlations in the case of a relatively smaller amount of operational data. 42. Audio encoder according to any one of paragraphs. 32-41, configured to select (250) which portion of the link layer and correlation information (220) of the original signal (212, y) should be encoded in the auxiliary information (228), based on the channel-by-channel metrics (252) such that to include link layer and correlation information (220) associated with more sensitive metrics in the auxiliary information (228). 43. Audio encoder according to any one of paragraphs. 32-44, in which the link layer and correlation information (220) of the original signal (212, y) is in the form of matrix entries (Cy). 44. The audio encoder of claim 43, wherein the matrix is symmetrical or Hermitian, wherein link layer and correlation information entries (220) are provided for all or less of all entries on a diagonal of the matrix (Cy) and/or for less than half of the non-diagonal elements of the matrix (Cy). 45. Audio encoder according to any one of paragraphs. 32-44, in which the bitstream recording module (226) is configured to encode identification data of at least one channel. 46. Audio encoder according to any one of paragraphs. 32-45, in which the original signal (212, y) or its processed version (216) is divided into a plurality of subsequent frames of equal duration. 47. The audio encoder of claim 46, configured to encode link layer information (220) and correlate the source signal (212, y) specific to each frame in the auxiliary information (228). 48. The audio encoder of claim 47, configured to encode the same link layer information (220) and correlate the original signal (212, y) collectively associated with the plurality of successive frames into auxiliary information (228). 49. Audio encoder according to any one of paragraphs. 47, 48, configured to select a number of consecutive frames in which the same link layer and source signal correlation information (220) (212, y) is selected such that: - a comparatively higher bit rate or larger volume of operating data implies an increase in the number of consecutive frames with which the same link layer and correlation information (220) of the source signal (212, y) is associated, and vice versa. 50. Audio encoder according to any one of paragraphs. 48, 49, configured to reduce the number of consecutive frames with which the same link layer information (220) is associated and correlate the original signal (212, y) when a transition portion is detected. 51. Audio encoder according to any one of paragraphs. 46-50, in which each frame is divided into an integer number of consecutive intervals. 52. The audio encoder of claim 51, configured to estimate link layer and correlation information (220) for each slot and encode in the auxiliary information (228) the sum or average or other predetermined linear combination of the link layer information (220) and correlation estimated for various intervals, wherein the audio encoder is configured to perform transient analysis (258) on the time domain version of the frame to determine the presence of a transient portion in the frame. 53. The audio decoder according to claim 52, configured to determine in which frame interval there is a transition part, and: - encoding the link layer information (220) and the correlation of the original signal (212, y) associated with the interval in which there is a transition part, and/or with subsequent intervals in the frame, - without encoding the channel layer information (220) and the correlation of the original signal (212, y) associated with the intervals preceding the transition part. 54. The audio encoder according to claim 52 or 53, configured to transmit in the service signals (261) in the auxiliary information (228) the presence of a transition part present in one frame interval. 55. The audio encoder according to claim 54, configured to transmit in the service signals (261) in the auxiliary information (228) in which frame interval there is a transition part. 56. Audio encoder according to any one of paragraphs. 52-54, configured to evaluate the link layer information (220) and the correlation of the original signal (212, y) associated with a plurality of frame intervals, and add them or average them or linearly combine them to obtain link layer information (220) and correlation associated with the frame. 57. Audio encoder according to any one of paragraphs. 32-56, in which the original signal (212, y) is converted (263) into a frequency domain signal (264, 266), wherein the audio encoder is configured to encode link layer information (220) and correlate the original signal (212, y) by frequency bands in supporting information (228), wherein the audio encoder is configured to aggregate (265) a certain number of frequency bands of the original signal (212, y) into a more reduced number of frequency bands (266) so as to encode channel level information (220) and correlations of the original signal (212, y) according to aggregated frequency bands in the supporting information (228). 58. The audio encoder according to claim 57, configured to, in case of detection of a transition part in the frame, additional aggregation (265) of the frequency band in such a way that: - the number of frequency bands (266) is reduced; and/or - the width of at least one frequency band is increased by aggregation with another frequency band. 59. Audio encoder according to any one of paragraphs. 57, 58, further configured to encode (226) in the bit stream (248) at least one link layer information (220) and correlate one frequency band as an increment relative to the previously encoded link layer information and correlation. 60. Audio encoder according to any one of paragraphs. 32-59, configured to encode in the auxiliary information (228) bit stream (248) an incomplete version of the link layer information (220) and correlation with respect to the link layer information (220) and correlation estimated by the estimation module (218). 61. The audio encoder of claim 60, configured to adaptively select from all the link layer and correlation information (220) evaluated by the estimator (218) the selected information to be encoded in the auxiliary information (228) of the bit stream (248), such that such that the link layer and/or correlation information (220) for the remaining unselected information estimated by the estimator (218) is not encoded. 62. The audio encoder of claim 60, configured to recover link layer and correlation information (220) from selected link layer and correlation information (220), thereby simulating an estimate in the decoder (300) of unselected link layer and correlation information (220), and calculating error information between: - unselected link layer information (220) and correlation estimated by the encoder; And - unselected link layer and correlation information recovered by estimation modeling in the decoder (300), uncoded link layer and correlation information (220); And - in such a way as to distinguish, based on the calculated error information: - properly restored link layer and correlation information; from - link level and correlation information that cannot be properly restored, - in such a way as to make a decision regarding: - selecting properly unrecoverable link layer and correlation information to be encoded in the auxiliary information (228) of the bit stream (248); And - not selecting properly recoverable link layer and correlation information, thereby not encoding in the auxiliary information (228) a stream (248) of bits of properly recoverable link layer and correlation information. 63. Audio encoder according to any one of paragraphs. 61, 62, in which the link layer and correlation information (220) is indexed according to a given ordering, wherein the encoder is configured to signal in the auxiliary information (228) a stream (248) of index bits associated with the given ordering, wherein the indices indicate which of the link layer and correlation information (220) is encoded. 64. The audio encoder of claim 63, wherein the indices are transmitted via a bitmap. 65. Audio encoder according to any one of paragraphs. 63, 64, in which the indices are defined according to a combinatorial number system that associates a one-dimensional index with matrix entries. 66. Audio encoder according to any one of paragraphs. 64, 65, configured to select between: - adaptive provision of link layer information (220) and correlation, in which indices associated with a given ordering are encoded in the auxiliary information of the bit stream; And - fixedly providing link layer and correlation information (220) such that the link layer and correlation information (220) that is encoded is specified and ordered according to a given fixed ordering without providing indexes. 67. The audio encoder of claim 66, configured to signal in the auxiliary information (228) the bit stream (248) whether the link layer and correlation information (220) is provided according to adaptive provision or according to fixed provision. 68. Audio encoder according to any one of paragraphs. 32-67, further configured to encode (226) in the stream (248) bits of current link layer information (220t) and correlation as an increment (220k) relative to previous link layer information (220(t-1)) and correlation. 69. Audio encoder according to any one of paragraphs. 32-68, further configured to generate a downmix signal (246) according to the static downmix (244). 70. Audio encoder according to any one of paragraphs. 32-69, wherein the audio encoder is agnostic with respect to the audio data synthesis device. 71. A method for generating a synthesis signal from a downmix signal, wherein the synthesis signal has a plurality of synthesis channels, the method comprising the steps of: - receiving a downmix signal (246, x), wherein the downmix signal (246, x) has a plurality of downmix channels and auxiliary information (228), wherein the auxiliary information (228) includes: - link layer and correlation information (220) of the original signal (212, y), wherein the original signal (212, y) has a plurality of original channels; - forming a synthesis signal using channel level information (220) and correlation of the original signal (212, y) and covariance information (C _x ) of the downmixing signal (246, x), Moreover, the method additionally contains the steps of: recover (386) the target version (C _yR ) of the covariance information (C _y ) of the original signal based on the estimated version ( ) of the original covariance information (C _y ), while the estimated version ( ) the original covariance information (C _y ) is reported to the mentioned number of synthesis channels, and the evaluated version ( ) of the original covariance information is obtained from the covariance information (C _x ) of the downmix signal (246, x), wherein the estimated version ( ) of the original covariance information (220) is obtained by applying to the covariance information (C _x ) the downmix signal (324, 246, x) an estimation rule (Q) that is or is associated with a prototype rule for calculating the prototype signal (326). 72. The method according to claim 71, wherein the method contains the steps of: - calculating a prototype signal from the downmix signal (246, x), wherein the prototype signal has said number of synthesis channels; - calculating a mixing rule using link layer information and correlation of the original signal (212, y) and covariance information of the downmixing signal (246, x); And - generate a synthesis signal using a prototype signal and mixing rules. 73. A method for generating a downmix signal (246, x) from an original signal (212, y), wherein the original signal (212, y) has a number of original channels, and wherein the downmix signal (246, x) has a number of downmix channels , and the method contains the steps of: - evaluate (218) link layer and correlation information (220) of the source signal (212, y), wherein the link layer and correlation information (220) of the source signal (212, y) includes at least one inter-channel level difference (ICLD) , wherein the link layer and correlation information (220) of the original signal (212, y) encoded in the auxiliary information (228) further includes at least correlation information (220, 908) describing the energy relationships between at least one pair different source channels, but not all source channels, - encoding (226) the downmix signal (246, x) into a bit stream (248) such that the downmix signal (246, x) is encoded into the bit stream (248) such that it has auxiliary information (228) including includes information (220) of the channel level and correlation of the original signal (12, y). 74. A permanent storage unit storing instructions that, when executed by a processor, cause the processor to carry out the method of any one of claims. 71-73.

Images