CN116529815A - Apparatus and method for encoding a plurality of audio objects and apparatus and method for decoding using two or more related audio objects - Google Patents

Abstract

An apparatus for encoding a plurality of audio objects, comprising: an object parameter calculator (100) configured to: calculating parameter data of at least two related audio objects for one or more of a plurality of frequency bins related to the time frame, wherein the number of the at least two related audio objects is lower than the total number of the plurality of audio objects; and an output interface (200) configured to output an encoded audio signal comprising information about parametric data of at least two related audio objects of one or more frequency bins.

Description

Device and method for encoding multiple audio objects and device and method for decoding using two or more related audio objects

Technical Field

The present invention relates to encoding of audio signals (eg audio objects) and decoding of encoded audio signals (eg encoded audio objects).

Background Art

introduction

This document describes a parametric approach to encoding and decoding object-based audio content at low bitrates using Directional Audio Coding (DirAC). The presented embodiments are used as part of the 3GPP Immersive Voice and Audio Services (IVAS) codec and provide an advantageous alternative to the low bitrate Independent Streams with Metadata (ISM) mode, a discrete encoding method.

Prior art

Discrete encoding of objects

The most straightforward way to encode object-based audio content is to encode and transmit the objects individually together with the corresponding metadata. The main drawback of this approach is that the bit consumption required to encode the objects becomes prohibitively high as the number of objects increases. A simple solution to this problem is to adopt a "parametric approach", where a few relevant parameters are calculated from the input signal, quantized and transmitted together with a suitable downmix signal combining several object waveforms.

Spatial Audio Object Coding (SAOC)

Spatial Audio Object Coding [SAOC_STD, SAOC_AES] is a parametric approach where the encoder computes a downmix signal based on some downmix matrix D and a set of parameters and sends both to the decoder. These parameters represent the psychoacoustically relevant properties and relationships of all the individual objects. At the decoder, the downmix is rendered to a specific speaker layout using a rendering matrix R.

The main parameter of SAOC is the object covariance matrix E of size N*N, where N refers to the number of objects. This parameter is transmitted to the decoder as object level difference (OLD) and optional inter-object covariance (IOC).

The elements e _i,j of the matrix E are given by:

Object-level difference (OLD) is defined as

in, and the absolute object energy (NRG) is described as

as well as

Wherein, i and j are object indices of objects x _i and x _j respectively, n indicates a time index, and k indicates a frequency index. l indicates a time index set, and m indicates a frequency index set. ε is an additional constant to avoid division by zero, for example, ε=10.

The similarity measure of the input objects (IOC) can be given, for example, by the cross-correlation:

The downmix matrix D of size N_dmx*N is defined by the elements d _i,j , where i refers to the channel index of the downmix signal and j refers to the object index. For a stereo downmix (N_dmx=2), d _i,j is calculated from the parameters DMG and DCLD as

Where DMG _i and DCLD _i are given by:

For the mono downmix case (N_dmx=1), d _i,j is calculated based on the DMG parameters only as

in,

Spatial Audio Object Coding-3D (SAOC-3D)

Spatial Audio Object Coding 3D Audio Rendering (SAOC-3D) [MPEGH_AES, MPEGH_IEEE, MPEGH_STD, SAOC_3D_PAT] is an extension of the above-mentioned MPEG SAOC technology, which compresses and renders channel and object signals in a very bit rate efficient way.

The main differences from SAOC are:

While the original SAOC only supports up to two downmix channels, SAOC-3D can map multi-object input to an arbitrary number of downmix channels (and associated side information).

· Direct rendering to multi-channel output, compared to classic SAOC which already used MPEG Surround as multi-channel output processor.

Some tools, such as residual coding tools, were dropped.

Despite these differences, SAOC-3D is identical to SAOC from a parameter perspective. The SAOC-3D decoder - similar to the SAOC decoder - receives the multi-channel downmix X, the covariance matrix E, the rendering matrix R and the downmix matrix D.

The rendering matrix R is defined by the input channels and input objects, and is received from the format converter (channel) and the object renderer (object), respectively.

The downmix matrix D is defined by elements d _i,j , where i refers to the channel index of the downmix signal and j refers to the object index and is calculated from the downmix gain (DMG):

in,

The output covariance matrix C of size N_out*N_out is defined as:

C＝RER ^*

Related Programs

There are several other schemes that are similar in nature to SAOC, but with the following minor differences:

• Binaural Cue Coding (BCC) for objects has been described in, for example, [BCC2001] and is a predecessor of the SAOC technique.

Joint Object Coding (JOC) and Advanced Joint Object Coding (A-JOC) perform a similar function to SAOC while providing roughly separated objects on the decoder side without rendering them to a specific output speaker layout [JOC_AES, AC4_AES]. This technique sends the elements of the upmix matrix from the downmix to the separated objects as parameters

(instead of OLD).

Directional Audio Coding (DirAC)

Another parameterized approach is directional audio coding. DirAC [Pulkki 2009] is a perceptually driven reproduction of spatial sound. Assumption: At one time instance and for one critical frequency band, the spatial resolution of the human auditory system is limited to decoding a directional cue and another interaural coherence cue.

Based on these assumptions, DirAC represents spatial sound in a frequency band by fading in and out two streams: a non-directional diffuse stream and a directional non-diffuse stream. The DirAC processing is performed in two stages: analysis and synthesis, as shown in Figures 12a and 12b.

In the DifAC analysis stage, a B-format first-order coincident microphone is considered as input and the diffuseness and arrival direction of the sound are analyzed in the frequency domain.

In the DifAC synthesis stage, the sound is divided into two streams, a non-diffuse stream and a diffuse stream. The non-diffuse stream is reproduced as a point source using amplitude panning, which can be done by using vector basis amplitude panning (VBAP) [Pulkki1997]. The diffuse stream is responsible for the sense of envelopment and is produced by sending mutually decorrelated signals to the speakers.

The analysis stage in Figure 12a includes a band filter 1000, an energy estimator 1001, an intensity estimator 1002, time averaging elements 999a and 999b, a diffuseness calculator 1003 and a direction calculator 1004. The calculated spatial parameters are a diffuseness value between 0 and 1 for each time/frequency bin and a direction of arrival parameter for each time/frequency bin generated by block 1004. In Figure 12a, the direction parameters include an azimuth and an elevation indicating the direction of arrival of the sound relative to a reference or listening position and specifically relative to the location of the microphone from which the four component signals input to the band filter 1000 are collected. In Figure 12a, these component signals are first order surround sound components including an omnidirectional component W, a directional component X, another directional component Y and yet another directional component Z.

The DirAC synthesis stage shown in FIG12 b includes a bandpass filter 1005 for generating a time/frequency representation of the B-format microphone signals W, X, Y, Z. The corresponding signals of each time/frequency region are input to a virtual microphone stage 1006, which generates a virtual microphone signal for each channel. Specifically, in order to generate a virtual microphone signal of, for example, a central channel, the virtual microphone is pointed in the direction of the central channel, and the resulting signal is the corresponding component signal of the central channel. The signal is then processed via a direct signal branch 1015 and a diffuse signal branch 1014. Both branches include corresponding gain adjusters or amplifiers, which are controlled by diffuseness values derived from the original diffuseness parameters in blocks 1007, 1008, and are further processed in blocks 1009, 1010 to obtain a certain microphone compensation.

The component signals in the direct signal branch 1015 are also gain adjusted using gain parameters derived from the directional parameters consisting of azimuth and elevation angles. Specifically, these angles are input into a VBAP (Vector Basis Amplitude Translation) gain table 1011. For each channel, the results are input into a loudspeaker gain averaging stage 1012 and a further normalizer 1013, and the resulting gain parameters are then forwarded to the amplifier or gain adjuster in the direct signal branch 1015. In a combiner 1017, the diffuse signal generated at the output of the decorrelator 1016 and the direct signal or non-diffuse stream are combined, and then the other subbands are added in a further combiner 1018, which may be, for example, a synthesis filter bank. Thus, a loudspeaker signal for a certain loudspeaker is generated, and the same process is performed for other channels of other loudspeakers 1019 in a certain loudspeaker setup.

A high-quality version of DirAC synthesis is shown in FIG12b, where the synthesizer receives all B-format signals, from which a virtual microphone signal is calculated for each speaker direction. The directional pattern used is typically a dipole. The virtual microphone signal is then modified in a nonlinear manner, depending on the metadata discussed with respect to branches 1016 and 1015. A low-bitrate version of DirAC is not shown in FIG12b. However, in this low-bitrate version, a single audio channel is sent. The difference in processing is that all virtual microphone signals will be replaced by a single audio channel received. The virtual microphone signal is divided into two streams, a diffuse stream and a non-diffuse stream that are processed separately. Non-diffuse sound is reproduced as a point source by using vector basis amplitude translation (VBAP). When panning, the monophonic sound signal is applied to a subset of speakers after being multiplied by a speaker-specific gain factor. The gain factor is calculated using information about the speaker settings and the specified panning direction. In the low-bitrate version, the input signal is simply panned to the direction implied by the metadata. In the high-quality version, each virtual microphone signal is multiplied by the corresponding gain factor, which produces the same effect as panning, but it is less prone to any non-linear artifacts.

The goal of diffuse sound synthesis is to create the perception of sound surrounding the listener. In the low bitrate version, the diffuse flow is reproduced by decorrelating the input signal and reproducing it from each loudspeaker. In the high quality version, the virtual microphone signals of the diffuse flow are already somewhat incoherent, and they only need to be decorrelated slightly.

The DirAC parameters (also called spatial metadata) consist of a tuple of diffuseness and direction, represented in spherical coordinates by two angles, i.e., azimuth and elevation. If both the analysis and synthesis stages are run on the decoder side, the time-frequency resolution of the DirAC parameters can be chosen to be the same as the filterbank used for DirAC analysis and synthesis, i.e., a different set of parameters for each time slot and frequency interval of the filterbank representation of the audio signal.

Some work has been done to reduce the size of metadata to enable the DirAC paradigm to be used in spatial audio coding and teleconferencing scenarios [Hirvonen2009].

In [WO2019068638], a general spatial audio coding system based on DirAC is introduced. Compared with the classic DirAC designed for B-format (first-order surround sound format) input, the system can accept first-order or higher-order surround sound, multi-channel or object-based audio input, and also allows mixed-type input signals. All signal types are efficiently encoded and sent separately or in combination. The former combines different representations at the renderer (decoder side), while the latter uses an encoder-side combination of different audio representations in the DirAC domain.

Compatibility with DirAC framework

This embodiment builds on a unified framework for arbitrary input types as proposed in [WO2019068638] and - similar to what [WO2020249815] did for multi-channel content - aims to eliminate the problem that DirAC parameters (direction and diffuseness) cannot be efficiently applied to object inputs. In fact, the diffuseness parameter is not needed at all, but it was found that a single directional cue per time/frequency unit is not sufficient to reproduce high-quality object content. Therefore, this embodiment proposes to adopt multiple directional cues per time/frequency unit and introduces an adapted parameter set accordingly, which replaces the classic DirAC parameters in the case of object input.

Flexible system for low bit rates

In contrast to DirAC, which uses a scene-based representation from the listener's perspective, SAOC and SAOC-3D are designed for channel-based and object-based content, where parameters describe the relationship between channels/objects. In order to use scene-based representation for object input and thus be compatible with the DirAC renderer while ensuring efficient representation and high-quality reproduction, an adapted parameter set is required in order to also allow signaling of multiple directional cues.

An important goal of this embodiment is to find a method that can efficiently encode the object input at a low bit rate and has good scalability for an increasing number of objects. Discrete encoding of each object signal does not provide such scalability: each additional object causes the overall bit rate to rise significantly. If the number of objects increases beyond the allowed bit rate, this will directly lead to a very noticeable degradation of the output signal; this degradation is another argument in favor of this embodiment.

It is an object of the present invention to provide improved concepts for encoding a plurality of audio objects or for decoding an encoded audio signal.

This object is achieved by a device for encoding according to claim 1, a decoder according to claim 18, an encoding method according to claim 28, a decoding method according to claim 29, a computer program according to claim 30 or an encoded audio signal according to claim 31.

In one aspect of the present invention, the present invention is based on the following discovery: for one or more frequency intervals among multiple frequency intervals, at least two related audio objects are defined, and parameter data related to the at least two related objects are included on the encoder side and used on the decoder side to obtain a high-quality and efficient audio encoding/decoding concept.

According to another aspect of the invention, the invention is based on the finding that a specific downmix adapted to the directional information associated with each object is performed such that each object with associated directional information valid for the entire object (i.e. for all frequency bins in a time frame) is used for downmixing the object into a plurality of transmission channels. The use of the directional information is, for example, equivalent to generating the transmission channels as virtual microphone signals with certain adjustable characteristics.

On the decoder side, a specific synthesis relying on covariance synthesis is performed, which in certain embodiments is particularly suitable for high quality covariance synthesis that is not affected by artifacts introduced by the decorrelator. In other embodiments, advanced covariance synthesis is used that relies on specific improvements with respect to standard covariance synthesis in order to improve the audio quality and/or reduce the amount of computation required to calculate the mixing matrix used within the covariance synthesis.

However, even in more classical synthesis where audio rendering is performed by explicitly determining the individual contributions within a time/frequency interval based on the sent selection information, the audio quality is superior to prior art object encoding methods or channel downmixing methods. In this case, each time/frequency interval has object identification information, and when audio rendering is performed, i.e. when the directional contribution of each object is considered, the object identification is used to look up the direction associated with the object information in order to determine the gain values of the individual output channels for each time/frequency interval. Therefore, when there is only a single relevant object in the time/frequency interval, based on the "codebook" of the object ID and the directional information of the associated object, only the gain value of the single object is determined for each time/frequency interval.

However, when there is more than one relevant object in a time/frequency interval, a gain value is calculated for each relevant object so as to allocate the corresponding time/frequency interval of the transmission channel to the corresponding output channel controlled by the output format provided by the user (e.g., whether a certain channel format is a stereo format, a 5.1 format, etc.). Regardless of whether the gain values are used for the purpose of covariance synthesis (i.e., for the purpose of applying a mixing matrix to mix the transmission channels into the output channels), or whether the gain values are used to explicitly determine the individual contribution of each object in the time/frequency interval (possibly enhanced by the addition of a diffuse signal component) by multiplying the gain value by the corresponding time/frequency interval of one or more transmission channels and then summing the contribution of each output channel in the corresponding time/frequency interval, the output audio quality is still enhanced due to the flexibility provided by determining one or more relevant objects per frequency interval.

This determination may be very efficient, however, since only one or more object IDs for a time/frequency bin have to be encoded and sent to the decoder together with the direction information for each object. This is because: for a frame, only a single direction information exists for all frequency bins.

Hence, regardless of whether the synthesis is performed using preferably enhanced covariance synthesis or using a combination of explicit transmission channel contributions for each object, an efficient and high quality object downmix is obtained, which is preferably enhanced by using a specific object direction-dependent downmix that relies on downmix weights that reflect the generation of the transmission channels as virtual microphone signals.

The aspects related to two or more related objects per time/frequency interval may preferably be combined with the aspects of performing a specific direction-dependent downmix of an object to a transmission channel. However, these two aspects may also be applied independently of each other. In addition, although covariance synthesis of two or more related objects per time/frequency interval is performed in some embodiments, advanced covariance synthesis and advanced transmission channel to output channel upmixing may also be performed by sending only a single object identifier per time/frequency interval.

In addition, whether there is a single or several related objects per time/frequency interval, upmixing can also be performed by calculating the mixing matrix in standard or enhanced covariance synthesis, or upmixing can be performed by individually determining the contribution of the time/frequency interval (which is based on the object identification used to obtain specific direction information from the direction "codebook") to determine the gain value of the corresponding contribution. These are then summed to obtain the complete contribution of each time/frequency interval in the case where there are two or more related objects per time/frequency interval. The output of this summing step is then equivalent to the output of the mixing matrix application, and the final filter bank processing is performed to generate the time domain output channel signal of the corresponding output format.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:

FIG1a is an implementation of an audio encoder according to the first aspect with at least two related objects per time/frequency interval;

FIG1 b is an implementation of an encoder according to the second aspect with direction-dependent object downmix;

Fig. 2 is a preferred implementation of an encoder according to the second aspect;

FIG3 is a preferred implementation of an encoder according to the first aspect;

FIG4 is a preferred implementation of a decoder according to the first aspect and the second aspect;

FIG5 is a preferred implementation of the covariance synthesis process of FIG4;

FIG6a is an implementation of a decoder according to the first aspect;

Fig. 6b is a decoder according to the second aspect;

FIG7a is a flow chart for illustrating determining parameter information according to the first aspect;

FIG7 b is a preferred implementation of further determining parameterized data;

Fig. 8a shows a high resolution filter bank time/frequency representation;

FIG8b shows the transmission of relevant auxiliary information for frame J according to a preferred implementation of the first aspect and the second aspect;

FIG8c shows a "directional codebook" included in the encoded audio signal;

FIG9a shows a preferred encoding method according to the second aspect;

FIG9b shows an implementation of static downmixing according to the second aspect;

FIG9c shows an implementation of dynamic downmixing according to the second aspect;

Figure 9d shows another embodiment of the second aspect;

FIG10a shows a flow chart of a preferred implementation of the decoder side of the first aspect;

FIG10b shows a preferred implementation of the output channel calculation of FIG10a according to an embodiment where the contribution of each output channel is summed;

FIG10c shows a preferred manner of determining power values for a plurality of related objects according to the first aspect;

FIG10d shows an embodiment of calculating the output channels of FIG10a using covariance synthesis that relies on the calculation and application of a mixing matrix;

FIG11 shows several embodiments for high-level calculation of a mixing matrix for time/frequency bins;

FIG12a shows a prior art DirAC encoder; and

Fig. 12b shows a prior art DirAC decoder.

DETAILED DESCRIPTION

FIG. 1a shows a device for encoding a plurality of audio objects, which receives the original audio objects and/or metadata of the audio objects at the input. The encoder comprises an object parameter calculator 100, which provides parameter data of at least two related audio objects for a time/frequency interval, and forwards the data to an output interface 200. Specifically, the object parameter calculator calculates parameter data of at least two related audio objects for one or more frequency intervals in a plurality of frequency intervals associated with a time frame, wherein, in particular, the number of at least two related audio objects is lower than the total number of a plurality of audio objects. Therefore, the object parameter calculator 100 actually performs a selection, rather than merely indicating that all objects are related. In a preferred embodiment, the selection is performed by correlation, and the correlation is determined by an amplitude-related measurement value (e.g., amplitude, power, loudness, or another measurement value obtained by increasing the amplitude to a power different from 1 and preferably greater than 1). Then, if a certain number of related objects are available for the time/frequency interval, the object with the most relevant characteristics (i.e., the object with the highest power among all objects) is selected, and the data about these selected objects are included in the parameter data.

The output interface 200 is configured to output a coded audio signal including information about parameter data of at least two related audio objects for one or more frequency intervals. Depending on the implementation, the output interface may receive other data (e.g., object downmix, or one or more transmission channels representing the object downmix, or additional parameters, or object waveform data in a mixed representation of several objects downmixed, or other objects in a separate representation) and input it into the coded audio signal. In this case, the object is directly introduced or "copied" into the corresponding transmission channel.

FIG1 b shows a preferred implementation of an apparatus for encoding a plurality of audio objects according to the second aspect, wherein audio objects and associated object metadata are received, the associated object metadata indicating directional information about a plurality of audio objects, i.e. one directional information for each object or a group of objects (if the group of objects has the same directional information associated therewith). The audio objects are input into a downmixer 400 so as to downmix the plurality of audio objects to obtain one or more transmission channels. In addition, a transmission channel encoder 300 is provided, which encodes the one or more transmission channels to obtain one or more encoded transmission channels, which are then input into the output interface 200. Specifically, the downmixer 400 is connected to an object directional information provider 110, which receives at input any data from which the object metadata can be derived, and outputs directional information actually used by the downmixer 400. The directional information forwarded from the object directional information provider 110 to the downmixer 400 is preferably dequantized directional information, i.e. the same directional information then available at the decoder side. To this end, the object direction information provider 110 is configured to derive or extract or obtain non-quantized object metadata, and then quantize the object metadata to derive quantized object metadata representing the quantization index in the "other data" shown in Figure 1b, which is provided to the output interface 200 in a preferred embodiment. In addition, the object direction information provider 110 is configured to dequantize the quantized object direction information to obtain the actual direction information forwarded from the block 110 to the downmixer 400.

Preferably, the output interface 200 is configured to additionally receive parameter data of audio objects, object waveform data, one or several identifications of single or multiple related objects per time/frequency bin, and quantized direction data as discussed previously.

Subsequently, other embodiments are shown. A parameterized method for encoding audio object signals is proposed, which allows efficient transmission at low bit rates and high-quality reproduction on the consumer side. Based on the DirAC principle of a directional hint considering each key band and time instance (time/frequency zone), the most dominant object is determined for each such time/frequency zone of the time/frequency representation of the input signal. Since this proves to be insufficient for the object input, an additional, second most dominant object is determined for each time/frequency zone, and based on these two objects, a power ratio is calculated to determine the impact of each of the two objects on the considered time/frequency zone. Note: It is also conceivable to consider more than two most dominant objects for each time/frequency unit, especially for an increasing number of input objects. For simplicity, the following description is mainly based on two dominant objects per time/frequency unit.

Therefore, the parameterized side information sent to the decoder includes:

• Power ratios calculated for the relevant (dominant) object subset for each time/frequency region (or parameter band).

• An object index representing the relevant subset of objects for each time/frequency region (or parameter band).

• Direction information associated with the object index and provided for each frame (where each time-domain frame comprises a plurality of parameter bands, and each parameter band comprises a plurality of time/frequency bins).

The directional information may be obtained via an input metadata file associated with the audio object signal. For example, the metadata may be specified on a frame basis. In addition to the auxiliary information, a downmix signal combining the input object signals is also sent to the decoder.

During the rendering phase, the sent directional information (derived via the object index) is used to translate the sent downmix signal (or more generally: the transmission channel) to the appropriate direction. The downmix signal is distributed to the two relevant object directions based on the sent power ratio, which is used as a weighting factor. This process is performed for each time/frequency region of the time/frequency representation of the decoded downmix signal.

This section summarizes the encoder-side processing and then describes the parameters and downmix calculations in detail. The audio encoder receives one or more audio object signals. For each audio object signal, a metadata file describing the object properties is associated. In this embodiment, the object properties described in the associated metadata file correspond to directional information provided on a frame basis, where a frame corresponds to 20 milliseconds. Each frame is identified by a frame number and is also included in the metadata file. The directional information is given in the form of azimuth and elevation information, where the azimuth takes values from (-180, 180] degrees and the elevation takes values from [-90, 90] degrees. Other properties provided in the metadata may include, for example, distance, propagation, gain; these properties are not considered in this embodiment.

The information provided in the metadata file is used together with the actual audio object files to create a parameter set that is sent to the decoder and used to render the final audio output file. More specifically, the encoder estimates the parameters, i.e., power ratios, of the main object subsets for each given time/frequency region. The main object subsets are represented by object indices, which are also used to identify the object directions. These parameters are sent to the decoder along with the transmission channel and direction metadata.

FIG. 2 gives an overview of the encoder, wherein the transmission channel includes a downmix signal calculated from an input object file and directional information provided in the input metadata. The number of transmission channels is always less than the number of input target files. In the encoder of the embodiment, the encoded audio signal is represented by the encoded transmission channel, and the encoded parameterized auxiliary information is indicated by the encoded object index, the coding power ratio and the encoding direction information. The encoded transmission channel and the encoded parameterized auxiliary information together form a bit stream output by the multiplexer 220. Specifically, the encoder includes a filter bank 102 that receives the input object audio file. In addition, the object metadata file is provided to the extractor direction information block 110a. The output of block 110a is input into a quantized direction information block 110b, which outputs the direction information to the downmixer 400 that performs the downmix calculation. In addition, the quantized direction information (i.e., the quantized index) is forwarded from block 110b to the encoding direction information 202 block, which preferably performs some entropy coding to further reduce the required bit rate.

In addition, the output of filter bank 102 is input to signal power calculation block 104, and the output of signal power calculation block 104 is input to object selection block 106 and is additionally input to power ratio calculation block 108. Power ratio calculation block 108 is also connected to object selection block 106, so that power ratio (that is, only for the combined value of selected object) is calculated. In block 210, the calculated power ratio or combined value is quantized and encoded. As will be outlined later, power ratio is preferred, so as to save the transmission of a power data item. However, in other embodiments that do not need this preservation, it is not power ratio, but actual signal power or other values derived from the signal power determined by block 104 can be input to quantizer and encoder under the selection of object selector 106. Then, power ratio calculation 108 is not needed, and object selection 106 ensures that only relevant parameterized data (that is, power-related data of relevant objects) are input to block 210 for the purpose of quantization and encoding.

Comparing Fig. 1a with Fig. 2, blocks 102, 104, 110a, 110b, 106, 108 are preferably included in the object parameter calculator 100 of Fig. 1a, and blocks 202, 210, 220 are preferably included in the output interface block 200 of Fig. 1a.

Furthermore, the core encoder 300 in Fig. 2 corresponds to the transmission channel encoder 300 of Fig. 1b, the downmix calculation block 400 corresponds to the downmixer 400 of Fig. 1b, and the object direction information provider 110 of Fig. 1b corresponds to the blocks 110a, 110b of Fig. 2. Furthermore, the output interface 200 of Fig. 1b is preferably implemented in the same manner as the output interface 200 of Fig. 1a and comprises the blocks 202, 210, 220 of Fig. 2.

Figure 3 shows an encoder variant where the downmix calculation is optional and does not depend on the input metadata. In this variant, the input audio files can be fed directly into the core encoder, which creates transport channels from them, so the number of transport channels corresponds to the number of input object files; this is particularly interesting if the number of input objects is 1 or 2. For a large number of objects, the downmix signal will still be used to reduce the amount of data to be sent.

In FIG. 3 , similar reference numerals refer to similar functions of FIG. 2 . This is valid not only for FIG. 2 and FIG. 3 , but also for all other figures described in this specification. Unlike FIG. 2 , FIG. 3 performs a downmix calculation 400 without any directional information. Thus, the downmix calculation may be, for example, a static downmix using a pre-known downmix matrix, or may be an energy-dependent downmix that is independent of any directional information associated with an object included in the input object audio file. However, the directional information is extracted in block 110 a and quantized in block 110 b, and the quantized value is forwarded to a directional information encoder 202 so as to have the encoded directional information in the encoded audio signal (e.g., a binary encoded audio signal) forming the bitstream.

In the case of not too many input audio object files or in the case of having enough available transmission bandwidth, the downmix calculation block 400 can also be omitted so that the input audio object files directly represent the transmission channels encoded by the core encoder. In this implementation, blocks 104, 104, 106, 108, 210 are also not needed. However, a preferred implementation leads to a hybrid implementation in which some objects are directly introduced into the transmission channels, while other objects are downmixed into one or more transmission channels. In this case, all blocks shown in FIG. 3 will be necessary to directly generate a bitstream having one or more objects and one or more transmission channels generated by the downmixer 400 of FIG. 2 or FIG. 3 within the encoded transmission channel.

Parameter calculation

The time domain audio signal including all input object signals is converted into the time domain/frequency domain using a filter bank. For example: the CLDFB (complex low delay filter bank) analysis filter converts a 20 millisecond frame (corresponding to 960 samples at a sampling rate of 48kHz) into a time/frequency region of size 16x60, with 16 time slots and 60 frequency bands. For each time/frequency unit, the instantaneous signal power is calculated as:

P _i (k, n) = | x _i (k, n) | ² ,

Where k denotes the frequency band index, n denotes the time slot index, and i denotes the object index. Since the transmission parameters for each time/frequency zone are very costly in terms of the final bit rate, grouping is employed in order to calculate the parameters for a reduced number of time/frequency zones. For example: 16 time slots can be grouped together into a single time slot, and 60 frequency bands can be grouped into 11 frequency bands based on a psychoacoustic scale. This reduces the initial size of 16x60 to 1x11 corresponding to 11 so-called parametric bands. The instantaneous signal power values are summed on a grouping basis to obtain the signal power of the reduced dimension:

where T corresponds to 15 in this example, and _BS and _BE define the parameter band boundaries.

To determine the subset of the most dominant objects for which parameters are calculated, the instantaneous signal power values of all N input audio objects are sorted in descending order. In this embodiment, we determine the two most dominant objects, and the corresponding object indices ranging from 0 to N-1 are stored as part of the parameters to be sent. In addition, the power ratio that relates the two dominant object signals to each other is calculated:

Or in a more general way that is not limited to two objects:

where, in this context, S represents the number of primary objects to be considered, and:

In the case of two main objects, for each of the two objects, a power ratio of 0.5 means that both objects are equally present in the corresponding parameter band, while power ratios of 1 and 0 indicate that one of the two objects is absent. These power ratios are stored as the second part of the parameters to be sent. Since the power ratios sum to 1, it is sufficient to send S-1 values instead of S.

In addition to the object index and the power ratio value for each parameter band, the direction information for each object extracted from the input metadata file must be sent. Since the information is originally provided on a frame basis, this is done for each frame (wherein each frame comprises 11 parameter bands or a total of 16x60 time/frequency bins in the described example). The object index therefore indirectly indicates the object direction. Note: Since the power ratios sum to 1, the number of power ratios to be sent per parameter band can be reduced by 1; for example: in the case where 2 related objects are considered, sending 1 power ratio value is sufficient.

Both the directional information and the power ratio value are quantized and combined with the object index to form parameterized side information. This parameterized side information is then encoded and mixed into the final bitstream representation together with the coded transport channel/downmix signal. For example, by quantizing the power ratio using 3 bits per value, a good compromise between output quality and extended bitrate is achieved. The directional information can be provided with an angular resolution of 5 degrees and then quantized with 7 bits per azimuth value and 6 bits per elevation value to give a practical example.

Downmix calculation

All input audio object signals are combined into a downmix signal comprising one or more transmission channels, wherein the number of transmission channels is smaller than the number of input object signals. Note: In this embodiment, if only one input object is present, only a single transmission channel is present, which therefore means that the downmix calculation is skipped.

If the downmix comprises two transmission channels, the stereo downmix may for example be calculated as a virtual cardioid microphone signal. The virtual cardioid microphone signal is determined by applying the direction information provided for each frame in the metadata file (here, all elevation angle values are assumed to be zero):

w _L = 0.5 + 0.5 * cos (azimuth - pi / 2)

w _R = 0.5 + 0.5 * cos (azimuth + pi / 2)

Here, the virtual cardioid is located at 90° and -90°. Therefore, individual weights for each of the two transmission channels (left and right) are determined and applied to the corresponding audio object signals:

In this context, N is the number of input objects greater than or equal to 2. If the virtual cardioid weights are updated for each frame, a dynamic downmix adapted to the direction information is employed. Another possibility is to employ a fixed downmix, where each object is assumed to be located at a static position. This static position may correspond, for example, to the initial orientation of the object, which then results in a static virtual cardioid weight that is the same for all frames.

If the target bitrate allows, more than two transmission channels can be envisaged. In case of three transmission channels, the cardioids can be arranged uniformly at, for example, 0°, 120° and -120°. If four transmission channels are used, a fourth cardioid can face upwards or the four cardioids can be arranged horizontally again in a uniform manner. The arrangement can also be customized to the object position if it is, for example, only a part of one hemisphere. The resulting downmix signal is processed by the core encoder and converted into a bitstream representation together with the encoded parameterized auxiliary information.

Alternatively, the input object signals may be fed into the core encoder without being combined into a downmix signal. In this case, the resulting number of transmission channels corresponds to the number of input object signals. Typically, a maximum number of transmission channels is given that is relevant to the total bit rate. The downmix signal is then only employed if the number of input object signals exceeds this maximum number of transmission channels.

Fig. 6a shows a decoder for decoding a coded audio signal (e.g., a signal output by Fig. 1a or Fig. 2 or Fig. 3), the coded audio signal comprising directional information of a plurality of audio objects and one or more transmission channels. In addition, for one or more frequency intervals of a time frame, the coded audio signal comprises parameter data of at least two related audio objects, wherein the number of at least two related objects is lower than the total number of the plurality of audio objects. Specifically, the decoder comprises an input interface for providing one or more transmission channels in the form of a spectrum representation, the spectrum representation having a plurality of frequency intervals in the time frame. This represents a signal forwarded to an audio renderer block 700 from an input interface block 600. Specifically, the audio renderer 700 is configured to render one or more transmission channels as a plurality of audio channels using the directional information included in the coded audio signal, the number of audio channels being preferably two channels for a stereo output format, or preferably more than two channels for a larger number of output formats (e.g., 3 channels, 5 channels, 5.1 channels, etc.). Specifically, the audio renderer 700 is configured to: for each frequency interval in the one or more frequency intervals, calculate the contribution of the one or more transmission channels according to the first direction information associated with the first related audio object in the at least two related audio objects and according to the second direction information associated with the second related audio object in the at least two related audio objects. Specifically, the direction information of the multiple audio objects includes the first direction information associated with the first object and the second direction information associated with the second object.

FIG8b shows parameter data of a frame, which in a preferred embodiment includes directional information 810 of a plurality of audio objects, and in addition includes a power ratio of each parameter band in a certain number of parameter bands shown at 812, and one (preferably two or even more) object index for each parameter band indicated at block 814. Specifically, the directional information of a plurality of audio objects 810 is shown in more detail in FIG8c. FIG8c shows a table with a first column of a certain object ID from 1 to N, where N is the number of a plurality of audio objects. In addition, a second column is provided, which has directional information for each object, preferably as an azimuth value and an elevation value, or in the case of a two-dimensional case, as an azimuth value only. This is shown at 818. Therefore, FIG8c shows a "directional codebook" included in the encoded audio signal input to the input interface 600 of FIG6a. The directional information from column 818 is uniquely associated with a certain object ID from column 816 and is valid for the "entire" object in the frame (i.e., for all frequency bands in the frame). Thus, regardless of the number of frequency bins, whether time/frequency bins in a high resolution representation or time/parameter bands in a lower resolution representation, only a single direction information is sent and input to the interface for each object identification.

In this context, Fig. 8a shows the time/frequency representation generated by the filter bank 102 when Fig. 2 or Fig. 3 is implemented as the CLDFB (complex low delay filter bank) discussed before. For the frame that gives the direction information as discussed before about Fig. 8b and Fig. 8c, the filter bank generates 16 time slots ranging from 0 to 15 and 60 frequency bands ranging from 0 to 59 in Fig. 8a. Therefore, one time slot and one frequency band represent time/frequency area 802 or 804. However, in order to reduce the bit rate of auxiliary information, it is preferred that the high-resolution representation is converted to a low-resolution representation, as shown in Fig. 8b, wherein there is only a single time interval, and wherein the 60 frequency bands are converted into 11 parameter bands as shown in 812 places in Fig. 8b. Therefore, as shown in Fig. 10c, the high-resolution representation is indicated by the time slot index n and the frequency band index k, and the low-resolution representation is given by the grouping time slot index m and the parameter band index 1. However, in the context of the present specification, the time/frequency interval may include the high resolution time/frequency zones 802, 804 of Figure 8a or the low resolution time/frequency units identified by the grouped slot index and parameter band index at the input of block 731c in Figure 10c.

In the embodiment of Fig. 6a, the audio renderer 700 is configured to calculate, for each frequency interval in the one or more frequency intervals, the contribution of the one or more transmission channels according to the first directional information associated with the first related audio object in the at least two related audio objects and according to the second directional information associated with the second related audio object in the at least two related audio objects. In the embodiment shown in Fig. 8b, the block 814 has an object index for each related object in the parameter band, i.e., has two or more object indices, so that there are two contributions for each time-frequency interval.

As will be outlined later with respect to Figure 10a, the calculation of the contribution can be performed indirectly via a mixing matrix, wherein a gain value for each related object is determined and used to calculate the mixing matrix. Alternatively, as shown in Figure 10b, the contribution can be explicitly calculated again using the gain value, and then the explicitly calculated contribution is summed for each output channel in a certain time/frequency interval. Therefore, regardless of whether the contribution is calculated explicitly or implicitly, the audio renderer still uses the directional information to render one or more transmission channels into multiple audio channels, so that for each frequency interval in one or more frequency intervals, the contribution of one or more transmission channels is included in the multiple audio channels according to the first directional information associated with the first related audio object of the at least two related audio objects and according to the second directional information associated with the second related audio object of the at least two related audio objects.

Figure 6b shows a decoder for decoding an encoded audio signal according to the second aspect, the encoded audio signal comprising: directional information of multiple audio objects and one or more transmission channels; and parameter data of the audio objects for one or more frequency intervals of a time frame. Similarly, the decoder includes an input interface 600 for receiving the encoded audio signal, and the decoder includes an audio renderer 700 for rendering one or more transmission channels as multiple audio channels using the directional information. Specifically, the audio renderer is configured to calculate direct response information based on one or more audio objects in each frequency interval in the multiple frequency intervals and the directional information associated with one or more related audio objects in the frequency interval. The direct response information preferably includes a gain value for covariance synthesis or advanced covariance synthesis or for explicitly calculating the contribution of one or more transmission channels.

Preferably, the audio renderer is configured to use direct response information of one or more related audio objects in time/frequency band and use information about multiple audio channels to calculate covariance synthesis information. In addition, the covariance synthesis information (which is preferably a mixing matrix) is applied to one or more transmission channels to obtain multiple audio channels. In another implementation, the direct response information is a direct response vector of each audio object in the one or more audio objects, and the covariance synthesis information is a covariance synthesis matrix, and the audio renderer is configured to perform matrix operations for each frequency interval when applying the covariance synthesis information.

In addition, the audio renderer 700 is configured to: when calculating the direct response information, derive the direct response vector of one or more audio objects, and for one or more audio objects, calculate the covariance matrix according to each direct response vector. In addition, when calculating the covariance synthesis information, the target covariance matrix is calculated. However, relevant information of the target covariance matrix (i.e., the direct response matrix or vector of one or more most important objects, and the diagonal matrix of the direct power indicated as E determined by applying the power ratio) can be used instead of the target covariance matrix.

Therefore, the target covariance information does not necessarily have to be an explicit target covariance matrix, but is derived from the covariance matrix of one audio object or the covariance matrices of more audio objects in a time/frequency interval, from power information about the corresponding one or more audio objects in the time/frequency interval, and from power information derived from one or more transmission channels for one or more time/frequency intervals.

The bitstream representation is read by the decoder and the encoded transport channel and the encoded parametric auxiliary information contained therein are made available for further processing. The parametric auxiliary information includes:

Directional information as quantized azimuth and elevation values (for each frame)

An object index representing a subset of related objects (for each parameter band)

Quantified power ratios relating related objects to each other (for each parameter band)

All processing is done in a frame-by-frame manner, where each frame contains one or more subframes. For example, a frame may consist of four subframes, in which case a subframe will have a duration of 5 milliseconds. Figure 4 shows a simplified overview of the decoder.

FIG4 shows an audio decoder implementing the first and second aspects. The input interface 600 shown in FIG6a and FIG6b comprises a demultiplexer 602, a core decoder 604, a decoder 608 for decoding an object index, a decoder 612 for decoding and dequantizing a power ratio, and a decoder for decoding and dequantizing the direction information indicated at 612. In addition, the input interface comprises a filter bank 606 for providing a transmission channel in a time/frequency representation.

The audio renderer 700 comprises a direct response calculator 704, a prototype matrix provider 702, controlled by an output configuration received, for example, by a user interface, a covariance synthesis block 706, and a synthesis filter bank 708, so as to ultimately provide an output audio file including the number of audio channels in a channel output format.

Hence, items 602, 604, 606, 608, 610, 612 are preferably comprised in the input interface of Figs. 6a and 6b and items 702, 704, 706, 708 of Fig. 4 are part of the audio renderer indicated with reference numeral 700 of Fig. 6a or 6b.

The coding parameterized auxiliary information is decoded and quantized power ratio, quantized azimuth value and quantized elevation value (direction information) and object index are re-obtained. A power ratio value that is not sent is obtained by utilizing the fact that the sum of all power ratios is 1. Their resolution (l, m) corresponds to the grouping of the time/frequency region adopted by the encoder side. During the further processing step using a finer time/frequency resolution (k, n), the parameters of the parameter band are valid for all time/frequency regions included in the parameter band, corresponding to the expansion that makes (l, m) → (k, n).

The coded transport channel is decoded by the core decoder. Using a filter bank (matching the filter bank used in the encoder), each frame of the audio signal so decoded is converted into a time/frequency representation whose resolution is typically better than (but at least equal to) that used for the parametric side information.

Output signal rendering/compositing

The following description applies to one frame of the audio signal; T represents the transpose operator:

Using the decoded transmission channels x=x(k,n)=[ _X1 (k,n), _X2 (k,n)] ^T , i.e. the audio signal in time-frequency representation (comprising two transmission channels in this case) and the parametric side information, a mixing matrix M is derived for each subframe (or frame for reducing computational complexity) to synthesize a time-frequency output signal y=y(k,n)=[ _Y1 (k,n), _Y2 (k,n), _Y3 (k,n),...] ^T comprising a plurality of output channels (e.g. 5.1, 7.1, 7.1+4, etc.):

For all (input) objects, using the sent object directions, so-called direct response values are determined, which describe the panning gains to be used for the output channels. These direct response values are specific to the target layout, i.e. the number and positions of loudspeakers (provided as part of the output configuration). Examples of panning methods include Vector Basis Amplitude Panning (VBAP) [Pulkki 1997] and Edge Fading Amplitude Panning (EFAP) [Borβ 2014]. Each object has a vector dr _i of direct response values associated with it (containing as many elements as there are loudspeakers). These vectors are calculated once per frame. Note: If the object position corresponds to a loudspeaker position, the vector contains the value 1 for that loudspeaker; all other values are 0. If the object is located between two (or three) loudspeakers, the corresponding number of non-zero vector elements is two (or three).

The actual synthesis step (in this example, covariance synthesis [Vilkamo2013]) consists of the following sub-steps (see Figure 5 for a visualization):

o For each parameter band, the object index describing the main subset of objects in the input objects grouped into the time/frequency regions in this parameter band is used to extract the vector subset _dri required for further processing. Since only eg 2 relevant objects are considered, 2 vectors _dri associated with these 2 relevant objects are required.

оBased on the direct response values _dri , the covariance matrix _Ci with the size of output channel * output channel is then calculated for each relevant object:

_Ci = _dri * _dri ^T

o For each time/frequency bin (within the parameter band), determine the audio signal power P(k,n). In the case of two transmission channels, add the signal power of the first channel to the signal power of the second channel. Multiply each of these power ratios by the signal power, resulting in a direct power value for each relevant/dominant object i:

DP _i (k, n) = PR _i (k, n) * P (k, n)

o For each frequency band k, the final target covariance matrix C _Y of size output channels * output channels is obtained by summing over all time slots n within the (sub)frame and over all relevant objects:

Fig. 5 shows a detailed overview of the covariance synthesis step performed in block 706 of Fig. 4. Specifically, the embodiment of Fig. 5 includes a signal power calculation block 721, a direct power calculation block 722, a covariance matrix calculation block 73, a target covariance matrix calculation block 724, an input covariance matrix calculation block 726, a mixing matrix calculation block 725 and a rendering block 727, which additionally includes the filter bank block 708 of Fig. 4 for Fig. 5, so that the output signal of block 727 preferably corresponds to the time domain output signal. However, when block 708 is not included in the rendering block of Fig. 5, the result is a spectral domain representation of the corresponding audio channel.

(The following steps are part of the state-of-the-art [Vilkamo2013] and are added for clarity.)

o For each (sub)frame and each frequency band, an input covariance matrix C _x = xx ^T of size transmission channel * transmission channel is calculated from the decoded audio signal. Optionally, only the entries of the main diagonal may be used, in which case the other non-zero entries are set to zero.

o A prototype matrix of size output_channels * transmission_channels is defined, which describes the mapping of transmission channels to output channels (provided as part of the output configuration), whose number is given by the target output format (e.g., target loudspeaker layout). This prototype matrix can be static or can vary from frame to frame. Example: If only a single transmission channel is sent, then that transmission channel is mapped to each output channel. If two transmission channels are sent, then the left (first) channel is mapped to all output channels at positions within (+0°, +180°), i.e., the "left" channels. The right (second) channel is correspondingly mapped to all output channels at positions within (-0°, -180°), i.e., the "right" channels. (Note: 0° describes a position in front of the listener, positive angles describe positions to the left of the listener, and negative angles describe positions to the right of the listener. If a different convention is used, the signs of the angles need to be adjusted accordingly.)

o Using the input covariance matrix C _x , the target covariance matrix C _Y and the prototype matrix, a mixing matrix [Vilkamo2013] is calculated for each (sub)frame and each frequency band, resulting in, for example, 60 mixing matrices per (sub)frame.

o The mixing matrix is interpolated (eg linearly) between (sub)frames, corresponding to temporal smoothing.

oFinally, the output channels y are synthesized band-by-band to the corresponding bands of the time/frequency representation of the decoded transport channel x by multiplying the final set of mixing matrices M by each of the size outputchannels*transmitchannels:

y＝Mx

Note that we do not use the residual signal r as described in [Vilkamo2013].

• Use a filter bank to convert the output signal y back to the time domain representation y(t).

Optimized Covariance Synthesis

Due to how the input covariance matrix _Cx and the target covariance matrix C _Y are calculated for this embodiment, certain optimizations of the optimal mixing matrix calculation using covariance synthesis from [Vilkamo2013] can be achieved, resulting in a significant reduction in the computational complexity of the mixing matrix calculation. Please note that in this section, the Hadamard operator. represents an element-by-element operation on the matrix, that is, instead of following the rules of, for example, matrix multiplication, the corresponding operation is performed element by element. This operator means: the corresponding operation is not performed on the entire matrix, but on each element separately. The multiplication of matrices A and B will, for example, not correspond to the matrix multiplication AB=C, but to the element-by-element operation a_ij*b_ij=c_ij.

SVD(.) stands for singular value decomposition. The algorithm from [Vilkamo2013] presented as a Matlab function (Listing 1) is as follows (prior art):

Input: A matrix C _x of size m×m containing the covariance of the input signal

Input: A matrix C _Y of size n×n containing the target covariance of the output signal

Input: matrix Q of size n×m, prototype matrix

Input: scalar α, regularization factor for S _x ([Vilkamo2013] recommends α = 0.2)

Input: scalar β, for Regularization factor ([Vilkamo2013] recommends β = 0.001)

Input: Boolean value a indicating whether energy compensation should be performed instead of calculating the residual covariance C _r

Output: Matrix M of size n×m, the optimal mixing matrix

Output: A matrix C _r of size n×n, containing the residual covariance

Decomposition of %C _Y ([Vilkamo2013]), equation (3))

As described in the previous section, only the main diagonal elements of C _x are optionally used, and all other entries are set to zero. In this case, C _x is a diagonal matrix, and the valid decomposition satisfying equation (3) of [Vilkamo2013] is

And the SVD of line 3 from the prior art algorithm is no longer needed.

Consider the formula for generating the target covariance from the direct response _dri and the direct power (or direct energy) in the previous section

_Ci = _dri * _dri ^T

DP _i (k, n) = PR _i (k, n) * P (k, n)

The last formula can be rearranged and written as

If we now define

Thus obtaining

It can be easily seen that if we arrange the direct responses of the k most dominant objects in the direct response matrix R = [dr ₁ ...dr _k ] and create a diagonal matrix of direct powers as E, where e _i,i = E _i , C _Y can also be expressed as

C _Y ＝RER ^H

And the effective decomposition of C _Y that satisfies equation (3) of [Vilkamo2013] is given by:

Therefore, the SVD of row 1 from the prior art algorithm is no longer needed.

This leads to an optimization algorithm for covariance synthesis within the present embodiment, which also takes into account that we always use the energy compensation option and therefore do not need a residual target covariance _Cr :

Input: A diagonal matrix C _x of size m×m containing the covariance of the input signal with m channels

Input: A matrix R of size n×k containing the direct responses to the k main subjects

Input: Diagonal matrix E containing the target powers of the main objects

Input: matrix Q of size n×m, prototype matrix

Input: scalar α, regularization factor S _x ([Vilkamo2013] recommends α = 0.2)

Input: scalar β, (Regularization factor [Vilkamo2013] recommends β = 0.001)

Output: Matrix M of size n×m, the optimal mixing matrix

%C _Y decomposition (innovative step)

A careful comparison of existing algorithms and the proposed algorithm shows that the former requires three SVDs of matrices of size m×m, n×n, and m×n, respectively, where m is the number of downmix channels and n is the number of output channels to which the object is rendered.

The proposed algorithm requires only one SVD of a matrix of size m×k, where k is the number of primary objects. Furthermore, since k is typically much smaller than n, this matrix is smaller than the corresponding matrix from the prior art algorithm.

For m×n matrices [Golub2013], the complexity of the standard SVD implementation is roughly O(c ₁ m ² n + c ₂ n ³ ), where c ₁ and c ₂ are constants that depend on the algorithm used. Therefore, the computational complexity of the proposed algorithm is significantly reduced compared to the state-of-the-art algorithms.

Subsequently, preferred embodiments related to the encoder side of the first aspect are discussed with respect to Figures 7a, 7b. Furthermore, preferred implementations of the encoder side implementation of the second aspect are discussed with respect to Figures 9a to 9d.

FIG. 7 a shows a preferred implementation of the object parameter calculator 100 of FIG. 1 a. In block 120, the audio object is converted into a spectral representation. This is achieved by the filter bank 102 of FIG. 2 or FIG. 3. Then, in block 122, the calculation of the selection information is, for example, as shown in block 104 of FIG. 2 or FIG. 3. For this purpose, an amplitude-related measurement value can be used, such as the amplitude itself, power, energy, or any other amplitude-related measurement value obtained by increasing the amplitude to a power different from 1. The result of block 122 is a set of selection information for each object in the corresponding time/frequency interval. Then, in block 124, an object ID for each time/frequency interval is derived. In a first aspect, two or more object IDs for each time/frequency interval are derived. According to a second aspect, the number of object IDs for each time/frequency interval can even be only a single object ID, so that the most important or strongest or most relevant object in the information provided by block 122 is identified in block 124. Block 124 outputs information about parameter data and includes a single or multiple indexes of the most relevant one or more objects.

In the case of two or more related objects per time/frequency interval, the functionality of block 126 is useful for calculating amplitude correlation measures characterizing the objects in the time/frequency interval. This amplitude correlation measure can be the same as the amplitude correlation measure already calculated for the selection information in block 122, or preferably, a combined value is calculated using information already calculated by block 102, as shown by the dotted line between block 122 and block 126, and then the amplitude correlation measure or one or more combined values are calculated in block 126 and forwarded to the quantizer and encoder block 212 so as to have an encoded amplitude correlation value or an encoded combined value in the auxiliary information as additional parameterized auxiliary information. In the embodiments of Figures 2 or 3, these values are "coded power ratios" included in the bitstream together with the "coded object index". In the case of only a single object ID per frequency interval, power ratio calculation and quantization encoding are not necessary, and the index of the most relevant object in the time-frequency interval is sufficient to perform decoder-side rendering.

FIG. 7 b shows a preferred implementation of the calculation of the selection information 102 of FIG. 7 b. As shown in block 123, the signal power is calculated as the selection information for each object and each time/frequency interval. Then, in block 125 showing a preferred implementation of block 124 of FIG. 7 a, the object ID of a single or preferably two or more objects with the highest power is extracted and output. In addition, in the case of two or more related objects, the calculation of the power ratio is shown in block 127 as a preferred implementation of block 126, wherein the power ratio is calculated for the extracted object ID, which is related to the power of all extracted objects with the corresponding object ID found by block 125. This process is advantageous because only the combined values whose number is less than 1 of the number of objects in the time/frequency interval must be sent, because in this embodiment there is a rule known to the decoder, which stipulates that the power ratios of all objects must sum to 1. Preferably, the functions of blocks 120, 122, 124, 126 of FIG. 7a and/or 123, 125, 127 of FIG. 7b are implemented by the object parameter calculator 100 of FIG. 1a, and the function of block 212 of FIG. 7a is implemented by the output interface 200 of FIG. 1a.

Subsequently, the device for encoding according to the second aspect shown in FIG. 1b is described in more detail with respect to several embodiments. In step 110a, the directional information is either extracted from the input signal, such as shown in FIG. 12a, or extracted by reading or parsing the metadata information included in the metadata part or metadata file. In step 110b, the directional information of each frame and audio object is quantized, and the quantization index of each frame and each object is forwarded to the encoder or output interface, such as the output interface 200 of FIG. 1b. In step 110c, the directional quantization index is dequantized so as to have a dequantized value that can also be directly output by block 110b in some implementations. Then, based on the dequantized directional index, block 422 calculates the weight of each transmission channel and each object based on a certain virtual microphone setting. The virtual microphone setting may include two virtual microphone signals arranged at the same position and having different orientations, or may be a setting in which there are two different positions relative to a reference position or orientation (e.g., a virtual listener position or orientation). A setting with two virtual microphone signals will result in weights for two transmission channels for each object.

In case three transmission channels are generated, the virtual microphone setup may be considered to include three virtual microphone signals from microphones arranged at the same position and with different orientations, or at three different positions relative to a reference position or orientation, where the reference position or orientation may be a virtual listener position or orientation.

Alternatively, four transmission channels may be generated based on a virtual microphone setup that generates four virtual microphone signals from microphones arranged at the same position and with different orientations, or that generates four virtual microphone signals from microphones arranged at four different positions relative to a reference position or a reference orientation, wherein the reference position or orientation may be a virtual listener position or a virtual listener orientation.

Furthermore, in order to calculate the weights wL and wR for each object and each transmission channel (taking two channels as an example), the virtual microphone signal is a signal derived from a virtual first-order microphone or a virtual cardioid microphone or a virtual figure-8 microphone or a dipole microphone or a bidirectional microphone, or is a signal derived from a virtual directional microphone or from a virtual sub-cardioid microphone or from a virtual unidirectional microphone or from a virtual supercardioid microphone or from a virtual omnidirectional microphone.

In this context, it should be noted that for calculating the weights, no placement of the actual microphones is required. Instead, the rules for calculating the weights vary depending on the virtual microphone setup, ie the placement of the virtual microphones and the characteristics of the virtual microphones.

In block 404 of FIG. 9 a, weights are applied to the objects so that for each object, the contribution of the object to a particular transmission channel is obtained with a weight different from 0. Thus, block 404 receives as input the object signal. Then, in block 406, the contribution of each transmission channel is summed so that, for example, the contribution of the object to a first transmission channel is added together and the contribution of the object to a second transmission channel is added together, etc. As shown in block 406, the output of block 406 is then the transmission channel, for example in the time domain.

Preferably, the object signal input to block 404 is a time domain object signal with full band information, and the applying in block 404 and the summing in block 406 are performed in the time domain. However, in other embodiments, these steps may also be performed in the spectral domain.

Figure 9b shows another embodiment of implementing a static downmix. To this end, the directional information of the first frame is extracted in block 130, and the weights are calculated depending on the first frame, as shown in block 403a. Then, for the other frames indicated in block 408, the weights remain as they are to implement a static downmix.

Fig. 9c shows another implementation of calculating dynamic downmixing. To this end, block 132 extracts the directional information of each frame, and updates the weight for each frame, as shown in block 403b. Then, in block 405, the updated weight is applied to the frame to realize the dynamic downmixing that changes from frame to frame. Other implementations between those extreme cases of Fig. 9b and Fig. 9c are also useful, wherein, for example, weights are updated only for every second, third or every nth frame, and/or weight smoothing over time is performed so that for the purpose of downmixing according to directional information, antenna characteristics will not change too much from time to time. Fig. 9d shows another implementation of the downmixer 400 controlled by the object direction information provider 110 of Fig. 1b. In block 410, the downmixer is configured to analyze the directional information of all objects in the frame, and in block 112, for the purpose of calculating the weights _wL and _wR of the stereo example, the microphone is placed to be consistent with the analysis result, wherein the placement of the microphone refers to the microphone position and/or the microphone direction. In block 414, similar to the static downmix discussed with respect to block 408 of FIG. 9b, the microphones are left for other frames, or updated as discussed with respect to block 405 of FIG. 9c, so as to obtain the functionality of block 414 of FIG. 9d. With respect to the functionality of block 412, the microphones may be placed so that good separation is obtained, so that the first virtual microphone "looks" at the first group of objects and the second virtual microphone "looks" at the second group of objects, the second group of objects being different from the first group of objects, and preferably different in that any object in one group is as little as possible from being included in the other group. Alternatively, the analysis of block 410 may be enhanced by other parameters, and the placement may also be controlled by other parameters.

Subsequently, preferred implementations of the decoder according to the first aspect or the second aspect and discussed with respect to, for example, Figs. 6a and 6b are given by the following Figs. 10a, 10b, 10c, 10d and 11.

In block 613, input interface 600 is configured to obtain individual object direction information associated with the object ID. This process corresponds to the functionality of block 612 of Figure 4 or Figure 5, and results in a "codebook of frames" as shown and discussed with respect to Figures 8b and particularly 8c.

In addition, in block 609, one or more object IDs of each time/frequency interval are obtained, regardless of whether these data are available for low-resolution parameter bands or high-resolution frequency areas. The result of block 609 corresponding to the process of block 608 in Fig. 4 is the specific ID of one or more related objects in the time/frequency interval. Then, in block 611, the specific object direction information of the specific one or more IDs of each time/frequency interval is obtained from "the codebook of the frame" (that is, from the exemplary table shown in Fig. 8c). Then, in block 704, for each time/frequency interval, the gain value of one or more related objects of each output channel controlled by the output format is calculated. Then, in block 730 or 706, 708, the output channel is calculated. The function of calculating the output channel can be performed in the explicit calculation of the contribution to one or more transmission channels as shown in Figure 10b, or can be performed by indirect calculation and use of the contribution to the transmission channel as shown in Figure 10d or Figure 11. Figure 10b shows the function of obtaining power value or power ratio in the block 610 corresponding to the function of Fig. 4. These power values are then applied to the respective transmission channels of each relevant object, as shown in blocks 733 and 735. Furthermore, these power values are applied to the respective transmission channels in addition to the gain values determined by block 704, so that blocks 733, 735 produce object-specific contributions of the transmission channels (e.g., transmission channels ch1, ch2, ...). These explicitly calculated channel transmission contributions are then added together in block 737 for each output channel of each time/frequency bin.

Then, depending on the implementation, a diffuse signal calculator 741 may be provided, which generates a diffuse signal for each output channel chl, ch2, ... in the corresponding time/frequency interval, and the combination of the diffuse signal and the contribution results of block 737 is combined so that the complete channel contribution in each time/frequency interval is obtained. When the covariance synthesis additionally depends on the diffuse signal, the signal corresponds to the input of the filter bank 708 of Figure 4. However, when the covariance synthesis 706 does not rely on the diffuse signal but only relies on the processing without any decorrelator, then at least the energy of the output signal of each time/frequency interval corresponds to the energy of the channel contribution at the output of block 739 of Figure 10b. In addition, without using the diffuse signal calculator 741, the result of block 739 corresponds to the result of block 706, i.e., the complete channel contribution with each time/frequency interval that can be converted separately for each output channel chl, ch2, so as to finally obtain an output audio file with time domain output channels, which can be stored or forwarded to a loudspeaker or any type of rendering device.

Figure 10c shows a preferred implementation of the functionality of the block 610 of Figure 10b or Figure 4. In step 610a, a combined (power) value or values are obtained for a certain time/frequency interval. In block 610b, corresponding other values of other related objects in the time/frequency interval are calculated based on the calculation rule that all combined values must sum to 1.

The result will then preferably be a low resolution representation with two power ratios for each packet slot index and each parameter band index. These power ratios represent low time/frequency resolution. In block 610c, the time/frequency resolution may be extended to a high time/frequency resolution such that it has power values for a time/frequency region with a high resolution slot index n and a high resolution band index k. The extension may include directly using one and the same low resolution index for corresponding slots within a packet slot and corresponding bands within a parameter band.

Figure 10d shows a preferred implementation of the function for calculating the covariance synthesis information in the block 706 of Figure 4, represented by a mixing matrix 725 for mixing two or more input transmission channels into two or more output signals. Therefore, when there are, for example, two transmission channels and six output channels, the size of the mixing matrix of each individual time/frequency interval will be six rows and two columns. In a block 723 corresponding to the function of the block 723 in Figure 5, the gain value or direct response value of each object in each time/frequency interval is received, and the covariance matrix is calculated. In block 722, power values or power ratios are received, and the direct power value of each object in the time/frequency interval is calculated, and the block 722 in Figure 10d corresponds to the block 722 of Figure 5.

The results of both blocks 721 and 722 are input to the target covariance matrix calculator 724. Additionally or alternatively, the explicit calculation of the target covariance matrix _Cy is not necessary. On the contrary, the relevant information included in the target covariance matrix (that is, for two or more related objects, the direct response value information indicated in the matrix R and the direct power value indicated in the matrix E) is input to the block 725a of the mixing matrix for calculating each time/frequency interval. In addition, the mixing matrix 725a receives information about the input covariance matrix _Cx and the prototype matrix Q derived from the two or more transmission channels shown in the block 726 corresponding to the block 726 of Fig. 5. The mixing matrix for each time/frequency bin and frame may be temporally smoothed, as shown in block 725b, and in block 727 corresponding to at least a portion of the rendering block of FIG5, the mixing matrix is applied in a non-smoothed or smoothed form to the transmission channels in the corresponding time/frequency bins to obtain the complete channel contributions in the time/frequency bins, substantially similar to the corresponding complete contributions discussed previously with respect to FIG10b at the output of block 739. Thus, FIG10b illustrates an implementation of explicit calculation of the transmission channel contributions, while FIG10d illustrates a process of implicitly calculating the transmission channel contributions for each time/frequency bin and each relevant object in each time-frequency bin via the target covariance matrix _Cy or via the relevant information R and E directly introduced into the mixing matrix calculation block 725a of blocks 723 and 722.

Subsequently, a preferred optimization algorithm for covariance synthesis is shown with respect to FIG. 11. It should be summarized that all steps shown in FIG. 11 are calculated in the covariance synthesis 706 of FIG. 4 or in the mixed matrix calculation block 725 of FIG. 5 or in 725a of FIG. 10d. In step 751, the first decomposition result _Ky is calculated. Due to the fact that, as shown in FIG. 10d, the gain value information included in the matrix R and the information from two or more related objects, specifically the direct power information included in the matrix ER, are directly used without explicitly calculating the covariance matrix, so the decomposition result can be easily calculated. Therefore, the first decomposition result in the block 751 can be directly calculated and without too much workload, because a specific singular value decomposition is no longer required.

In step 752, the second decomposition result is calculated as K _x . Since the input covariance matrix is regarded as a diagonal matrix with off-diagonal elements ignored, the decomposition result can also be calculated without explicit singular value decomposition.

Then, in step 753, a first regularization result based on the first regularization parameter α is calculated, and in step 754, a second regularization result is calculated based on the second regularization parameter β. Since _Kx is a diagonal matrix in the preferred implementation, the calculation of the first regularization result 753 is simplified relative to the prior art, because the calculation of _Sx is only a parameter change, rather than a decomposition as in the prior art.

Furthermore, for the calculation of the second regularization result in block 754, the first step is additionally only parameter renaming instead of multiplication with the matrix U _x ^HS in the prior art.

Furthermore, in step 755, the normalized matrix ^Gy is calculated, and based on step 755, in step 756, the unitary matrix P is calculated based on _Kx and the prototype matrix Q and the information of _Ky obtained by block 751. Since no matrix Λ is required here, the calculation of the unitary matrix P is simplified relative to the available prior art.

Then, in step 757, the mixing matrix without energy compensation, i.e. M _opt , is calculated, and for this purpose, the unitary matrix P, the result of block 754 and the result of block 751 are used. Then, in block 758, energy compensation is performed using the compensation matrix G. Energy compensation is performed so that no residual signal derived from the decorrelator is required. However, instead of performing energy compensation, in this implementation a residual signal with sufficient energy to fill the energy gap left by the mixing matrix M _opt without energy information will be added. However, for the purpose of the present invention, the decorrelated signal is not relied on to avoid any artifacts introduced by the decorrelator. However, energy compensation as shown in step 758 is preferred.

Therefore, the optimization algorithm for covariance synthesis provides advantages for the calculation of the unitary matrix P in steps 751, 752, 753, 754 and in step 756. It should be emphasized that the optimization algorithm even provides advantages over the prior art, wherein only one of steps 755, 752, 753, 754, 756 or only a subgroup of these steps is implemented, as shown in the figure, but the corresponding other steps are implemented as in the prior art. The reason is that these improvements do not depend on each other, but can be applied independently of each other. However, in terms of the complexity of implementation, the more improvements are implemented, the better the process will be. Therefore, the complete implementation of the embodiment of Figure 11 is preferred because it provides the greatest degree of complexity reduction, but even if only one of steps 751, 752, 753, 754, 756 is implemented according to the optimization algorithm and the other steps are implemented as in the prior art, complexity reduction is also obtained without any quality degradation.

Embodiments of the present invention may also be viewed as a process of generating comfort noise for a stereo signal by mixing three Gaussian noise sources (one for each channel) and a third common noise source for creating correlated background noise, or additionally or separately, controlling the mixing of the noise sources using a coherence value sent with a SID frame.

It is mentioned here that all alternatives or aspects discussed before and after and all aspects defined by the claims in the appended claims can be used separately, i.e., without any other alternatives or objectives different from the envisaged alternatives, objectives or independent claims. However, in other embodiments, two or more alternatives or aspects or independent claims can be combined with each other, and in other embodiments, all aspects or alternatives and all independent claims can be combined with each other.

The encoded signal of the present invention may be stored on a digital storage medium or a non-transitory storage medium, or may be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

Although some aspects have been described in the context of an apparatus, it will be clear that these aspects also represent a description of a corresponding method, wherein a block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also indicate a description of a feature of a corresponding block or item or a corresponding apparatus.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium (e.g., a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory) on which electronically readable control signals are stored, cooperating (or capable of cooperating) with a programmable computer system such that the corresponding method is performed.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative to perform one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a data carrier or a digital storage medium or a computer-readable medium having recorded thereon the computer program for performing one of the methods described herein.

Therefore, other embodiments of the method of the present invention are a data stream or a sequence of signals representing a computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by any hardware device.

The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the scope of the present invention be limited only by the scope of the appended patent claims and not by the specific details given by way of the description and explanation of the embodiments herein.

Aspects (used independently of each other, or with all other aspects, or only a subgroup of other aspects)

The apparatus, method or computer program may comprise one or more of the following features:

Examples of inventions with regard to novel aspects:

Multi-wave thinking combined with object encoding (using more than one directional cue per T/F zone)

An object encoding method that is as close as possible to the DirAC paradigm to allow any kind of input type in IVAS (object content not currently covered)

Inventive example about parameterization (encoder):

For each T/F zone: the selection information of the n most relevant objects in the T/F zone plus the power ratio between the contributions of these n most relevant objects

For each frame, for each object: one direction

Inventive example about rendering (decoder):

Get the direct response value for each relevant object from the sent object index and orientation information and the target output layout

Get the covariance matrix from the direct response

Calculate the direct power based on the downmix signal power and transmit power ratio of each relevant object

Obtain the final target covariance matrix from the direct power and covariance matrices

Only the diagonal elements of the input covariance matrix are used

Optimized Covariance Synthesis

Some side notes on the differences with SAOC:

Consider n main objects instead of all objects

→ Power ratio is therefore related to OLD, but is calculated differently

SAOC does not use direction at the encoder -> direction information is only introduced at the decoder (rendering matrix)

→SAOC-3D decoder receives object metadata for rendering matrices

SAOC uses the downmix matrix and sends the downmix gain

The embodiment of the present invention does not consider the diffusion degree

Subsequently, other examples of the present invention are summarized.

1. An apparatus for encoding a plurality of audio objects and associated metadata indicating directional information about the plurality of audio objects, comprising:

A downmixer (400) for downmixing the plurality of audio objects to obtain one or more transmission channels;

A transmission channel encoder (300) for encoding one or more transmission channels to obtain one or more encoded transmission channels; and

an output interface (200) for outputting a coded audio signal comprising said one or more coded transmission channels,

The downmixer (400) is configured to downmix the plurality of audio objects in response to directional information about the plurality of audio objects.

2. The apparatus of example 1, wherein the downmixer (400) is configured to:

generating two transmission channels as two virtual microphone signals arranged at the same position and with different orientations, or arranged at two different positions relative to a reference position or orientation, such as a virtual listener position or orientation, or

generating three transmission channels as three virtual microphone signals arranged at the same position and with different orientations, or arranged at three different positions relative to a reference position or orientation, such as a virtual listener position or orientation, or

generating four transmission channels as four virtual microphone signals arranged at the same position and with different orientations, or arranged at four different positions relative to a reference position or orientation such as a virtual listener position or orientation, or

The virtual microphone signal is a virtual first-order microphone signal, or a virtual cardioid microphone signal, or a virtual figure-8 or dipole or bidirectional microphone signal, or a virtual directional microphone signal, or a virtual sub-cardioid microphone signal, or a virtual unidirectional microphone signal, or a virtual supercardioid microphone signal, or a virtual omnidirectional microphone signal.

3. The apparatus of example 1 or 2, wherein the downmixer (400) is configured to:

For each audio object of the plurality of audio objects, deriving (402) weighting information for each transmission channel using directional information of the corresponding audio object;

weighting the corresponding audio object using the weighting information of the audio object for a specific transmission channel (404) to obtain an object contribution for the specific transmission channel, and

The object contributions of the plurality of audio objects to the specific transmission channel are combined (406) to obtain the specific transmission channel.

4. The device according to one of the preceding examples,

The downmixer (400) is configured to calculate the one or more transmission channels as one or more virtual microphone signals, the one or more virtual microphone signals being arranged at the same position and having different orientations, or being arranged at different positions relative to a reference position or orientation such as a virtual listener position or orientation, the directional information being related to the reference position or orientation,

Wherein the different positions or orientations are on or to the left of the center line and on or to the right of the center line, or wherein the different positions or orientations are uniformly or unevenly distributed to horizontal positions or orientations, for example, +90 degrees or -90 degrees relative to the center line, or -120 degrees, 0 degrees and +120 degrees relative to the center line, or wherein the different positions or orientations include at least one position or orientation pointing upward or downward relative to a horizontal plane where a virtual listener is located, and wherein the directional information about the multiple audio objects is related to the virtual listener position or a reference position or orientation.

5. The apparatus according to one of the preceding examples, further comprising:

A parameter processor (110) is used to quantize metadata indicating directional information about the plurality of audio objects to obtain quantized directional items of the plurality of audio objects,

The downmixer (400) is configured to: operate in response to the quantized direction term as the direction information, and

The output interface (200) is configured to introduce information about the quantization direction item into the encoded audio signal.

6. The device according to one of the preceding examples,

The downmixer (400) is configured to: perform analysis on directional information about the plurality of audio objects, and place one or more virtual microphones according to the result of the analysis so as to generate the transmission channel.

7. The device according to one of the preceding examples,

wherein the downmixer (400) is configured to: perform downmixing (408) using a downmixing rule that is static over a plurality of time frames, or

wherein the directional information is variable over a plurality of time frames, and wherein the downmixer (400) is configured to perform the downmix (405) using a downmix rule that is variable over the plurality of time frames.

8. The device according to one of the preceding examples,

The downmixer (400) is configured to perform downmixing in the time domain using sample-by-sample weighting and combination of samples of the plurality of audio objects.

9. The apparatus according to one of the preceding examples, further comprising:

An object parameter calculator (100) is configured to: calculate parameter data of at least two related audio objects for one or more frequency intervals of a plurality of frequency intervals associated with a time frame, wherein the number of the at least two related audio objects is less than the total number of the plurality of audio objects, and

The output interface (200) is configured to introduce information about parameter data of the at least two related audio objects for the one or more frequency intervals into the encoded audio signal.

10. The apparatus according to example 9, wherein the object parameter calculator (100) is configured to:

converting (120) each audio object of the plurality of audio objects into a spectral representation having the plurality of frequency bins,

calculating (122) selection information for each audio object for the one or more frequency intervals, and

deriving (124) an object identification as parameter data indicative of the at least two related audio objects based on the selection information, and

Therein, the output interface (200) is configured to introduce information about the object identification into the encoded audio signal.

11. An apparatus according to example 9 or 10, wherein the object parameter calculator (100) is configured to: quantize and encode (212) one or more amplitude-related measurement values of the relevant audio object in the one or more frequency intervals or one or more combined values derived from the amplitude-related measurement values as the parameter data, and

The output interface (200) is configured to introduce the quantized one or more amplitude-related measurement values or the quantized one or more combined values into the encoded audio signal.

2. The apparatus according to example 10 or 11,

wherein the selection information is an amplitude-related measurement value of the audio object, such as an amplitude value, a power value or a loudness value, or an amplitude raised to a power different from 1, and

wherein the object parameter calculator (100) is configured to: calculate (127) a combined value, such as a ratio of an amplitude-related measure of the associated audio object to a sum of two or more amplitude-related measures of the associated audio object, and

The output interface (200) is configured to introduce information about the combination value into the encoded audio signal, wherein the number of information items about the combination value in the encoded audio signal is at least equal to 1 and less than the number of related audio objects in the one or more frequency intervals.

13. The device according to any one of examples 10 to 12,

The object parameter calculator (100) is configured to select the object identifier based on the order of the selection information of the plurality of audio objects in the one or more frequency intervals.

14. The apparatus according to any one of examples 10 to 13, wherein the object parameter calculator (100) is configured to:

calculating (122) signal power as the selection information,

For each frequency interval, deriving (124) object identifications of two or more audio objects having maximum signal power values in one or more frequency intervals, respectively,

calculating (126) as the parameter data a power ratio between a sum of signal powers of two or more audio objects having the maximum signal power value and a signal power of at least one of the audio objects having the derived object identification, and

quantizing and encoding the power ratio (212), and

Wherein, the output interface (200) is configured to: introduce the quantized and encoded power ratio into the encoded audio signal.

15. The apparatus of any one of Examples 10 to 14, wherein the output interface (200) is configured to introduce the following into the encoded audio signal:

One or more coded transmission channels,

two or more coded object identifications of the audio objects associated with each of one or more of the plurality of frequency intervals in the time frame, and one or more coded combination values or coded amplitude related measurement values, as the parameter data, and

quantizing and encoding directional data for each audio object in the time frame, the directional data being constant for all frequency bins of the one or more frequency bins.

16. The apparatus according to any one of Examples 9 to 15, wherein the object parameter calculator (100) is configured to: calculate parameter data of at least the most dominant object and the second most dominant object in the one or more frequency intervals, or

wherein the number of audio objects in the plurality of audio objects is three or more, the plurality of audio objects include a first audio object, a second audio object, and a third audio object, and

The object parameter calculator (100) is configured to: for a first frequency interval among the one or more frequency intervals, only calculate a first group of audio objects such as the first audio object and the second audio object as the related audio objects; and for a second frequency interval among the one or more frequency intervals, only calculate a second group of audio objects such as the second audio object and the third audio object or the first audio object and the third audio object as the related audio objects, wherein the first group of audio objects is different from the second group of audio objects in at least one group member.

17. The apparatus according to any one of examples 9 to 16, wherein the object parameter calculator (100) is configured to:

calculating original parametric data having a first time or frequency resolution and combining the original parametric data into combined parametric data having a second time or frequency resolution lower than the first time or frequency resolution, and calculating parametric data for at least two related audio objects relative to the combined parametric data having the second time or frequency resolution, or

A parameter band having a second time or frequency resolution different from the first time or frequency resolution used in the time or frequency decomposition of the plurality of audio objects is determined, and parameter data for at least two related audio objects is calculated for the parameter band having the second time or frequency resolution.

18. A decoder for decoding a coded audio signal, the coded audio signal comprising: one or more transmission channels and direction information of a plurality of audio objects; and parameter data of the audio objects for one or more frequency intervals of a time frame, the decoder comprising:

an input interface (600) for providing the one or more transmission channels in the form of a spectral representation, the spectral representation having a plurality of frequency bins in the time frame; and

an audio renderer (700), configured to render the one or more transmission channels into a plurality of audio channels using the direction information,

The audio renderer (700) is configured to calculate direct response information (704) based on the one or more audio objects in each frequency interval of the multiple frequency intervals and direction information (810) associated with one or more related audio objects in the frequency interval.

19. The decoder according to example 18,

wherein the audio renderer (700) is configured to calculate (706) covariance synthesis information using the direct response information and the information about the plurality of audio channels (702), and to apply (727) the covariance synthesis information to the one or more transmission channels to obtain the plurality of audio channels, or

wherein the direct response information (704) is a direct response vector of each audio object in one or more audio objects, and wherein the covariance synthesis information is a covariance synthesis matrix, and wherein the audio renderer (700) is configured to perform matrix operations for each frequency interval when applying (727) the covariance synthesis information.

20. The decoder of example 18 or 19, wherein the audio renderer (700) is configured to:

When calculating the direct response information (704), direct response vectors of the one or more audio objects are derived, and for the one or more audio objects, a covariance matrix is calculated according to each direct response vector.

When calculating the covariance synthesis information, target covariance information is derived (724) from: a covariance matrix of an audio object or covariance matrices of multiple audio objects, power information about the corresponding one or more audio objects, and power information derived from the one or more transmission channels.

21. The decoder of example 20, wherein the audio renderer (700) is configured to:

When calculating the direct response information, direct response vectors of the one or more audio objects are derived, and for each of the one or more audio objects, a covariance matrix is calculated (723) based on each direct response vector,

deriving (726) input covariance information from the transmission channel, and

deriving (725a, 725b) mixing information from the target covariance information, the input covariance information and information about a plurality of channels, and

The mixing information is applied (727) to the transmission channels of each frequency interval in the time frame.

22. The decoder of example 21, wherein a result of applying the mixing information for each frequency bin in the time frame is converted (708) into the time domain to obtain a plurality of audio channels in the time domain.

23. The decoder of any one of Examples 18 to 22, wherein the audio renderer (700) is configured to:

in decomposing (752) an input covariance matrix derived from the transmission channel, using only the main diagonal elements of the input covariance matrix, or

performing a decomposition of a target covariance matrix using a direct response matrix and a power matrix of the subject or transmission channel (751), or

performing (752) a decomposition of the input covariance matrix by taking the root of each main diagonal element of the input covariance matrix, or

Compute (753) the regularized inverse of the factorized input covariance matrix, or

A singular value decomposition is performed (756) when computing the optimal matrix to be used for energy compensation without expanding the identity matrix.

24. A decoder according to any one of Examples 18 to 23, wherein the parameter data of the one or more audio objects includes parameter data of at least two related audio objects, wherein the number of the at least two related audio objects is less than the total number of the plurality of audio objects, and

The audio renderer (700) is configured to calculate, for each frequency interval in the one or more frequency intervals, a contribution from the one or more transmission channels based on first directional information associated with a first related audio object in the at least two related audio objects and based on second directional information associated with a second related audio object in the at least two related audio objects.

25. The decoder according to example 24,

The audio renderer (700) is configured to: for the one or more frequency intervals, ignore direction information of audio objects different from the at least two related audio objects.

26. The decoder according to example 24 or 25,

wherein the encoded audio signal comprises an amplitude-related measurement value of each relevant audio object in the parameter data or a combined value associated with at least two relevant audio objects, and

The audio renderer (700) is configured to: operate based on first directional information associated with a first related audio object among the at least two related audio objects and based on second directional information associated with a second related audio object among the at least two related audio objects, taking into account the contribution from the one or more transmission channels, or determine the quantitative contribution of the one or more transmission channels based on the amplitude-related measurement value or the combined value.

27. The decoder of example 26, wherein the encoded signal comprises a combined value in the parameter data, and

The audio renderer (700) is configured to determine the contribution of the one or more transmission channels using a combined value of one of the related audio objects and directional information of the one related audio object, and

The audio renderer (700) is configured to determine the contribution of the one or more transmission channels using a value derived from a combined value of another related audio object in the one or more frequency intervals and directional information of the other related audio object.

28. The decoder of any one of Examples 24 to 27, wherein the audio renderer (700) is configured to:

The direct response information is calculated based on the relevant audio object in each frequency interval of the plurality of frequency intervals and the direction information associated with the relevant audio object in the frequency interval (704).

29. The decoder according to example 28,

The audio renderer (700) is configured to determine (741) a diffuse signal for each of the multiple frequency intervals using diffuseness information such as a diffuseness parameter included in the metadata or a decorrelation rule, and to combine a direct response determined by the direct response information and the diffuse signal to obtain a spectral domain rendering signal for a channel of the multiple channels.

30. A method of encoding a plurality of audio objects and associated metadata indicating directional information about the plurality of audio objects, comprising:

downmixing the plurality of audio objects to obtain one or more transmission channels;

encoding the one or more transport channels to obtain one or more coded transport channels; and

outputting an encoded audio signal comprising said one or more encoded transmission channels,

The downmixing includes downmixing the multiple audio objects in response to directional information about the multiple audio objects.

31. A method for decoding an encoded audio signal, the encoded audio signal comprising: one or more transmission channels and direction information of a plurality of audio objects; and parameter data of the audio objects for one or more frequency intervals of a time frame, the method comprising:

providing the one or more transmission channels in the form of a frequency spectrum representation having a plurality of frequency bins in the time frame; and

rendering the one or more transmission channel audio into a plurality of audio channels using the direction information,

The audio rendering includes: calculating direct response information according to one or more audio objects in each frequency interval of the multiple frequency intervals and direction information associated with one or more related audio objects in the frequency interval.

32. A computer program for executing the method according to example 30 or the method according to example 31 when run on a computer or a processor.

2 References

[Pulkki2009] V. Pulkki, MV. Laitinen, J. Vilkamo, J. Ahonen, T. Lokki, and T. "Directional audio coding perception-based reproduction of spatial sound", International Workshop on the Principles and Application onSpatial Hearing, Nov.2009, Zao; Miyagi, Japan.

[SAOC_STD]ISO/IEC, "MPEG audio technologies Part 2: Spatial AudioObject Coding (SAOC)." ISO/IEC JTC1/SC29/WG11(MPEG)International Standard23003-2.

[SAOC_AES] J. Herre, H. Purnhagen, J. Koppens, O. Hellmuth, J. J. Hilpert, L. Villemoes, L. Terentiv, C. Falch, A. MLValero, B.Resch, H.MundtH, and H.Oh, "MPEGspatial audio object coding-the ISO/MPEG standard for efficient coding of interactive audio scenes," J.AES, vol.60, no.9, pp.655- 673, Sep.2012.

[MPEGH_AES] J. Herre, J. Hilpert, A. Kuntz, and J. Plogsties, "MPEG-H audio-the new standard for universal spatial/3D audio coding," in Proc.137 ^th AESConv., Los Angeles, CA , USA, 2014.

[MPEGH_IEEE] J.Herre, J.Hilpert, A.Kuntz, and J.Plogsties, "MPEG-H 3DAudio-The New Standard for Coding of Immersive Spatial Audio", IEEE JOURNAL OFSELECTED TOPICS IN SIGNAL PROCESSING, VOL.9, NO. .5, AUGUST 2015

[MPEGH_STD]Text of ISO/MPEG 23008-3/DIS 3D Audio, Sapporo, ISO/IECJTC1/SC29/WG11 N14747, Jul.2014.

[SAOC_3D_PAT] APPARATUS AND METHOD FOR ENHANCED SPATAL AUDIO OBJECTCODING，WO 2015/011024 A1

[Pulkki1997]V.Pulkki, "Virtual sound source positioning using vectorbase amplitude panning," J.Audio Eng.Soc., vol.45, no.6, pp.456-466, Jun.1997.

[DELAUNAY]C.B.Barber, D.P.Dobkin, and H.Huhdanpaa, "The quickhullalgorithm for convex hulls," in Proc.ACM Trans.Math.Software(TOMS), New York, NY, USA, Dec.1996, vol.22, pp.469-483.

[Hirvonen2009] T.Hirvonen, J.Ahonen, and V.Pulkki, "Perceptualcompression methods for metadata in Directional Audio Coding applied to audiovisual teleconference", AES 126 ^th Convention 2009, May 7-10, Munich, Germany.

[Borβ2014] C. Borβ, "A Polygon-Based Panning Method for 3D LoudspeakerSetups", AES 137 ^th Convention 2014, October 9-12, Los Angeles, USA.

[WO2019068638]Apparatus, method and computer program for encoding, decoding, scene processing and other procedures related to DirAC based spatialaudio coding, 2018

[WO2020249815]PARAMETER ENCODING AND DECODING FOR MULTICHANNEL AUDIOUSING DIRAC，2019

[BCC2001] C.Faller, F.Baumgarte: "Efficient representation of spatialaudio using perceptual parametrization", Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat.No.01TH8575).

[JOC_AES] Heiko Purnhagen; Toni Hirvonen; Lars Villemoes; Jonas Samuelsson; Janusz Klejsa: "Immersive Audio Delivery Using Joint ObjectCoding", ^140th AESConvention, Paper Number: 9587, Paris, May 2016.

[AC4_AES]K. J. M. Wolters, J. Riedmiller, A. Biswas, P. Ekstrand, A. P. Hedelin, T. Hirvonen, H. J.Klejsa, J.Koppens, K.Krauss, HM.Lehtonen, K.Linzmeier, H.Muesch, H.Mundt, S.Norcross, J.Popp, H.Purnhagen, J.Samuelsson, M.Schug, L. R.Thesing, L.Villemoes, andM.Vinton: "AC-4-The Next Generation Audio Codec", ^140th AES Convention, PaperNumber: 9491, Paris, May2016.

[Vilkamo2013] J. Vilkamo, T. A. Kuntz, "Optimized covariancedomain framework for time-frequency processing of spatial audio", Journal of the Audio Engineering Society, 2013.

[Golub2013]Gene H.Golub and Charles F.Van Loan, "Matrix Computations", Johns Hopkins University Press, 4th edition, 2013.

Claims (32)

1. An apparatus for encoding a plurality of audio objects, comprising: An object parameter calculator (100), configured to: for one or more frequency intervals in a plurality of frequency intervals associated with a time frame, calculate parameter data of at least two related audio objects, wherein the at least two related the number of audio objects is less than the total number of said plurality of audio objects, and An output interface (200) configured to output an encoded audio signal comprising information about parameter data of said at least two associated audio objects of said one or more frequency intervals. 2. The apparatus according to claim 1, wherein the object parameter calculator (100) is configured to: converting (120) each audio object of the plurality of audio objects into a spectral representation having a plurality of frequency bins, calculating (122) selection information for each audio object of said one or more frequency intervals, and Based on said selection information, deriving (124) an object identification as parameter data indicative of said at least two related audio objects, and Wherein said output interface (200) is configured to introduce information about said object identification into said encoded audio signal. 3. The apparatus according to claim 1 or 2, wherein the object parameter calculator (100) is configured to: one or more amplitude-related measurements of related audio objects in the one or more frequency intervals value or one or more combined values derived from magnitude-related measurements are quantized and encoded (212) as said parameter data, and Wherein, the output interface (200) is configured to introduce quantized one or more amplitude-related measurement values or quantized one or more combined values into the encoded audio signal. 4. The device according to claim 2 or 3, wherein said selection information is an amplitude-related measure of said audio object, such as an amplitude value, a power value or a loudness value, or an amplitude raised to a power different from 1, and Wherein said object parameter calculator (100) is configured to calculate (127) a combined value, such as a ratio of an amplitude-related measure of a related audio object to a sum of two or more amplitude-related measures of the related audio object, as well as Wherein, the output interface (200) is configured to introduce information about the combined value into the encoded audio signal, wherein the number of information items about the combined value in the encoded audio signal is at least equal to 1 and less than the number of related audio objects for the one or more frequency bins. 5. The device according to one of claims 2 to 4, Wherein, the object parameter calculator (100) is configured to select the object identifier based on an order of selection information of the plurality of audio objects in the one or more frequency intervals. 6. The apparatus according to one of claims 2 to 5, wherein the object parameter calculator (100) is configured to: calculating (122) signal power as said selection information, deriving (124) object identities corresponding to two or more audio objects having maximum signal power values in one or more frequency intervals for each frequency interval, respectively, calculating (126) a power ratio between the sum of the signal powers of the two or more audio objects having said maximum signal power value and the signal power of each of the audio objects having the derived object identity as the parameter data, and quantizing and encoding (212) the power ratio, and Wherein said output interface (200) is configured to introduce the quantized and encoded power ratio into said encoded audio signal. 7. The apparatus according to one of claims 1 to 6, wherein the output interface (200) is configured to introduce into the encoded audio signal: one or more encoded transport channels, As said parametric data, two or more coded object identifiers of associated audio objects for each of one or more of the plurality of frequency intervals in said time frame, and one or more coded combination values or coded magnitude-related measurements, and Quantized and encoded orientation data for each audio object in the time frame, the orientation data being constant for all of the one or more frequency bins. 8. The apparatus according to one of claims 1 to 7, wherein the object parameter calculator (100) is configured to: calculate at least the most dominant object and the second most dominant the parameter data of the object, or Wherein, the number of audio objects in the plurality of audio objects is three or more, and the plurality of audio objects includes a first audio object, a second audio object and a third audio object, and Wherein, the object parameter calculator (100) is configured to: for the first frequency interval in the one or more frequency intervals, only calculate the first audio object such as the first audio object and the second audio object a set of audio objects as said related audio objects; and for a second frequency interval in said one or more frequency intervals, only computing such as said second audio object and said third audio object or said first audio object and a second group of audio objects of the third audio object as the related audio objects, wherein the first group of audio objects differs from the second group of audio objects in at least one group member. 9. The apparatus according to one of claims 1 to 8, wherein the object parameter calculator (100) is configured to: computing raw parametric data having a first time or frequency resolution and combining said raw parametric data into combined parametric data having a second time or frequency resolution lower than said first time or frequency resolution , and calculating parameter data of said at least two related audio objects with respect to combined parameterization data having said second time or frequency resolution, or determining a parameter band having a second time or frequency resolution different from a first time or frequency resolution used in the time or frequency decomposition of the plurality of audio objects, and for A parameter strip computes parameter data for said at least two related audio objects. 10. The apparatus according to one of the preceding claims, wherein said plurality of audio objects comprises associated metadata indicating directional information (810) about said plurality of audio objects, and Wherein, the device also includes: A downmixer (400), configured to downmix the plurality of audio objects to obtain one or more transmission channels, wherein the downmixer (400) is configured to: respond to information about the plurality of audio downmixing the plurality of audio objects using the direction information of the objects; and a transport channel encoder (300), configured to encode one or more transport channels to obtain one or more encoded transport channels; and Wherein, the output interface (200) is configured to introduce the one or more transmission channels into the encoded audio signal. 11. The apparatus according to claim 10, wherein the downmixer (400) is configured to: generating two transmission channels as two virtual microphone signals arranged at the same position and having different orientations, or at two different positions or orientations relative to a reference such as a virtual listener position or orientation at the location, or generating three transmission channels as three virtual microphone signals arranged at the same position and having different orientations, or at three different positions or orientations relative to a reference such as a virtual listener position or orientation at the location, or The four transmission channels are generated as four virtual microphone signals arranged at the same position and with different orientations, or at four positions or orientations relative to a reference position or orientation such as a virtual listener position or orientation. at different locations, or Wherein, the virtual microphone signal is a virtual first-order microphone signal, or a virtual cardioid microphone signal, or a virtual 8-shaped or dipole or two-way microphone signal, or a virtual directional microphone signal, or a virtual subcardioid microphone signal, or a virtual single directional microphone signal, or virtual hypercardioid microphone signal, or virtual omnidirectional microphone signal. 12. The apparatus according to claim 10 or 11, wherein the downmixer (400) is configured to: For each audio object of the plurality of audio objects, deriving (402) weighting information for each transmission channel using direction information of the corresponding audio object; weighting (404) an audio object for a particular transport channel using the weighting information of the corresponding audio object to obtain an object contribution for the particular transport channel, and Combining (406) the object contributions of the plurality of audio objects to the particular transport channel to obtain the particular transport channel. 13. Device according to one of claims 10 to 12, Wherein, the down-mixer (400) is configured to: calculate the one or more transmission channels as one or more virtual microphone signals, the one or more virtual microphone signals being arranged at the same position and having different orientations , or arranged at a different position relative to a reference position or orientation, such as a virtual listener position or orientation, to which the direction information is related, wherein said different positions or orientations are on or to the left of said center line and on or to the right of said center line, or wherein said different positions or orientations are distributed evenly or unevenly to horizontal positions or Orientation, such as +90 degrees or -90 degrees relative to the centerline, or -120 degrees, 0 degrees and +120 degrees relative to the centerline, or wherein the different positions or orientations include relative to the virtual listener At least one position or orientation pointing upwards or downwards on the horizontal plane, wherein the direction information about the plurality of audio objects is related to the virtual listener position or reference position or orientation. 14. The apparatus of any one of claims 10 to 13, further comprising: a parameter processor (110), configured to quantize metadata indicating direction information about the plurality of audio objects, to obtain quantized direction items for the plurality of audio objects, Wherein, the downmixer (400) is configured to: operate in response to a quantization direction term as the direction information, and Wherein, the output interface (200) is configured to introduce information about the quantization direction term PA231367 into the encoded audio signal. 15. Device according to one of claims 10 to 14, Wherein, the downmixer (400) is configured to: perform (410) an analysis on the directional information about the plurality of audio objects, and place (412) one or more virtual microphones according to the results of the analysis to generate transmission channel. 16. Device according to one of claims 10 to 15, Wherein, the downmixer (400) is configured to: downmix (408) using a downmix rule that is static over multiple time frames, or wherein the direction information is variable over a plurality of time frames, and wherein the downmixer (400) is configured to: downmix using a downmix rule that is variable over the plurality of time frames mixed (405). 17. The apparatus according to one of claims 10 to 16, the downmixer (400) being configured to downmix in the time domain using a sample-by-sample weighted and combined combination of samples of the plurality of audio objects. 18. A decoder for decoding an encoded audio signal comprising: one or more transmission channel and direction information for a plurality of audio objects, and one or more frequency intervals for a time frame Parameter data of at least two related audio objects, wherein the number of said at least two related audio objects is lower than the total number of said plurality of audio objects, said decoder comprising: an input interface (600) for providing said one or more transmission channels in a spectral representation having a plurality of frequency bins in said time frame; and An audio renderer (700), configured to render the one or more transmission channels into a plurality of audio channels using the direction information, so that according to the considering the contribution from the one or more transmission channels based on the first direction information of the at least two related audio objects and the second direction information associated with the second related audio object of the at least two related audio objects, or Wherein, the audio renderer (700) is configured to: for each frequency interval in the one or more frequency intervals, according to the first associated audio object in the at least two associated audio objects Contributions from the one or more transmission channels are calculated based on the first direction information and second direction information associated with a second related audio object of the at least two related audio objects. 19. The decoder of claim 18, Wherein, the audio renderer (700) is configured to ignore direction information of audio objects different from the at least two related audio objects for the one or more frequency intervals. 20. A decoder as claimed in claim 18 or 19, wherein said encoded audio signal comprises an amplitude-related measure (812) for each associated audio object in said parameter data or a combined value (812) associated with at least two associated audio objects, and Wherein said audio renderer (700) is configured to determine (704) a quantitative contribution of said one or more transmission channels from said magnitude-related measurement or said combined value. 21. A decoder according to claim 20, wherein an encoded signal comprises said combined value in said parameter data, and Wherein said audio renderer (700) is configured to: determine (704, 733) the contribution of said one or more transmission channels using a combined value of one of the related audio objects and direction information of the one related audio object, as well as Wherein, the audio renderer (700) is configured to: use the combined value of another related audio object from among the related audio objects in the one or more frequency intervals and the direction information of the another related audio object The derived value determines (704, 735) the contribution of the one or more transmission channels. 22. The decoder according to one of claims 18 to 21, wherein the audio renderer (700) is configured to: Direct response information is calculated (704) based on the associated audio objects for each frequency bin of the plurality of frequency bins and direction information associated with the associated audio objects in the frequency bins. 23. The decoder of claim 22, Wherein, the audio renderer (700) is configured to: use diffuseness information included in the metadata, such as a diffuseness parameter, or a decorrelation rule to determine (741) the a diffuse signal for each frequency bin, and combining the direct response determined by the direct response information and the diffuse signal to obtain a spectral domain rendered signal for a channel of the plurality of channels, or calculating (706) composite information using said direct response information (704) and information (702) about said plurality of audio channels, and applying (727) covariance composite information to said one or more transmission channels, to obtain the plurality of audio channels, or wherein said direct response information (704) is a direct response vector for each associated audio object, and wherein said covariance synthesis information is a covariance synthesis matrix, and wherein said audio renderer (700) is configured to : performing a matrix operation for each frequency bin when applying (727) said covariance synthesis information. 24. The decoder according to claim 22 or 23, wherein the audio renderer (700) is configured to: When calculating said direct response information (704), derive a direct response vector for each relevant audio object; and for each relevant audio object, calculate a covariance matrix from each direct response vector, In computing said covariance composite information, target covariance information is derived (724) from: a covariance matrix from each of said related audio objects, power information about the corresponding associated audio object, and Power information derived from the one or more transmission channels. 25. The decoder of claim 24, wherein the audio renderer (700) is configured to: In computing said direct response information (704), deriving a direct response vector for each associated audio object; and for each associated audio object, computing (723) a covariance matrix from each direct response vector, deriving (726) input covariance information from said transmission channel, and deriving (725a, 725b) blending information from said target covariance information, said input covariance information and information about said plurality of channels, and Applying (727) the mixing information to the transmission channels of each frequency bin in the time frame. 26. The decoder of claim 25, wherein the result of applying the mixing information for each frequency bin in the time frame is transformed (708) into the time domain to obtain a plurality of audio frequencies in the time domain aisle. 27. The decoder according to one of claims 22 to 26, wherein the audio renderer (700) is configured to: When decomposing (752) the input covariance matrix derived from said transmission channel, only the main diagonal elements of said input covariance matrix are used, or performing a decomposition (751) of the target covariance matrix using the direct response matrix and the power matrix of the subject or transmission channel, or performing (752) the decomposition of the input covariance matrix by taking the root of each main diagonal element of the input covariance matrix, or computing (753) the regularized inverse of the decomposed input covariance matrix, or A singular value decomposition is performed (756) in computing the optimal matrix to be used for energy compensation without expanding the identity matrix. 28. A method of encoding a plurality of audio objects and associated metadata indicating directional information about the plurality of audio objects, comprising: downmixing the plurality of audio objects to obtain one or more transport channels; encoding the one or more transmission channels to obtain one or more encoded transmission channels; and outputting an encoded audio signal comprising said one or more encoded transport channels, Wherein, the downmixing includes downmixing the plurality of audio objects in response to direction information about the plurality of audio objects. 29. A method of decoding an encoded audio signal comprising: one or more transmission channels and direction information of a plurality of audio objects, and at least two of one or more frequency intervals for a time frame Parameter data of related audio objects, wherein the number of said at least two related audio objects is lower than the total number of said plurality of audio objects, said decoding method comprising: providing said one or more transmission channels in a spectral representation having a plurality of frequency bins in said time frame; and rendering the one or more transport channel audio into a plurality of audio channels using the direction information, Wherein, the audio rendering includes: for each frequency interval of the one or more frequency intervals, according to the first direction information associated with the first related audio object among the at least two related audio objects and according to Computing the contribution from the one or more transmission channels with second direction information associated with a second related audio object of the at least two related audio objects, or making the contribution from the at least two related audio objects According to the first direction information associated with the first related audio object of the at least two related audio objects and the second direction information associated with the second related audio object of the at least two related audio objects, considering the transmission channel from the one or more contribute. 30. A computer program for carrying out the method according to claim 28 or the method according to claim 29 when run on a computer or a processor. 31. An encoded audio signal comprising information on parametric data of at least two related audio objects of one or more frequency intervals. 32. The encoded audio signal of claim 31 , further comprising: one or more encoded transport channels, Two or more coded object identities for each of one or more of the plurality of frequency bins in the time frame, and one or more coded combination values or coded amplitude related measures of the associated audio object value, as information about the parameter data, and Quantized and encoded orientation data for each audio object in the time frame, the orientation data being constant for all of the one or more frequency bins.