ES2913849T3 - Concept for audio encoding and decoding for audio channels and audio objects - Google Patents

Abstract

Audio encoder for encoding audio input data (101) to output audio data (501) comprising: an input interface (100) configured to receive a plurality of audio channels, a plurality of audio objects, and metadata related to one or more of the plurality of audio objects; a mixer (200) configured to mix the plurality of audio objects and the plurality of audio channels to obtain a plurality of premixed channels, each premixed channel comprising audio data of one audio channel and audio data of at least minus one audio object; a core encoder (300) configured to centrally encode input data from the core encoder; and a metadata compressor (400) configured to compress metadata related to one or more of the plurality of audio objects to obtain compressed metadata; and wherein the audio encoder is configured to operate in both a first mode and a second mode of a group of at least two modes comprising the first mode, in which the central encoder (300) is configured to encode individually the plurality of audio channels and the plurality of audio objects received by the input interface (100) as the input data of the central encoder without any interaction by the mixer (200), and the second mode, in the which the core encoder (300) is configured to receive, as input data from the core encoder, the plurality of premixed channels generated by the mixer (200) and to encode the plurality of premixed channels generated by the mixer (200) , and an output interface (500) for providing an output signal as the audio output data (501), the output signal comprising, when the audio encoder operates in the first mode, cana encoded audio files in the first mode, encoded audio channels and encoded audio objects as an output of the core encoder (300) and the compressed metadata, and comprising the output signal, when the audio encoder operates in the second mode, the output of the core encoder (300) without any metadata related to the one or more of the plurality of audio objects.

Description

DESCRIPTION

Concept for audio encoding and decoding for audio channels and audio objects

The present invention relates to audio encoding/decoding and, in particular, to spatial audio encoding and encoding of a spatial audio object.

Spatial audio coding tools are well known in the art and are, for example, standardized in the MPEG surround standard. Spatial audio encoding begins with original input channels such as five or seven channels that are identified by their placement in a playback configuration, i.e., a left channel, a center channel, a right channel, a left surround channel, a right surround channel and a low-frequency power channel. A spatial audio encoder typically derives one or more downmix channels from the original channels, and additionally derives parametric data related to spatial signals such as level differences between channels in channel coherence values, phase differences between channels, time differences between channels, etc. One or more downmix channels are transmitted along with parametric side information indicating the spatial signals to a spatial audio decoder which decodes the downmix channel and associated parametric data in order to finally obtain output channels that are a rough version of the original input channels. The placement of the channels in the output configuration is normally fixed and is, for example, a 5.1 format, a 7.1 format, etc.

Additionally, spatial audio object encoding tools are well known in the art and are standardized in the MPEG SAOC standard (SAOC = Spatial Audio Object Encoding). In contrast to a spatial audio encoding that starts at original channels, the encoding of a spatial audio object starts with audio objects that are not automatically dedicated for a certain render playback configuration. Instead, the placement of audio objects in the playback scene is flexible and can be determined by the user by inputting certain rendering information to a spatial audio object encoding decoder. Alternatively or additionally, rendering information, ie information about the position at which an audio object should be placed in the playback configuration typically over time may be transmitted as additional side information or metadata. In order to obtain a certain data compression, a number of audio objects are encoded by means of an SAOC encoder that calculates, from the input objects, one or more transport channels by downmixing the objects. according to some downmix information. Additionally, the SAOC encoder calculates parametric side information that represents signals between objects such as object level differences (OLD), object coherence values, etc. As in SAC (SAC = Spatial Audio Coding), the inter-object parametric data is computed for individual time/frequency tiles, i.e. for a given frame of the audio signal comprising, for example, 1024 or 2048 samples , 24, 32, or 64, etc., frequency bands are considered in such a way that, in the end, there is parametric data for each frame and each frequency band. As an example, when an audio piece has 20 frames and when each frame is subdivided into 32 frequency bands, then the number of time/frequency tiles is 640.

Until now there is no flexible technology that combines channel coding on the one hand and object coding on the other hand in such a way that acceptable audio qualities are obtained at low bit rates.

WO 201212544 A1 discloses a comprehensive solution for creating, encoding, transmitting, decoding and playing spatial audio soundtracks. The provided soundtrack encoding format is compatible with legacy surround sound encoding formats, so that soundtracks encoded in the new format can be decoded and played on legacy playback equipment without losing quality compared to legacy formats . Audio objects are included in a base downmix at the encoder side and the thus obtained downmix and explicitly encoded audio objects are transmitted to a decoder side. On the decoder side, objects are removed from the transmitted downmix and are independently rendered and combined with the residual downmix corresponding to the base downmix.

US 2010324915 A1 discloses an encoding apparatus for a multichannel high quality audio codec (HQMAC) and a decoding apparatus for the HQMAC. Encoding/decoding apparatus for HQMAC can perform channel-based high-quality multi-channel audio codec (HQMAC-CB) encoding or HQMAC-CB decoding based on characteristics of input audio signals to provide lower channel compatibility .

It is an object of the present invention to provide an improved concept for audio coding.

This object is achieved by an audio decoder according to claim 1, an audio encoding method according to claim 8 or a computer program according to claim 9.

The present invention is based on the finding that an optimal system that is flexible on the one hand and provides good compression efficiency with good audio quality on the other hand is achieved by the combination of spatial audio coding, i.e. , channel-based audio coding with coding of a spatial audio object, ie object-based coding. In particular, providing a mixer to mix the objects and channels that are already on the encoder side provides good flexibility, particularly for low bitrate applications, since any transmission of objects may then be unnecessary or the number of objects to be transmitted can be reduced. On the other hand, flexibility is required such that the audio encoder can be controlled in two different ways, i.e. in the mode in which the objects are mixed with the channels before encoding to the kernel, while in the other mode object data on the one hand and channel data on the other hand are encoded directly to the kernel without any mixing between them.

This ensures that the user can either separate the processed objects and channels on the encoder side such that full flexibility is available on the decoder side but at the price of enhanced bit transfer. On the other hand, when the bit transfer requirements are more stringent, then the present invention already allows for pre-rendering mixing on the encoder side, i.e. some or all of the audio objects are already mixed with the audio channels. such that the core encoder only encodes channel data and no required bits are required to transmit audio object data, either in the form of a downmix or in the form of parametric inter-object data.

On the decoder side, the user again has high flexibility due to the fact that the same audio decoder allows operation in two different modes, i.e. the first mode where the encoding of individual or independent channels and objects takes place and the decoder has the complete flexibility to process the objects and mix with the channel data. On the other hand, when a mixdown/pre-render has already been performed on the encoder side, the decoder is configured to perform post-rendering without rendering any intermediate objects. On the other hand, post-processing can also be applied to the data in the other mode, i.e. when object rendering/mixing takes place on the decoder side. Thus, the present invention enables a processing task framework that allows for high resource reuse not only on the encoder side but also on the decoder side. Post-processing may refer to downmixing and binarization or any other processing to obtain a final channel scenario such as an intended playback layout.

Additionally, in the case of very low bit rate requirements, the present invention provides the user with sufficient flexibility to react to low bit rate requirements, i.e. by pre-rendering on the encoder side. such that, for the price of some flexibility, nonetheless very good audio quality on the decoder side is obtained due to the fact that the bits that have been saved by no longer providing any encoder object data to the decoder can be used to better encode the channel data such as by finer quantization of the channel data or by other means to improve quality or to reduce encoding loss when sufficient bits are available.

In a preferred embodiment of the present invention, the encoder additionally comprises a SAOC encoder and additionally allows not only encoding of object input to the encoder but also SAOC encoding of channel data in order to obtain good quality. audio even at lower bit rates. Additional embodiments of the present invention allow post-processing functionality comprising a binaural renderer and/or a format converter. Additionally, it is preferred that the full processing on the decoder side already takes place for a certain large number of loudspeakers, such as a 22 or 32 channel loudspeaker configuration. However, after the format converter, for example, determines that only 5.1 output is required, that is, output for a playback arrangement that has less than the maximum number of channels, then it is preferred that the converter control both the USAC decoder and the SAOC decoder or both devices to restrict the core decoding operation and the SAOC decoding operation so that no channel is generated on decoding which, in the end, is nonetheless mismixed. descending manner in a format conversion. Typically, the generation of upmixed channels requires decorrelation processing and each decorrelation processing introduces some level of artifacts. Therefore, by controlling the core decoder and/or the SAOC decoder by the ultimately required output format, a large amount of additional decorrelation processing is saved when compared to a situation when this interaction does not exist, which not only results in improved audio quality but also results in reduced decoder complexity and ultimately in reduced power consumption, which is particularly useful for mobile devices that encompass the encoder of the invention or the decoder of the invention. The inventive encoders or corresponding decoders, however, can not only be inserted into mobile devices such as mobile phones, smartphones, laptops or navigation devices, but can also be used in simple desktop computers or other non-mobile devices. .

The above implementation, ie not outputting some channels, may not be optimal, since some information (such as the level difference between the channels to be downmixed) may be lost. This level difference information may not be critical, but may result in a different downmix output signal, if the downmix applies different downmix gains to the upmixed channels. An improved solution only turns off demapping on the upmix, but still outputs all upmix channels with correct level differences (as indicated by the parametric SAC). The second solution results in better audio quality, but the first solution results in a further reduction in complexity.

Preferred embodiments are discussed below with respect to the accompanying drawings, in which: Figure 1 illustrates a first embodiment of an encoder;

Figure 2 illustrates a first example of a decoder not forming part of the invention;

Figure 3 illustrates a second embodiment of an encoder;

Figure 4 illustrates a second example of a decoder not forming part of the invention;

Figure 5 illustrates a third embodiment of an encoder;

Figure 6 illustrates a third example of a decoder not forming part of the invention;

Figure 7 illustrates a map indicating individual modes in which encoders according to embodiments of the present invention or exemplary decoders may be operated;

Figure 8 illustrates a specific implementation of the format converter;

Figure 9 illustrates a specific implementation of the binaural converter;

Figure 10 illustrates a specific implementation of the core decoder which is not part of the invention; and Figure 11 illustrates a specific implementation of an encoder for processing a Quad Channel Element (QCE) and corresponding QCE decoder not forming part of the invention.

The examples related to the decoder do not form part of the invention but are useful for its understanding.

Encoder-related embodiments form part of the invention.

Figure 1 illustrates an encoder according to one embodiment of the present invention. The encoder is configured to encode audio input data 101 to obtain audio output data 501. The encoder comprises an input interface for receiving a plurality of audio channels indicated by CH and a plurality of audio objects indicated by OBJ. Additionally, as illustrated in FIG. 1, input interface 100 additionally receives metadata related to one or more of the plurality of OBJ audio objects. Additionally, the encoder comprises a mixer 200 for mixing the plurality of objects and the plurality of channels to obtain a plurality of premixed channels, wherein each premixed channel comprises audio data from one channel and audio data from per channel. least one object. Additionally, the encoder comprises a core encoder 300 for centrally encoding input data from the core encoder, a metadata compressor 400 for compressing metadata related to one or more of the plurality of audio objects. Additionally, the encoder may comprise a mode controller 600 for controlling the mixer, the core encoder, and/or an output interface 500 in one of many modes of operation, wherein in the first mode, the core encoder is configured to encode the plurality of audio channels and the plurality of audio objects received by the input interface 100 without any interaction by the mixer, that is, without any mixing done by the mixer 200. In a second mode, however, in which mixer 200 was active, the central encoder encodes the plurality of mixed channels, ie, the output generated by block 200. In the latter case, it is preferred not to encode any object data anymore. Instead, the metadata indicating positions of the audio objects is already used by the mixer 200 to render the objects on the channels as indicated by the metadata. In other words, the mixer 200 uses the metadata related to the plurality of audio objects to pre-render the audio objects and then the pre-rendered audio objects are mixed with the channels to obtain mixed channels at the output of the mixer. In this embodiment, any objects may not necessarily be transmitted and this also applies to metadata compressed as output by block 400. However, if not all objects input to interface 100 are shuffled but only a certain number of objects are shuffled , then only the remaining unscrambled objects and associated metadata are nonetheless transmitted to the central encoder 300 or metadata compressor 400, respectively.

Figure 3 illustrates a further embodiment of an encoder further comprising a SAOC encoder 800. The SAOC encoder 800 is configured to generate one or more transport channels and parametric data from the encoder input data. of spatial audio objects. As illustrated in Figure 3, the input data of the spatial audio object encoder are objects that have not been processed by the prerender/mixer. Alternatively, provided that the prerender/mixer has been passed through as in mode one where a single object/channel encoding is active, all object inputs at input interface 100 are encoded by SAOC encoder 800.

Additionally, as illustrated in Figure 3, the core encoder 300 is preferably implemented as a USAC encoder, that is, as an encoder as defined and standardized in the MPEG-USAC standard (USAC = speech and audio coding). unified). The output of the complete encoder illustrated in Figure 3 is an MPEG 4 data stream having container-like structures for individual data types. Additionally, the metadata is indicated as "OAM" data and the metadata compressor 400 in Figure 1 corresponds to the OAM encoder 400 to obtain compressed OAM data that is input to the USAC encoder 300 which, as can be seen in Figure 3 , additionally comprises the output interface for obtaining the MP4 output data stream which not only has encoded object/channel data but also has the compressed OAM data.

Figure 5 illustrates a further embodiment of the encoder, where in contrast to Figure 3, the SAOC encoder can be configured interchangeably to encode, with the SAOC encoding algorithm, the channels provided in the pre-render/mixer 200 that is not active in this mode or, alternatively, to SAOC-encode the pre-rendered channels plus objects. Thus, in FIG. 5, SAOC encoder 800 can operate on three different classes of input data, ie, channels with no pre-rendered objects, channels and pre-rendered objects, or objects only. Additionally, it is preferred to provide an additional OAM decoder 420 in Figure 5 such that the SAOC encoder 800 uses, for its processing, the same data found on the decoder side, i.e. data obtained by a lossy compression instead of the original OAM data.

The encoder of figure 5 can work in several individual modes.

In addition to the first and second modes, as discussed in the context of Figure 1, the Figure 5 encoder may additionally operate in a third mode in which the central encoder generates the one or more transport channels from of the individual objects when the pre-render/mixer 200 was not active. Alternatively or additionally, in this third mode, the SAOC encoder 800 may generate one or more alternate or additional transport channels from the original channels, i.e. again when the pre-render/mixer 200 corresponding to the mixer 200 of Figure 1 was not active.

Finally, the SAOC encoder 800 may encode, when the encoder is configured in the fourth mode, the channels plus pre-rendered objects as generated by the pre-render/mixer. Thus, in the fourth mode lower bit rate applications will provide good quality due to the fact that the channels and objects have been completely transformed into individual SAOC transport channels and associated side information as indicated in the figures 3 and 5 as "SAOC-SI" and, additionally, any uncompressed metadata does not have to be transmitted in this fourth mode.

Figure 2 illustrates an exemplary decoder. The decoder receives, as input, the encoded audio data, i.e. data 501 in Figure 1.

The decoder comprises a metadata decompressor 1400, a core decoder 1300, an object processor 1200, a mode controller 1600, and a post processor 1700.

Specifically, the audio decoder is configured to decode encoded audio data and the input interface is configured to receive the encoded audio data, the encoded audio data comprising a plurality of encoded channels and the plurality of encoded objects and related compressed metadata with the plurality of objects in a certain mode.

Additionally, the core decoder 1300 is configured to decode the plurality of encrypted channels and the plurality of encrypted objects, and additionally, the metadata decompressor is configured to decompress the compressed metadata.

Additionally, object processor 1200 is configured to process the plurality of decoded objects as generated by core decoder 1300 using decompressed metadata to obtain a predetermined number of output channels comprising object data and the decoded channels . These output channels as indicated at 1205 are then input to a post processor 1700. Post processor 1700 is configured to convert the number of output channels 1205 to a certain output format which can be either a binaural output format or a video format. speaker output, such as 5.1,7.1 output format, etc.

Preferably, the decoder comprises a mode controller 1600 that is configured to analyze the encoded data to detect a mode indication. Therefore, the mode controller 1600 is connected to the input interface 1100 in FIG. 2. Alternatively, however, the mode controller need not be there. Instead, the soft decoder may be preconfigured by any other kind of control data such as user input or any other control. The audio decoder in Figure 2, preferably controlled by mode controller 1600, is configured to both bypass the object processor and feed the plurality of decoded channels into post processor 1700. This is mode 2 operation, that is that is, in which only the pre-rendered channels are received, that is, when mode 2 has been applied in the encoder of Figure 1. Alternatively, when mode 1 has been applied in the encoder, that is, when the encoder has performed individual channel/object encoding, object processor 1200 is not bypassed, but the plurality of decoded channels and the plurality of decoded objects are fed to object processor 1200 along with decompressed metadata generated by metadata decompressor 1400.

Preferably, the indication of whether mode 1 or mode 2 should be applied is included in the encoded audio data and then the mode controller 1600 analyzes the encoded data to detect a mode indication. Mode 1 is used when the mode indication indicates that the coded audio data comprises coded channels and coded objects and mode 2 is applied when the mode indication indicates that the coded audio data does not contain any audio objects, i.e. , only contain pre-rendered channels obtained by mode 2 of the encoder of Figure 1.

Figure 4 illustrates another preferred embodiment compared to the decoder of Figure 2 and the example of Figure 4 corresponds to the encoder of Figure 3. In addition to the implementation of the decoder of Figure 2, the decoder in Figure 4 comprises a decoder of SAOC 1800. Additionally, the object renderer 1200 of FIG. 2 is implemented as a separate object renderer 1210 and mixer 1220 while, depending on the mode, the functionality of the object renderer 1210 may also be implemented by the SAOC 1800 decoder.

Additionally, post processor 1700 may be implemented as a binaural renderer 1710 or format converter 1720. Alternatively, a direct data output 1205 of FIG. 2 may also be implemented as illustrated by 1730. Therefore, it is preferred to perform the processing in the decoder on the higher number of channels such as 22.2 or 32 for flexibility and then post-processing if a smaller format is required. However, when it is clear from the very beginning that only a small format such as a 5.1 format is required, then it is preferred, as indicated by Figure 2 or 6 by shortcut 1727, that some control over the decoder SAOC and/or USAC decoder may be applied in order to avoid unnecessary upmix operations and subsequent downmix operations.

In one example, object processor 1200 comprises SAOC decoder 1800 and the SAOC decoder is configured to decode one or more transport channels broadcast by the core decoder and associated parametric data and use decompressed metadata to obtain the plurality of objects from rendered audio. Up to this point, the OAM output is connected to box 1800.

Additionally, the object processor 1200 is configured to render decoded objects output by the core decoder that are not encoded on the SAOC transport channels but are individually encoded into elements on individual channels as indicated by the object renderer. 1210. Additionally, the decoder comprises an output interface corresponding to output 1730 for outputting a mixer output to speakers.

In a further example, object processor 1200 comprises a spatial audio object encoding decoder 1800 for decoding one or more transport channels and associated parametric side information representing encoded audio objects or encoded audio channels, wherein the decoder spatial audio object coding is configured to transcode the associated parametric information and decompressed metadata into transcoded parametric side information usable for direct rendering of the output format, as defined for example in an earlier version of SAOC. Post processor 1700 is configured to compute audio channels of the output format using the decoded transport channels and transcoded parametric side information. The processing performed by the post processor may be similar to MPEG surround processing or it may be any other processing such as BCC processing or the like.

In a further example, object processor 1200 comprises a spatial audio object encoding decoder 1800 configured to directly upmix and render channel signals to the output format using the decoded transport channels (by the core decoder). and parametric lateral information.

Additionally, and importantly, the object processor 1200 of Figure 2 further comprises the mixer 1220 which receives, as input, data generated by the USAC decoder 1300 directly when there are pre-rendered objects mixed with channels, that is, when the mixer 200 of Figure 1 was active. Additionally, mixer 1220 receives data from the object renderer which performs object rendering without SAOC decoding. Additionally, the mixer receives output data from the SAOC decoder, ie objects rendered by SAOC.

The 1220 mixer connects to the 1730 output interface, the 1710 binaural renderer, and the 1720 format converter. The 1710 binaural renderer is configured to render the output channels into two binaural channels using head-related transfer functions or responses to binaural room pulses (BRIR). The format converter 1720 is configured to convert the output channels to an output format that has a smaller number of channels than the output channels 1205 of the mixer and the format converter 1720 requires information about the playback layout such as 5.1 speaker. or the like.

The figure 6 decoder is different from the figure 4 decoder because SAOC decoder can not only generate rendered objects but also rendered channels and this is the case when the figure 5 encoder has been used and the connection 900 between the channels /pre-rendered objects and the input interface of the SAOC 800 encoder is active.

Additionally, a vector basis amplitude panning (VPAP) stage 1810 is configured which receives from the SAOC decoder the playback layout information and which outputs a rendering matrix to the SAOC decoder such that the SAOC decoder can, in the end, provide rendered channels without any additional mixer operation in the high channel format of 1205, ie 32 loudspeakers.

The VBAP block preferably receives the decoded OAM data to derive the render arrays. More generally, it preferably requires geometric information not only of the playback layout but also of the positions where the input signals are to be rendered in the playback layout. This geometric input data may be OAM data for channel position information or objects for channels that have been transmitted using SAOC.

However, if only a specific output interface is required, then the VBAP 1810 state may already provide the required render array for the output, eg 5.1. The SAOC decoder 1800 then performs a direct rendering of the SAOC transport channels, associated parametric data and uncompressed metadata, a direct rendering in the required output format without any interaction from the mixer 1220. However, when applying a certain mixing between modes, i.e. where several channels are encoded with SAOC but not all channels are encoded with SAOC or where several objects are encoded with SAOC but not all objects are encoded with SAOC or when only a certain number of objects pre-rendered with channels are SAOC-decoded and the remaining channels are not SAOC-processed then the mixer will unify the data from the individual input portions, i.e. directly from the core decoder 1300, object renderer 1210 and SAOC decoder 1800.

Subsequently, Figure 7 is discussed to indicate certain encoder/decoder modes that can be implemented by the high quality and highly flexible audio encoder concept of the invention or the high quality and highly flexible concept for the inventive decoders.

According to the first encoding mode, the mixer 200 in the encoder of Figure 1 is bypassed and therefore the object processor in the decoder of Figure 2 is not bypassed.

In the second mode, the mixer 200 in Figure 1 is active and the object renderer in Figure 2 is passed through.

Then, in the third encoding mode, the SAOC encoder of Figure 3 is active but only SAOC encodes objects instead of channels or channels as output by the mixer. Therefore, mode 3 requires that, on the decoder side illustrated in Figure 4, the SAOC decoder is only active for objects and generates rendered objects.

In a fourth encoding mode as illustrated in Figure 5, the SAOC encoder is configured for SAOC encoding of pre-rendered channels, ie the mixer is active as in the second mode. On the decoder side, SAOC decoding is performed for pre-rendered objects in such a way that the object renderer is passed through as in the second encoding mode.

Additionally, there is a fifth encoding mode that can be mixed by any of modes 1 to 4. In particular, a mixed encoding mode will exist when the mixer 1220 in Figure 6 receives channels directly from the USAC decoder and, in a way, Additionally, receive channels with pre-rendered objects from the USAC decoder. Additionally, in this shuffle coding mode, objects are coded directly using, preferably, a single channel element of the USAC decoder. In this context, the object renderer 1210 will then render these decoded objects and send them to the mixer 1220. Additionally, various objects are further encoded by a SAOC encoder such that the SAOC decoder will output rendered objects to the mixer. and/or rendered channels when there are several channels encoded by SAOC technology.

Each input portion of mixer 1220 may then, exemplarily, have at least one potential to receive the number of channels such as 32 as indicated at 1205. Thus, basically, the mixer could receive 32 channels from the USAC decoder. and additionally 32 mixed/prerendered channels of the USAC decoder and additionally 32 object renderer “channels” and additionally 32 SAOC decoder “channels”, where each “channel” between the blocks 1210 and 1218 on the one hand and block 1220 on the other hand have a contribution from the corresponding objects on a corresponding speaker channel and then mixer 1220 mixes, ie adds individual contributions for each speaker channel.

In a preferred embodiment of the present invention, the encoding/decoding system is based on an MPEG-D USAC codec to encode the channel and object signals. To increase the efficiency for encoding a large number of objects, the SAOC technology of MPEG has been adapted. Three types of renderers perform the task of rendering objects to channels, rendering channels to headphones, or rendering channels to different speaker configurations. When object signals are transmitted explicitly or parametrically encoded using SAOC, the corresponding object metadata information is compressed and multiplexed into the encoded output data.

In one embodiment, mixer/prerender 200 is used to convert an object plus channel input scene to a channel scene prior to encoding. Functionally, this is identical to the mixer/object processor combination on the decoder side as illustrated in Figure 4 or Figure 6 and as indicated by the object processor 1200 of Figure 2. Object pre-rendering ensures a determining signal entropy at the encoder input which is basically independent of the number of simultaneously active object signals. With object pre-rendering, object metadata transmission is not required. The individual object signals are rendered to the channel layout that the encoder is configured to use. The weight of the objects for each channel is obtained from the associated object OAM metadata as indicated by arrow 402.

As an encoder/decoder/core for speaker channel signals, individual object signals, object downmix signals and pre-rendered signals, a USAC technology is preferred. It handles the encoding of the large number of signals by creating channel and object mapping information (the semantic and geometric information of the input channel and object mapping). This mapping information describes how input objects and channels are mapped to USAC channel elements as illustrated in Figure 10, i.e. Channel Pair Elements (CPE), Single Channel Elements (SCE), quad channel (QCE) and the corresponding information is transmitted to the central decoder from the central encoder. All additional payloads such as SAOC data or object metadata have been passed through the extension elements and accounted for in the encoder's rate control.

Object encoding is possible in different ways, depending on the bitrate/distortion requirements and the interactivity requirements for the renderer. The following object encoding variants are possible:

• Pre-rendered Objects: Object signals are pre-rendered and mixed with the 22.2 channel signals before encoding. The subsequent encoding chain sees signals from channel 22.2.

• Individual object waveforms: Objects are supplied as mono waveforms to the encoder. The encoder uses simple channel elements SCE to transmit the objects in addition to the channel signals. The decoded objects are rendered and mixed on the receiver side. The object's compressed metadata information is passed to the receiver/renderer all the way.

• Parametric object waveforms: The properties of the objects and their relationship to each other are described by the SAOC parameters. The downmix of the object signals is encoded with USAC. The parametric information is transmitted in its entire length. The number of channels for downmixing is chosen depending on the number of objects and the overall data rate. The object's compressed metadata information is passed to the SAOC renderer.

The SAOC encoder and decoder for the object signals are based on SAOC MPEG technology. The system is capable of recreating, modifying and rendering a number of audio objects based on fewer transmitted channels and additional parametric data (OLD, IOC (Inter Object Coherence), DMGs (Down Mix Gains)). The additional parametric data exhibits a data rate significantly lower than that required to transmit all the objects individually, making the encoding very efficient.

The SAOC encoder takes the object/channel signals as monophonic waveforms as input and outputs the parametric information (which is packaged into the 3D Audio bitstream) and the SAOC transport channels (which are encoded with the use of simple and transmitted channel elements).

The SAOC decoder reconstructs the decoded SAOC transport channel object/channel signals and parametric information, and generates the output audio scene based on the playback layout, uncompressed object metadata information, and shape. optional to interaction with user information.

For each object, the associated metadata specifying the geometric position and volume of the object in 3D space is efficiently encoded by quantifying the properties of the object in time and space. The compressed metadata of cOAM objects is transmitted to the receiver as side information. The volume of the object may comprise information about a spatial degree and/or signal level information of the audio signal of this audio object.

The object renderer uses the object's compressed metadata to generate object waveforms according to the given playback format. Each object is rendered to certain output channels according to its metadata. The emission of this block is the result of the sum of the partial results.

If both channel-based content as well as individual/parametric objects are decoded, the channel-based waveforms and the rendered object waveforms are mixed before the resulting waveforms are output (or before they are fed into a post-processor module). such as the binaural renderer or the speaker renderer module).

The binaural renderer module produces a binaural downmix of multichannel audio material, such that each input channel is represented by a virtual sound source. Processing is done as frames in the QMF (Quadrature Mirror Filterbank) domain.

Binarization is based on measured responses to binaural room pulses.

Figure 8 illustrates an example of the format converter 1720. The loudspeaker renderer or format converter converts between the transmit channel setting and the desired playback format. This format converter converts down to a smaller number of output channels, ie creates downmixes. Up to this point, a downmix device 1722 preferably operating in the QMF domain receives output signals from the mixer 1205 and outputs signals from the loudspeaker. Preferably, a controller 1724 is provided to configure the downmixer 1722 which receives, as control input, a mixer output layout, i.e., the layout for which data 1205 is determined, and a normally desired playback layout. is entered into the format conversion block 1720 illustrated in Figure 6. Based on this information, the controller 1724 preferably automatically generates downmix matrices optimized for the given combination of input and output formats and applies these matrices to the input and output formats. downmix device block 1722 in the downmix process. The format converter allows for standard speaker configurations as well as random configurations with non-standard speaker positions.

As illustrated in the context of Figure 6, the SAOC decoder is designed to render to the predefined channel layout such as 22.2 with subsequent format conversion to the 22.2 channel layout. wanted playback. Alternatively, however, the SAOC decoder is implemented to support "low power" mode where the SAOC decoder is configured to decode to the playback layout directly without subsequent format conversion. In this implementation, the SAOC decoder 1800 directly outputs the speaker signal such as 5.1 speaker signals and the SAOC decoder 1800 requires the playback layout and rendering matrix information such that the amplitude panning of vector base or any other kind of processor to generate downmix information can operate.

Figure 9 illustrates a further example of the binaural renderer 1710 of Figure 6. Specifically, for mobile devices, binaural rendering is required for headphones attached to such mobile devices or for speakers directly attached to typically small mobile devices. For such mobile devices, there may be limitations to limit the complexity of the decoder and rendering. In addition to omitting decorrelation in such processing scenarios, downmixing using downmixer 1712 is preferred first to intermediate downmixing, i.e., a smaller number of output channels which then results in a smaller number of output channels. amount of input channel to binaural converter 1714. As an example, material from channel 22.2 is downmixed by means of downmixer 1712 to an intermediate 5.1 downmix or, alternatively, the intermediate downmix is calculated directly by the SAOC decoder 1800 of FIG. 6 in a kind of "direct access" mode. So binaural rendering only has to apply ten HRTFs (Head Related Transfer Functions) or BRIRs to render the five individual channels at different positions, in contrast to applying 44 HRTFs for BRIRs if the 22.2 input channels are already have rendered directly. Specifically, the convolution operations required for binaural rendering require a large amount of processing power, and therefore reducing this processing power while still achieving acceptable audio quality is particularly useful for mobile devices.

Preferably, "direct access" as illustrated by control line 1727 comprises controlling set-top box 1300 to decode to a smaller number of channels, i.e., skipping the entire OTT processing block at the set-top box or a format that is converted to a smaller number of channels and, as illustrated in Figure 9, a binaural rendering is performed for the smaller number of channels. The same processing can be applied not only for binaural processing but also for format conversion as illustrated by line 1727 in Figure 6.

In a further example, efficient interface generation between processing blocks is required. In particular, in Figure 6, the path of the audio signal between the different processing blocks is represented. The binaural renderer 1710, the format converter 1720, the SAOC decoder 1800 and the USAC decoder 1300, in the case that SBR (spectral band replication) is applied, all operate in a QMF or hybrid QMF domain. According to one example, all of these processing blocks provide a QMF or hybrid QMF interface to allow audio signals to pass each other in the QMF domain in an efficient manner. Additionally, it is preferred to implement the mixer module and object renderer module to work in the QMF or hybrid QMF domain as well. As a consequence, the stand-alone QMF or hybrid QMF synthesis and analysis steps can be avoided resulting in considerable savings in complexity and then only a final QMF synthesis step is required to generate the indicated loudspeakers in 1730 or to generate the data. binaurals at the broadcast of block 1710 or to generate playback of layout loudspeaker signals at the broadcast of block 1720. Reference is then made to Fig. 11 for the purpose of explaining Quad Channel Elements (QCE). In contrast to a channel pair element as defined in the USAC-MPEG standard, a quad channel element requires four input channels 90 and produces an encoded QCE element 91. In one embodiment, a hierarchy of two surround boxes is provided. MPEG in 2-1-2 mode or two TTO boxes (TTO = two to one) and additional joint stereo encoding tools (eg MS-Stereo) as defined in MPEG USAC or MPEG Surround and QCE element not only comprises two co-stereo encoded downmix channels and optionally two co-stereo encoded residual channels and additionally parametric data derived from, for example, two TTO boxes. On the decoder side, a structure is applied where the joint stereo decoding of the two downmix channels and optionally two residual channels are applied and in a second stage with two OTT boxes the downmix and optional residual channels are subjected. upmix to all four output channels. However, alternative processing operations for a QCE encoder can be applied instead of the hierarchical operation. Thus, in addition to the joint channel coding of a group of two channels, the central encoder/decoder additionally uses a joint channel coding of a group of four channels.

Additionally, it is preferred to perform an enhanced noise filling procedure to allow uncommitted full band (18 kHz) encoding at 1200 kbps.

The encoder has been operated in a 'constant rate with bit reservation' mode, using a maximum of 6144 bits per channel as rate buffer for dynamic data.

All additional payloads such as SAOC data or object metadata have been passed through the extension elements and accounted for in the encoder's rate control.

In order to take advantage of SAOC functionalities also for 3D audio content, the following extensions for MPEG SAOC have been implemented:

• Downmix to arbitrary number of SAOC transport channels.

• Enhanced rendering for high speaker count output configurations (up to 22.2).

The binaural renderer module produces a binaural downmix of multi-channel audio material such that each input channel (not including LFE channels) is represented by a virtual sound source. The processing is carried out in the form of frames in the QMF domain.

Binarization is based on measured responses to binaural room pulses. The direct sound and early reflections are imprinted on the audio material by means of a convolutional approach in a pseudo FFT domain using fast convolution on top of the QMF domain. Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or element or feature of a corresponding apparatus. Some or all of the steps of the method may be executed by (or using) a hardware apparatus, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some of the most important steps of the method may be performed by said apparatus.

Depending on certain implementation requirements, embodiments of the invention may be implemented in hardware or in software. The implementation can be done using a non-transient storage medium such as a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM and EPROM, an EEPROM or a flash memory. , which has electronically readable control signals stored inside it, which cooperate (or are capable of cooperating) with a programmable computer system in such a way that the respective method is carried out. Therefore, the digital storage medium can be read by a computer.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

In general, embodiments of the present invention may be implemented as a computer program product with program code, the program code being operative to perform one of the methods when the computer program product is run on a computer. The program code may, for example, be stored on a machine-readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier.

In other words, an embodiment of the method of the invention is thus a computer program having program code for performing one of the methods described herein, when the computer program is run on a computer.

A further embodiment of the method of the invention is therefore a data carrier (either a digital storage medium or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described in the present document. The data carrier, digital storage medium, or recording medium is typically tangible and/or non-transient.

A further embodiment of the method of the invention is thus a data stream or sequence of signals representing the computer program for performing one of the methods described herein. The data stream or signal sequence can, for example, be configured to be transferred via a data communication connection, eg via the Internet.

A further embodiment comprises processing means, eg, a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed the software for performing one of the methods described herein.

A further embodiment in accordance with the invention comprises an apparatus or system configured to transfer (eg, electronically or optically) a computer program for performing one of the methods described herein to a recipient. The receiver can, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the recipient.

In some embodiments, a programmable logic device (eg, a field-programmable logic gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable logic gate array may cooperate with a microprocessor in order to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

The embodiments described above are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to others skilled in the art. It is intended, therefore, to be limited only by the scope of the imminent patent claims and not by the specific details depicted by way of description and explanation of the embodiments herein.

Claims (1)

Audio encoder for encoding audio input data (101) to obtain audio output data (501) comprising: an input interface (100) configured to receive a plurality of audio channels, a plurality of audio objects, and metadata related to one or more of the plurality of audio objects; a mixer (200) configured to mix the plurality of audio objects and the plurality of audio channels to obtain a plurality of premixed channels, each premixed channel comprising audio data of one audio channel and audio data of at least minus one audio object; a core encoder (300) configured to centrally encode input data from the core encoder; Y a metadata compressor (400) configured to compress metadata related to one or more of the plurality of audio objects to obtain compressed metadata; Y wherein the audio encoder is configured to operate in both a first mode and a second mode of a group of at least two modes comprising the first mode, in which the core encoder (300) is configured to individually encode the plurality of audio channels and the plurality of audio objects received by the input interface (100) as the input data of the core encoder without any interaction by the mixer (200), and the second mode, in which the core encoder (300) is configured to receive, as input data from the core encoder, the plurality of premixed channels generated by the mixer (200) and to encode the plurality of premixed channels generated by the mixer (200), and an output interface (500) for providing an output signal as audio output data (501), the output signal comprising, when the audio encoder operates in the first mode, audio channels encoded in the first mode, audio channels encoded and audio objects encoded as an output of the core encoder (300) and the compressed metadata, and the output signal comprising, when the audio encoder operates in the second mode, the output of the core encoder (300) without any metadata related to the one or more of the plurality of audio objects. Audio encoder according to claim 1, further comprising: a spatial audio object encoder (800) for generating one or more transport channels and parametric data from the input data of the spatial audio object encoder, wherein the audio encoder is configured to additionally operate in a third mode, in which the core encoder (300) encodes the one or more transport channels derived from the input data of the spatial audio object encoder, the input data of the spatial audio object encoder comprising the plurality of audio objects or two or more of the plurality of audio channels. Audio encoder according to claim 1 or claim 2, further comprising: a spatial audio object encoder (800) for generating one or more transport channels and parametric data from the input data of the spatial audio object encoder, wherein the audio encoder is configured to operate in yet another mode, in which the core encoder (300) encodes transport channels derived from the spatial audio object encoder (800) from the premixed channels as the input data from the spatial audio object encoder. Audio encoder according to any one of the preceding claims, further comprising a connector to connect an output of the input interface (100) to an input of the core encoder (300) in the first mode, and to connect the output of the input interface (100) to an input of the mixer (200) and to connect an output of the mixer (200) to the input of the central encoder (300) in the second mode, and a mode controller ( 600) to control the connector according to a mode indication received from a user interface or extracted from the audio input data (101). Audio encoder according to claim 2, further comprising: wherein the output interface (500) is configured to provide an output signal as the audio output data (501), the output signal comprising, in the third mode, an output from the core encoder (300), information SAOC side information and compressed metadata and comprising, even further, an output from the core encoder (300) and SAOC side information. Audio encoder according to any one of the preceding claims, wherein the mixer (200) is configured to pre-render the plurality of audio objects using the metadata and an indication of the position of each audio channel in a playback configuration with which the plurality of channels is associated, wherein the mixer (200) is configured to mix an audio object with at least two audio channels, where the audio object shall be placed between at least two audio channels in the playback configuration, as determined by the metadata. Audio encoder according to any one of the preceding claims, further comprising a metadata decompressor (420) for decompressing compressed metadata outputted by the metadata compressor (400), and wherein the mixer (200) is configured to mix the plurality of audio objects according to compressed metadata, wherein a compression operation performed by the metadata compressor (400) is a lossy compression operation comprising a quantification stage. Audio input data encoding method (101) to obtain audio output data (501) comprising: receiving (100) a plurality of audio channels, a plurality of audio objects, and metadata related to one or more of the plurality of audio objects; mixing (200) the plurality of audio objects and the plurality of audio channels to obtain a plurality of premixed channels, each premixed channel comprising audio data of one audio channel and audio data of at least one object of Audio; centrally encrypting (300) centrally encrypted input data; compressing (400) metadata related to one or more of the plurality of audio objects, wherein the audio input data encoding method (101) operates in both a first mode and a second mode of a group of two or more modes comprising: the first mode, in which the core encoding (300) individually encodes the received plurality of audio channels as the core encoding input data and the received plurality of audio objects as the core encoding input data without any interaction by mixing (200), and the second mode, in which the core encoding (300) receives, as the core encoding input data, the plurality of premixed channels generated by the coremixer (200) and centrally encodes the plurality of premixed channels generated by the mixed (200); Y provide an output signal as the audio output data (501), the output signal comprising, when the audio input data encoding method (101) is in the first mode, encoded audio channels and encoded audio objects as an output of the core encoding and compressed metadata, and the output signal comprising, when the audio input data encoding method (101) is in the second mode, the output of the core encoding without any metadata related to one or more of the plurality of audio objects. A computer program comprising instructions which, when the program is executed by a computer or processor, causes the computer or processor to carry out the method of claim 8.