RU2798019C2 - Audio data processing based on a directional volume map - Google Patents

Abstract

FIELD: computer technology for processing audio data.

SUBSTANCE: invention relates to increasing the accuracy of coding the input audio content. It is achieved through the steps that provide one or more coded audio signals based on one or more input audio signals or one or more signals received from them; and adapting (3200) generating one or more coded audio signals depending on one or more directional loudness maps that represent loudness information associated with a plurality of different directions of one or more signals to be coded.

EFFECT: increasing the accuracy of coding the input audio content.

33 cl, 34 dwg

Description

The field of technology to which the invention belongs

Embodiments in accordance with the invention relate to processing audio data based on a directional loudness map.

State of the art

With the advent of perceptual audio encoders, there has been considerable interest in developing algorithms that can predict the quality of audio data for encoded signals without relying on extensive subjective listening tests to save time and resources. There are widespread algorithms that perform the so-called objective quality assessment of monaural encoded signals, such as PEAQ [3] or POLQA [4]. However, their performance for signals encoded with spatial audio techniques is still considered unsatisfactory [5]. In addition, non-waveform-preserving techniques such as bandwidth expansion (BWE) are also known for overestimating quality loss [6] because many of the features extracted for analysis assume waveform-preserving conditions. Spatial audio and BWE techniques are mainly used when encoding audio data at a low bit rate (about 32 kbps per channel).

It is contemplated that spatial audio content of more than two channels can be transformed by converting for listening into a binaural representation of signals entering the left and right ear using sets of hearing aid transfer functions (HRTF) and/or binaural room impulse responses (BRIR) [5 , 7]. Most proposed extensions to binaural objective quality assessment are based on well-known binaural auditory cues related to human perception of audio data location and perceived sound source width, such as interaural level differences (ILD), interaural time differences (ITD), and interaural cross-correlation ( IACC) between signals entering the left and right ear [1, 5, 8, 9]. In the context of an objective quality assessment, features are extracted based on these spatial references from reference and test signals, and a measure of the distance between them is used as a distortion index. Consideration of these spatial landmarks and the perceived distortions associated with them has made significant progress in the context of developing a spatial audio coding algorithm [7]. However, in the case of using overall spatial audio quality prediction, the interaction of these landmark distortions with each other and with monaural/timbre distortions (especially in cases without waveform preservation) creates a complex scenario [10] with variable results when using features to predict a single quality score given by subjective quality tests such as MUSHRA [11]. Other alternative models have also been proposed [2], in which the output of the binaural model is further processed through a clustering algorithm to determine the number of participating sources in the instantaneous auditory image, and therefore also an abstraction of the classical models of auditory landmark distortion. However, the model in [2] is mainly focused on the movement of sources in space, and its performance is also limited by the accuracy and tracking ability of the corresponding clustering algorithm. The number of features added to enable this model is also significant.

Objective audio quality measurement systems should also use as few mutually independent and most relevant extracted signal features as possible to avoid the risk of overtraining with limited experimental data to match feature distortions with quality scores obtained from listening tests [3].

One of the most notable distortion characteristics noted in listening tests for spatially encoded audio signals at low bit rates has been described as stereo image center collapse and channel crosstalk [12].

Thus, it is desirable to develop a concept that provides improved, efficient and highly accurate audio data analysis, audio coding and audio decoding.

This is achieved through the subject matter of the independent claims of the present application.

Additional embodiments according to the invention are defined by the subject matter of the dependent claims of the present application.

Disclosure of invention

An embodiment according to the present invention relates to an audio data analysis module, such as an audio signal analysis module. The audio data analysis module is configured to obtain a spectral domain representation of two or more input audio signals. Thus, the audio data analysis module, for example, is configured to determine or obtain a representation in the spectral domain. According to an embodiment, the audio data analysis module is configured to obtain a spectral domain representation by decomposing two or more input audio signals into time-frequency bins. In addition, the audio data analysis module is configured to obtain direction information associated with spectral bands of representations in the spectral domain. The direction information is, for example, different directions (or positions) of audio components contained in two or more input audio signals. According to an embodiment, the direction information may be considered as a panning index, which describes, for example, the original location in the sound field created by two or more input audio signals in binaural processing. In addition, the audio data analysis module is configured to obtain loudness information associated with different directions as an analysis result, wherein the contributions to the loudness information are determined depending on the direction information. In other words, the audio data analysis module, for example, is configured to obtain loudness information associated with different pan directions or pan indices, or for a plurality of different estimated directional ranges, as an analysis result. According to an embodiment, various directions, such as pan directions, pan indices, and/or direction ranges, can be derived from the direction information. The loudness information contains, for example, a directional loudness map or level information or energy information. The contributions to the loudness information are, for example, the contributions of the spectral bands of the representations in the spectral domain to the loudness information. According to an embodiment, the contributions to the loudness information are contributions to the values of the loudness information associated with the different directions.

This embodiment is based on the idea that it is advantageous to determine the loudness information depending on the direction information obtained from two or more input audio signals. This makes it possible to obtain information about the loudness of other sources in a stereo sound combination realized by two or more audio signals. Thus, with the audio data analysis module, the perception of two or more audio signals can be analyzed very efficiently by obtaining loudness information associated with different directions as an analysis result. According to an embodiment, the loudness information may comprise or be a directional loudness map that gives, for example, loudness information of a combination of two or more signals in different directions, or loudness information of at least one common temporal signal of two or more input audio signals, averaged over all ERB (Equivalent Rectangular Bandwidth) frequency bands.

According to an embodiment, the audio data analysis module is configured to derive a plurality of weighted representations (e.g., "beam signals") in the spectral domain (e.g., time-frequency domain) based on the spectral-domain (e.g., time-frequency domain) representations. two or more input audio signals. The values of one or more representations in the spectral domain are weighted depending on different directions (e.g., pan direction) (e.g., represented by weighting factors) of audio components (e.g., spectral intervals or spectral bands) (e.g., melodic tones of instruments or a singer) in two or over input audio signals to get many weighted representations in the spectral domain (eg "directional signals"). The audio data analysis module is configured to obtain loudness information (e.g., loudness values for a plurality of different directions; e.g., "directional loudness maps") associated with different directions (e.g., pan directions) based on weighted representations in the spectral domain (e.g., "directional signals") as the result of the analysis.

This means, for example, that the audio data analysis module analyzes in which direction, from different directions of the audio components, the values of one or more representations in the spectral domain affect the loudness information. Each spectral interval is, for example, associated with a direction, in which the loudness information associated with a certain direction can be determined by the audio analysis module based on more than one spectral interval associated with that direction. The weighting may be performed for each bin or each spectral band of one or more representations in the spectral domain. In accordance with an embodiment, the values of the frequency interval or group of frequencies are processed using a window function by weighting in one of the different directions. For example, they are weighted in the direction with which they are associated and/or in adjacent directions. For example, the direction is associated with the direction in which the frequency interval or group of frequencies affects the loudness information. Values deviating from that direction, for example, are weighted less significantly. Thus, a plurality of weighted representations in the spectral domain can provide an indication of the spectral intervals or spectral bands affecting the loudness information in different directions. In accordance with an embodiment, the plurality of weighted representations in the spectral domain may be at least in part contributions to the loudness information.

In accordance with an embodiment, an audio data analysis module is configured to analyze (eg, transform) two or more input audio signals into a short time Fourier transform (STFT) domain (eg, using a Hann window) to obtain two or more transformed audio signals. The two or more converted audio signals may be spectral domain (eg, time-frequency domain) representations of two or more input audio signals.

According to an embodiment, the audio data analysis module is configured to group spectral intervals of two or more transformed audio signals into spectral bands of two or more transformed audio signals (e.g., such that the bandwidth of the groups or spectral bands increases with frequency) (e.g., by frequency selectivity of the anterior part of the human ear labyrinth). In addition, the audio data analysis module is configured to weight spectral bands (e.g., spectral intervals in spectral bands) using different weighting factors based on the outer ear and middle ear model to obtain one or more spectral domain representations of two or more input audio signals. By special grouping of spectral intervals into spectral bands and by weighting the spectral bands, two or more input audio signals are prepared in such a way that the loudness perception of two or more input audio signals by a user hearing said signals can be very accurately and efficiently estimated or determined by the audio data analysis module. in terms of determining loudness information. By this, the converted audio signals, respectively the spectral domain representations of the two or more input audio signals, are adapted to the human ear to enhance the richness of the loudness information obtained by the audio data analysis module.

According to an embodiment, two or more audio input signals are associated with different directions or different speaker positions (eg, L (left), R (right)). Different directions or different speaker positions may represent different channels for a stereo and/or multi-channel audio scene. Two or more input audio signals can be distinguished from each other by indexes, which can, for example, be represented by letters of the alphabet (for example, L (left), R (right), M (middle)) or, for example, a positive integer indicating the channel number from two or more input audio signals. Thus, the indexes may indicate the different directions or positions of the speakers to which two or more audio inputs are associated (eg, they indicate the position in the listening space from which the inputs originate). According to an embodiment, different directions (hereinafter, eg, first different directions) of two or more input audio signals do not refer to different directions (hereinafter, eg, second different directions) with which the loudness information obtained by the audio analysis module is associated. Thus, the direction from the first different directions may represent the signal path of the two or more input audio signals, and the direction from the second different directions may represent the direction of the audio signal component of the two or more input audio signals. Second different directions may be placed between the first directions. Additionally or alternatively, second different directions may be placed outside of the first directions and/or in the first directions.

According to an embodiment, the audio data analysis module is configured to determine direction-dependent weighting (e.g., based on pan directions) for each spectral interval (e.g., and also for each time step/frame) and for a plurality of given directions (desired pan directions) . The preset directions are, for example, equidistant directions that can be associated with preset pan directions/indices. Alternatively, the predetermined directions are, for example, determined using direction information associated with the spectral bands of the spectral domain representations obtained by the audio data analysis module. According to an embodiment, the direction information may comprise predetermined directions. Direction-dependent weighting is, for example, applied by the audio data analysis module to one or more spectral domain representations of two or more input audio signals. With direction-dependent weighting, a spectral spacing value is, for example, associated with one or more directions from a plurality of predetermined directions. This direction-dependent weighting is, for example, based on the idea that each spectral interval of the spectral domain representations of two or more input audio signals contributes to loudness information in one or more different directions from a set of given directions. Each spectral interval, for example, contributes primarily in one direction and only slightly in neighboring directions, whereby it is advantageous to weight the spectral interval value differently for different directions.

According to an embodiment, the audio data analysis module is configured to determine a direction-dependent weighting using a Gaussian function, whereby the direction-dependent weighting decreases as the deviation increases between the corresponding extracted direction values (e.g., associated with the time-frequency interval in question) and the corresponding direction values. The respective extracted direction values may represent the directions of the audio components in the two or more input audio signals. The interval for the respective extracted direction values may be between a full left direction and a full right direction, with the left and right directions being considered relative to a user receiving two or more audio inputs (eg facing speakers). According to an embodiment, the audio data analysis module may determine each extracted direction value as a given direction value, or equidistant direction values as given direction values. Thus, for example, one or more spectral intervals corresponding to the extracted direction are weighted in given directions adjacent to that extracted direction according to a Gaussian function less significantly than in the given direction corresponding to the extracted direction value. The greater the distance of the given direction from the extracted direction, the more the weighting of the spectral intervals or spectral bands is reduced, whereby, for example, the spectral interval has little or no effect on the perception of loudness at a location away from the corresponding extracted direction.

According to an embodiment, the audio data analysis module is configured to determine the pan index value as the extracted direction values. Pan index values, for example, will uniquely indicate the direction of the time-frequency components (ie, spectral intervals) of sources in a stereo downmix created by two or more audio inputs.

According to an embodiment, the audio data analysis module is configured to determine the extracted direction values depending on spectral domain values of the input audio signals (eg, spectral domain representation values of the input audio signals). The extracted direction values are, for example, determined based on an estimate of the amplitude panning of the signal components (eg, in time-frequency domains) between the input audio signals, or based on the relationship between the amplitudes of the respective values in the spectral domain of the input audio signals. In accordance with an embodiment, the extracted direction values determine a measure of similarity between values in the spectral domain of the input audio signals.

, ассоциированного с заданным направлением (например, представленным индексом

), временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенный индексом спектрального интервала k, в соответствии с

, где

- заданное значение (которое контролирует, например, ширину гауссова окна).

обозначает извлеченные значения направлений, ассоциированные со временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k, и

- значение направления, которое обозначает (или ассоциировано) заданное направление (например, имеющее индекс направления j). Зависящее от направления взвешивание основано на идее, что спектральные значения, или спектральные интервалы, или спектральные полосы с извлеченным значением направления (например, индексом панорамирования), равным

(например, равным заданному направлению), пропускаются без модификации зависящим от направления взвешиванием, а спектральные значения, или спектральные интервалы, или спектральные полосы с извлеченным значением направления (например, индексом панорамирования), отклоняющимся от

, взвешиваются. В соответствии с вариантом осуществления спектральные значения, или спектральные интервалы, или спектральные полосы с извлеченным значением направления около

взвешиваются и пропускаются, а остальная часть значений отбрасывается (например, далее не обрабатывается).According to an embodiment, the audio data analysis module is configured to obtain a direction dependent weighting

, associated with a given direction (for example, represented by the index

), a time (or time frame) denoted by time index m, and a spectral interval denoted by spectral interval index k, according to

, Where

is a given value (which controls, for example, the width of the Gaussian window).

denotes the extracted direction values associated with a time (or time frame) denoted by time index m and a spectral interval denoted by spectral interval index k, and

is a direction value that denotes (or is associated with) a given direction (eg, having a direction index j). Direction-dependent weighting is based on the idea that spectral values, or spectral intervals, or spectral bands with an extracted direction value (e.g., pan index) equal to

(e.g., equal to a given direction) are skipped without modification by direction-dependent weighting, and spectral values, or spectral intervals, or spectral bands with an extracted direction value (e.g., pan index) deviating from

, are weighed. According to an embodiment, spectral values, or spectral intervals, or spectral bands with an extracted direction value of about

are weighted and skipped, and the rest of the values are discarded (for example, not further processed).

According to an embodiment, the audio data analysis module is configured to apply direction-dependent weighting to one or more spectral domain representations of two or more input audio signals to obtain weighted spectral domain representations (eg, "directional signals"). Thus, weighted spectral domain representations comprise, for example, spectral intervals (i.e., time-frequency components) of one or more spectral domain representations of two or more input audio signals that correspond to one or more given directions, for example, within the value tolerance (eg, also spectral intervals associated with different predetermined directions bordering the selected predetermined direction). According to an embodiment, for each given direction, a weighted spectral domain representation may be implemented by direction-dependent weighting (e.g., a weighted spectral domain representation may contain direction-dependent weighted spectral values, spectral intervals, or spectral bands associated with a given direction, and /or associated with a direction near a given direction in time). Alternatively, for each spectral domain representation (eg, two or more input audio signals), one weighted spectral domain representation is obtained that represents, for example, the corresponding spectral domain representation weighted for all given directions.

According to an embodiment, an audio data analysis module is configured to obtain weighted representations in the spectral domain such that signal components having an associated first predetermined direction (e.g., a first pan direction) are boosted compared to signal components having other directions associated (which different from the first specified direction, and which are, for example, attenuated according to a Gaussian function) in the first weighted representation in the spectral domain, and also in such a way that the signal components having an associated second specified direction (which is different from the first specified direction) (for example , the second panning direction) are boosted compared to signal components having associated other directions (which are different from the second predetermined direction, and which are, for example, attenuated according to a Gaussian function) in the second weighted spectral domain representation. Thus, for example, for each given direction, a weighted representation in the spectral domain can be determined for each signal from two or more input audio signals.

, ассоциированных с входным аудиосигналом или объединением входных аудиосигналов, обозначенных индексом i, спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k, в соответствии с

, где

обозначает представление в спектральной области, ассоциированное с входным аудиосигналом или объединением входных аудиосигналов, обозначенных индексом i (например, i=L, или i=R, или i=DM; где L=left (левый), R=right (правый) и DM=downmix (микшированный с понижением)), спектральной полосой, обозначенной индексом b, временем (или временным кадром) обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k, и

обозначает зависящее от направления взвешивание (например, функцию взвешивания, такую как гауссова функция), ассоциированное с направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k. Таким образом, взвешенные представления в спектральной области могут быть определены, например, посредством взвешивания представления в спектральной области, ассоциированного с входным аудиосигналом или объединением входных аудиосигналов, с помощью зависящего от направления взвешивания.According to an embodiment, the audio data analysis module is configured to obtain weighted representations in the spectral domain

associated with the input audio signal or the combination of the input audio signals, indicated by the index i, the spectral band, indicated by the index b, the direction, indicated by the index

, a time (or time frame) denoted by time index m, and a spectral interval denoted by spectral interval index k, according to

, Where

denotes the spectral domain representation associated with the input audio signal or combination of the input audio signals, denoted by index i (e.g., i=L or i=R or i=DM; where L=left (left), R=right (right) and DM=downmix), a spectral band indicated by index b, a time (or time frame) indicated by time index m, and a spectral interval indicated by spectral interval index k, and

denotes a direction-dependent weighting (e.g., a weighting function such as a Gaussian function) associated with the direction indicated by the index

, a time (or time frame) denoted by time index m, and a spectral slot denoted by spectral slot index k. Thus, weighted spectral domain representations can be determined, for example, by weighting the spectral domain representation associated with the input audio signal or by combining the input audio signals with direction-dependent weighting.

In accordance with an embodiment, the audio data analysis module is configured to determine an average value over a plurality of band loudness values (e.g., associated with different frequency bands but with the same direction, e.g., associated with a given direction and/or directions near a given direction) to get the value of the combined volume (for example, associated with a certain direction or panning direction, i.e. with a given direction). The combined loudness value may represent loudness information obtained by the audio analysis module as an analysis result. Alternatively, the loudness information obtained by the audio analysis module as a result of the analysis may contain a combined loudness value. Thus, the loudness information may comprise combined loudness values associated with different predetermined directions, from which a directional loudness map can be derived.

According to an embodiment, an audio data analysis module is configured to derive band loudness values for a plurality of spectral bands (e.g., ERB bands) based on a weighted combined spectral domain representation representing a plurality of input audio signals (e.g., a combination of two or more input audio signals) ( wherein, for example, the weighted combined spectral representation may combine the weighted spectral domain representations associated with the input audio signals). Additionally, the audio data analysis module is configured to obtain, as an analysis result, a plurality of combined loudness values (covering a plurality of spectral bands; for example, as a single scalar value) based on the obtained frequency band loudness values for a plurality of different directions (or panning directions). Thus, for example, the audio data analysis module is configured to average over all frequency band loudness values associated with the same direction to obtain a combined loudness value associated with that direction (resulting in, for example, multiple combined loudness values). The audio data analysis module, for example, is configured to obtain a combined loudness value for each given direction.

In accordance with an embodiment, the audio data analysis module is configured to calculate the average of the squares of the spectral values of the weighted combined representation in the spectral domain over the spectral values of the frequency band (or over the spectral intervals of the frequency band) and apply exponentiation with an exponent between 0 and 1/2 ( and preferably less than or equal to 1/3 or ¼) to the average of the squares of the spectral values to determine the loudness values of the frequency bands (associated with the respective frequency band).

, ассоциированных со спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, в соответствии с

. Коэффициент K_b обозначает количество спектральных интервалов в частотной полосе, имеющей индекс частотной полосы b. Переменная k является бегущей переменной и обозначает спектральные интервалы в частотной полосе, имеющей индекс частотной полосы b, где b обозначает спектральную полосу.

обозначает взвешенное объединенное представление в спектральной области, ассоциированное со спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k.According to an embodiment, the audio data analysis module is configured to obtain loudness values of the frequency bands

, associated with the spectral band, denoted by the index b, the direction, denoted by the index

, time (or time frame) denoted by time index m, according to

. The coefficient K _b denotes the number of spectral slots in a frequency band having a frequency band index b. The variable k is a running variable and denotes the spectral intervals in the frequency band having the frequency band index b, where b denotes the spectral band.

denotes the weighted combined spectral domain representation associated with the spectral band denoted by index b, the direction denoted by index

, a time (or time frame) denoted by time index m, and a spectral interval denoted by spectral slot index k.

), ассоциированного с направлением, обозначенным индексом

, и временем (или временным кадром), обозначенным временным индексом m, в соответствии с

, Коэффициент B обозначает общее количество спектральных полос b, и

обозначает значения громкости частотных полос, ассоциированные со спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, и временем (или временным кадром), обозначенным временным индексом m.According to an embodiment, the audio data analysis module is configured to obtain a plurality of combined loudness values L(m,

) associated with the direction indicated by the index

, and the time (or time frame) denoted by time index m, according to

, coefficient B denotes the total number of spectral bands b, and

denotes the loudness values of the frequency bands associated with the spectral band indicated by the index b, the direction indicated by the index

, and time (or time frame) denoted by time index m.

According to an embodiment, the audio data analysis module is configured to distribute loudness contributions to histogram bins associated with different directions (e.g., second different directions as described above; e.g., predetermined directions) depending on direction information to obtain an analysis result. Loudness contributions, for example, are represented by a plurality of combined loudness values or a plurality of frequency band loudness values. Thus, for example, the analysis result contains a directional loudness map defined by histogram intervals. Each interval of the histogram, for example, is associated with one of the given directions.

According to an embodiment, the audio data analysis module is configured to obtain loudness information associated with spectral intervals based on spectral domain representations (eg, to obtain combined loudness for each T/F cell). The audio data analysis module is configured to add a loudness contribution to one or more histogram bins based on loudness information associated with a particular spectral bin. The loudness contribution associated with a particular spectral interval, for example, is added to different histogram intervals with different weightings (eg, depending on the direction corresponding to the histogram interval). The choice of which one or more histogram bins the loudness contribution is made (i.e., added to) is based on determining direction information (i.e., the extracted direction value) for a particular spectral bin. According to an embodiment, each histogram bin may represent a time/direction bin. Thus, the histogram interval, for example, is associated with the volume of the combined two or more audio inputs in some time frame and direction. To determine the direction information for a certain spectral interval, for example, the level information for the respective spectral intervals of representations in the spectral domain of two or more input audio signals is analyzed.

According to an embodiment, the audio data analysis module is configured to add loudness contributions to a plurality of histogram bins based on loudness information associated with a certain spectral bin, such that the largest contribution (e.g., the main contribution) is added to the histogram bin associated with direction. , which corresponds to the direction information associated with a particular spectral bin (i.e., the extracted direction value), and such that reduced contributions (eg, comparatively smaller than the largest contribution or main contribution) are added to one or more histogram bins, associated with additional directions (eg, adjacent to a direction that corresponds to direction information associated with a certain spectral interval). As described above, each histogram interval may represent a time/direction cell. According to an embodiment, a plurality of histogram bins may define a directional loudness map, where the directional loudness map defines, for example, loudness for different time directions for combining two or more audio inputs.

According to an embodiment, the audio data analysis module is configured to obtain direction information based on audio content from two or more input audio signals. The direction information contains, for example, directions of components or sources in the audio content from two or more input audio signals. In other words, the direction information may comprise pan directions or pan indices of sources in a stereo downmix signal from two or more audio inputs.

According to an embodiment, the audio data analysis module is configured to obtain direction information based on amplitude pan analysis of the audio content. Additionally or alternatively, the audio data analysis module is configured to obtain direction information based on an analysis of the phase relationship and/or time delay and/or correlation between the audio content of two or more input audio signals. Additionally or alternatively, the audio data analysis module is configured to obtain directional information based on the identification of extended (eg, decorrelated and/or panned) sources. An analysis of the amplitude panning of audio content may comprise analyzing the level correlation between respective spectral intervals of the spectral domain representations of two or more input audio signals (e.g., corresponding spectral intervals of the same level may be associated with a mid-direction of two loudspeakers each transmitting one of the two input audio signals) . Similarly, phase relationship and/or time delay and/or correlation analysis between audio content can be performed. Thus, for example, the phase relationship and/or time delay and/or correlation between the audio content is analyzed for the respective spectral intervals of the spectral domain representations of the two or more input audio signals. Additionally or alternatively, besides level/time difference comparisons between channels, there is an additional (eg, third) method for estimating direction information. This method consists of comparing the spectral information of the incoming audio data with pre-measured "template spectral responses/filters" of audio perceptual modeling functions (HRFs) in various directions.

For example: in some time-frequency cell, the spectrum envelope of the incoming signal at 35 degrees from the left and right channels can closely match the shape of the linear filters for the left and right ears, measured at an angle of 35 degrees. The optimization algorithm or pattern matching procedure will then assign the direction of arrival of the audio data to be 35°. More information can be found here: https://iem.kug.ac.at/fileadmin/media/iem/projects/2011/baumgartner_robert.pdf (see e.g. chapter 2). This method has the advantage of being able to estimate the incoming direction of vertically raised audio sources (in the sagittal plane) in addition to horizontal sources. This method is based, for example, on comparisons of spectral levels.

According to an embodiment, the audio data analysis module is configured to spread the loudness information in multiple directions (eg, outside the direction indicated by the direction information) according to a spreading rule (eg, a Gaussian spreading rule or a limited, discrete spreading rule). This means, for example, that loudness information corresponding to some spectral interval associated with some direction information may also contribute to neighboring directions (of some spectral interval direction) according to the spreading rule. According to an embodiment, the extension rule may contain or correspond to a direction-dependent weighting, where the direction-dependent weighting in this case, for example, determines differently weighted contributions of the loudness information of a certain spectral interval in multiple directions.

An embodiment according to the present invention relates to an audio similarity estimator that is configured to obtain first loudness information (e.g., a directional loudness map; e.g., one or more combined loudness values) associated with different directions (e.g., panning), based on the first set of two or more input audio signals. The audio similarity estimator is configured to compare the first loudness information with second (e.g., corresponding) loudness information (e.g., reference loudness information, reference directional loudness map, and/or reference combined loudness value) associated with different directions (e.g., pan), and with a set of two or more reference audio signals to obtain similarity information (e.g., a "model output variable" (MOV); e.g., one scalar) describing the similarity between the first set of two or more audio input signals and the set of two or more reference audio signals (or representing, for example, the quality of a first set of two or more input audio signals when compared to a set of two or more reference audio signals).

This embodiment is based on the idea that comparing directional loudness information (eg, first loudness information) of two or more input audio signals with directional loudness information (eg, second loudness information) of two or more reference audio signals is efficient and improves measure accuracy. the quality of the audio data (eg, similarity information). The use of loudness information associated with different directions is particularly advantageous with respect to stereo or multi-channel mixed signals, since different directions can be associated with, for example, directions (i.e. pan directions, pan indices) of sources (i.e. audio components) in the mixed signals. Thus, deterioration in the quality of the processed combination of two or more input audio signals can be effectively measured. Another advantage is that waveform-preserving processing of audio data, such as frequency band extension (BWE), has only minimal or no effect on the similarity information, since the loudness information for a stereo image or a multi-channel image, for example, is defined in region of the short-time Fourier transform (STFT). In addition, similarity information based on loudness information can be easily supplemented with monaural/timbre similarity information to improve perceptual prediction for two or more input audio signals. Thus, only one similarity information is used in addition to the monaural quality descriptors, which can reduce the number of independent and relevant signal features used by the objective audio quality measurement system in relation to known systems using only monaural quality descriptors. Using fewer features for the same performance reduces the risk of overlearning and indicates their higher perceptual significance.

According to an embodiment, the audio similarity estimator is configured to obtain first loudness information (e.g., directional loudness maps), wherein the first loudness information (e.g., a vector containing combined loudness values for a plurality of given directions) contains a plurality of combined loudness values, associated with the first set of two or more input audio signals and associated with the respective predetermined directions, wherein the combined loudness values of the first loudness information describe the loudness of the signal components of the first set of two or more input audio signals associated with the respective predetermined directions (with, for example, each value combined loudness associated with a separate direction). Thus, for example, each value of the combined loudness can be represented by a vector definition, for example, the change in loudness over time for some direction. This means, for example, that one combined loudness value may contain one or more loudness values associated with successive time frames. The given directions may be represented by pan directions/indices of the signal components of the first set of two or more input audio signals. Thus, for example, given directions may be given by amplitude panning techniques used to position directional signals in a stereo or multi-channel mix represented by a first set of two or more input audio signals.

According to an embodiment, the audio similarity estimator is configured to obtain first loudness information (e.g., directional loudness maps), wherein the first loudness information is associated with combinations of a plurality of spectral domain weighted representations (e.g., each audio signal) of a first set of two or more audio inputs associated with respective predetermined directions (eg, each combined loudness value and/or weighted spectral domain representation is associated with a separate predetermined direction). This means, for example, that at least one weighted spectral domain representation is computed for each input audio signal, and that all weighted spectral domain representations associated with the same given direction are then combined. Thus, the first loudness information represents, for example, loudness values associated with multiple spectral intervals associated with the same given direction. At least some of the multiple spectral bins, for example, are weighted differently than other bins of the multiple spectral bins.

According to an embodiment, the audio similarity evaluator is configured to determine a difference between the second loudness information and the first loudness information to obtain the difference loudness information. According to an embodiment, the loudness difference information may represent similarity information, or the similarity information may be determined based on the loudness difference information. The difference loudness information, for example, is regarded as a measure of the distance between the second loudness information and the first loudness information. Thus, the differential loudness information can be thought of as a directional loudness distance (eg, DirLoudDist). With this feature, the quality of two or more input audio signals associated with the first loudness information can be determined very efficiently.

According to an embodiment, the audio similarity estimator is configured to determine a value (eg, a single scalar value) that specifies the amount of difference across multiple directions (and optionally also over time, eg, across multiple frames). The audio similarity estimator, for example, is configured to determine an average value of difference loudness information across all directions (eg, panning directions) and over time as a value that determines the amount of difference. This defines, for example, a single number called the model output variable (MOV), where MOV defines the similarity of the first set of two or more input audio signals relative to the set of two or more reference audio signals.

According to an embodiment, an audio similarity estimator is configured to obtain first loudness information and/or second loudness information (e.g., as directional loudness maps) using an audio data analysis module according to one of the embodiments described herein.

According to an embodiment, an audio similarity estimator is configured to obtain a direction component (e.g., direction information) used to obtain loudness information associated with different directions (e.g., one or more directional loudness maps) using metadata representing information about the position of the speakers associated with the input audio signals. The different directions are not necessarily associated with a direction component. In accordance with an embodiment, the direction component is associated with two or more input audio signals. Thus, the direction component may represent a speaker identifier or a channel identifier allocated to, for example, different speaker directions or positions. Conversely, the different directions with which the loudness information is associated may represent directions or positions of audio components in an audio scene realized by two or more input audio signals. Alternatively, different directions may represent evenly spaced directions or positions within a range of positions (e.g., [-1; 1], where -1 represents cues panned all the way to the left and +1 represents cues panned all the way to the right) in which the an audio scene realized by two or more input audio signals. In accordance with an embodiment, different directions may be associated with the predetermined directions described here. A direction component, for example, is associated with endpoints of a position range.

An embodiment according to the present invention relates to an audio encoder for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The audio encoder is configured to provide one or more encoded (e.g., quantized and then lossless coded) audio signals (e.g., encoded spectral domain representations) based on one or more input audio signals (e.g., a left channel signal and a right channel signal) or one or more signals derived from them (for example, the center signal, or the downmix signal and the side signal, or the difference signal). Additionally, the audio encoder is adapted to adapt coding parameters (e.g., to provide one or more encoded audio signals; e.g., quantization parameters) depending on one or more directional loudness maps that represent loudness information associated with a plurality of different directions (e.g., pan direction). ) one or more signals to be encoded (for example, depending on the contributions of individual directional loudness maps of one or more signals to be quantized to a common directional loudness map, for example, associated with several input audio signals (for example, with each signal from one or more input audio signals))

Audio content containing one audio input may be associated with a monaural audio scene, audio content containing two audio inputs may be associated with a stereo audio scene, and audio content containing three or more audio inputs may be associated with a multi-channel audio scene. In accordance with an embodiment, an audio encoder provides, for each input audio signal, a separate encoded audio signal as an output, or provides one combined output signal containing two or more encoded audio signals from two or more input audio signals.

The directional loudness maps (ie DirLoudMap) on which the adaptation of coding parameters depends may be different for different audio content. Thus, for a monaural audio scene, a directional loudness map, for example, contains non-zero loudness values for only one direction (based on a single audio input), and, for example, contains zero loudness values for other directions. For a stereo audio scene, a directional loudness map represents, for example, loudness information associated with both input audio signals, with different directions, for example, associated with the positions or directions of the audio components of the two audio input signals. In the case of three or more input audio signals, the adaptation of the coding parameters depends on, for example, three or more directional loudness maps, with each directional loudness map corresponding to loudness information associated with two of the three audio input signals (for example, the first DirLoudMap may correspond to the first and second input audio signals; the second DirLoudMap may correspond to the first and third audio input signals; and the third DirLoudMap may correspond to the second and third audio input signals). As described in relation to a stereo audio scene, the different directions for directional loudness maps in the case of a multi-channel audio scene are, for example, associated with the positions or directions of audio components of multiple audio inputs.

Embodiments of this audio encoder are based on the idea that depending on one or more directional loudness adaptations of coding parameters is efficient and improves coding fidelity. The coding parameters are, for example, adapted depending on the difference between a directional loudness map associated with one or more input audio signals and a directional loudness map associated with one or more reference audio signals. In accordance with an embodiment, the overall directional loudness maps of the combination of all input audio signals and the combination of all reference audio signals are compared, or alternatively, the directional loudness maps of individual or paired signals are compared to the overall directional loudness map of all input audio signals (e.g., more than one difference can be determined ). The difference between DirLoudMaps may represent a qualitative measure for encoding. Thus, the coding parameters are, for example, adapted such that the difference is minimized to ensure high-quality audio content coding, or the coding parameters are adapted such that only those audio content signals that match the difference at a certain threshold are coded to reduce the coding complexity. Alternatively, the encoding parameters are, for example, adapted depending on the relationship (eg, contributions) of DirLoudMaps of individual signals or DirLoudMaps of pairs of signals to a common DirLoudMap (eg, DirLoudMap associated with the union of all input audio signals). Like difference, this ratio may indicate the similarity between individual signals or pairs of audio content signals, or between individual signals and a combination of all audio content signals, or pairs of signals and a combination of all audio content signals, resulting in high quality coding and/or reduction in coding complexity.

According to an embodiment, the audio encoder is adapted to adapt the bit distribution between one or more signals and/or parameters to be encoded (or, for example, between two or more signals and/or parameters to be encoded) (for example, between the difference signal and the mixed down signal, or between a left channel signal and a right channel signal, or between two or more signals provided by joint encoding of several signals, or between a signal and parameters provided by joint encoding of several signals) depending on the contributions of individual directional loudness maps of one or more signals and/or parameters to be encoded into a common directional loudness map. Adaptation of the bit allocation, for example, is considered as an adaptation of the coding parameters by the audio encoder. The bit allocation can also be considered as bit rate allocation. The bit allocation is, for example, adapted by controlling the quantization precision of one or more of the audio encoder input audio signals. In accordance with an embodiment, a high contribution may indicate a high value of the respective input audio signal or pair of input audio signals for high quality perception of the audio scene created by the audio content. Thus, for example, an audio encoder can be configured to provide a large number of bits for high contribution signals and provide only few or no bits for low contribution signals. Thus, efficient and high quality coding can be achieved.

In accordance with an embodiment, an audio encoder is configured to inhibit encoding of a particular one of the signals to be encoded (e.g., a difference signal) when the contributions of an individual directional loudness map of the determined one of the signals to be encoded (e.g., a difference signal) to the overall directional loudness map are lower (e.g., , given) threshold value. For example, encoding is prohibited if the average ratio or the ratio in the direction of the maximum relative contribution is below a threshold value. Alternatively or additionally, the contributions of signal pair directional loudness maps (e.g., individual signal pair directional loudness maps (e.g., a combination of two signals can be considered as a signal pair; for example, a combination of signals associated with different channels can be considered as a signal pair, and /or difference signals, and/or downmixed signals)) into a common directional loudness map can be used by the encoder to disable encoding of a specific one of the signals (for example, for three signals to be encoded: as described above, the three directional loudness maps of signal pairs can be analyzed with respect to the overall directional loudness map; thus, the encoder can be configured to determine the pair of signals with the highest contribution to the overall directional loudness map and encode only those two signals, and disable encoding for the remaining signal). The inhibition of signal coding, for example, is regarded as an adaptation of the coding parameters. Thus, it is not necessary to encode signals that are not very relevant to the listener's perception of the audio content, and this results in very efficient coding. According to an embodiment, the threshold value may be set to less than or equal to 5%, 10%, 15%, 20%, or 50% of the volume information of the overall directional loudness map.

According to an embodiment, the audio encoder is configured to adapt the quantization accuracy of one or more signals to be coded (e.g., between the difference signal and the downmix signal) depending on the contributions of individual directional loudness maps of the (corresponding) one or more signals to be coded to the overall map. directional volume. Alternatively, or additionally, similar to the prohibition described above, the contributions of the signal pair directional loudness maps to the overall directional loudness map may be used by the encoder to adapt the quantization accuracy of one or more signals to be encoded. Quantization precision adaptation can be considered as an example for adapting coding parameters by an audio encoder.

According to an embodiment, an audio encoder is configured to quantize a spectral domain representation of one or more input audio signals (e.g., a left channel signal and a right channel signal; for example, one or more input audio signals correspond to a plurality of different channels; thus, the audio encoder receives, for example, a input) or one or more signals derived from them (e.g., a center signal, or a downmix signal and a side signal, or a difference signal) using one or more quantization parameters (e.g., scaling factors or parameters describing how precise the quantization or which quantization step to which spectral intervals or frequency bands of one or more signals to be quantized should be applied) (with quantization parameters describing, for example, the distribution of bits to different signals to be quantized and/or different frequency bands) to obtain one or more quantized representations in the spectral areas. The audio encoder is configured to adjust one or more quantization parameters (e.g., to adapt the bit distribution between one or more signals to be encoded) depending on one or more directional loudness maps that represent loudness information associated with a plurality of different directions (e.g., direction panning) of one or more signals to be quantized in order to adapt the provision of one or more encoded audio signals (e.g., depending on the contributions of individual directional loudness maps of one or more signals to be quantized to a common directional loudness map, for example, associated with several input audio signals ( for example, with each signal from one or more input audio signals)). Additionally, the audio encoder is configured to encode one or more quantized representations in the spectral domain to obtain one or more encoded audio signals.

According to an embodiment, the audio encoder is configured to adjust one or more quantization parameters depending on the contributions of the individual directional loudness maps of the one or more signals to be quantized to the overall directional loudness map.

According to an embodiment, an audio encoder is configured to determine an overall directional loudness map based on input audio signals, where the overall directional loudness map represents loudness information associated with different directions (e.g., audio components; e.g., pan directions) of the audio scene represented by (or to be represented, e.g. after rendering at the decoder side) via input audio signals (perhaps in combination with knowledge or ancillary information regarding speaker positions and/or knowledge or ancillary information describing the positions of audio objects). The overall directional loudness map represents, for example, loudness information associated with all (eg, pooling) audio inputs.

According to an embodiment, one or more signals to be quantized are associated (e.g., in a fixed, signal-independent manner) with different directions (e.g., first different directions) or associated with different speakers (e.g., at different predetermined speaker positions) or associated with different audio objects (eg, with audio objects to be rendered at different positions, eg, according to object rendering information; eg, pan index).

According to an embodiment, the signals to be quantized comprise the components (eg, center signal and stereo center/side coding side signal) of a joint multi-signal coding of two or more input audio signals.

According to an embodiment, the audio encoder is configured to estimate the contribution of the joint multi-signal coding difference signal to the overall directional loudness map and adjust one or more quantization parameters depending thereon. The estimated contribution, for example, is represented by the contribution of the difference signal directional loudness map to the overall directional loudness map.

According to an embodiment, an audio encoder is adapted to adapt the bit distribution between one or more signals and/or parameters to be separately encoded for different spectral intervals or separately encoded for different frequency bands. Additionally or alternatively, the audio encoder is configured to adapt the quantization accuracy of one or more signals to be separately coded for different spectral intervals or separately coded for different frequency bands. By adapting the quantization precision, the audio encoder, for example, is also adapted to adapt the bit allocation. Thus, the audio encoder is, for example, adapted to adapt the bit distribution between one or more audio signals of the audio content to be encoded by the audio encoder. Additionally or alternatively, the bit distribution between the parameters to be encoded is adapted. The bit allocation adaptation may be performed by the audio encoder separately for different spectral intervals or separately for different frequency bands. According to an embodiment, it is also possible that the bit allocation between signals and parameters is adapted. In other words, each signal of the one or more signals to be encoded by the audio encoder may contain a separate bit allocation for different spectral intervals and/or different frequency bands (e.g., the corresponding signal), and this separate bit allocation for each one or more signals to be encoded may be adapted by audio encoder.

In accordance with an embodiment, an audio encoder is adapted to adapt the bit distribution between one or more signals and/or parameters to be encoded (for example, separately for each spectral interval or for each frequency band) depending on the spatial masking estimate between the two or more to be encoded. signals. In addition, the audio encoder is configured to estimate spatial concealment based on directional loudness maps associated with two or more signals to be encoded. This is based, for example, on the idea that directional loudness maps are spread out in space and/or time. Thus, for example, only few or no bits are wasted for masked signals, and more bits (e.g., more than for masked signals) are spent to encode relevant signals or signal components (e.g., signals or signal components that are not masked). other signals or components). According to an embodiment, the spatial concealment depends, for example, on the level associated with the spectral intervals and/or frequency bands of the two or more signals to be encoded, at the spatial distance between the spectral intervals and/or frequency bands, and/or at the time distance between the spectral intervals and/or frequency bands. intervals and/or frequency bands). Directional loudness maps can directly provide loudness information for individual spectral intervals and/or frequency bands for individual signals or combinations of signals (eg, pairs of signals), resulting in an efficient encoder spatial masking analysis.

According to an embodiment, the audio encoder is configured to evaluate the effect of masking the loudness contribution associated with the first direction of the first signal to be coded on the loudness contribution associated with the second direction (which is different from the first direction) of the second signal to be coded (in which, for example, the effect masking decreases as the angle difference increases). The masking effect determines, for example, the appropriateness of the spatial masking. This means, for example, that more bits are wasted for loudness contributions associated with a masking effect that is below a threshold than for signals (eg, spatially masked signals) associated with a masking effect that is above the threshold. According to an embodiment, the threshold value may be defined as 20%, 50%, 60%, 70%, or 75% masking from full masking. This means, for example, that the masking effect of neighboring spectral intervals or frequency bands is estimated depending on the loudness information of the directional loudness maps.

In accordance with an embodiment, an audio encoder comprises an audio parsing module, in accordance with one of the embodiments described herein, wherein loudness information (eg, a "directional loudness map") associated with different directions forms a directional loudness map.

According to an embodiment, the audio encoder is configured to adapt encoder-introduced noise (eg, quantization noise) depending on one or more directional loudness maps. Thus, for example, one or more directional loudness maps of one or more signals to be encoded can be compared by an encoder with one or more directional loudness maps of one or more reference signals. Based on this comparison, the audio encoder is, for example, configured to estimate a difference indicative of the introduced noise. The noise may be adapted by quantization adaptation performed by the audio encoder.

According to an embodiment, an audio encoder is configured to exploit the variance between a directional loudness map that is associated with a particular unencoded audio input (or a particular unencoded audio input pair) and a directional loudness map achievable by an encoded version of a particular audio input (or a particular input). pairs of audio signals), as a criterion (for example, a target criterion) for adapting the generation of a certain encoded audio signal (or a certain encoded pair of audio signals). The following examples are only described for one specific non-encoded audio input signal, but it is clear that they are also applicable to a specific non-encoded audio input pair. A directional loudness map associated with a particular uncoded audio input signal may be associated with a directional loudness reference map, or may represent a directional loudness reference map. Thus, the deviation between the directional loudness reference map and the directional loudness map of an encoded version of a particular input audio signal may be indicative of encoder-introduced noise. To reduce noise, the audio encoder may be configured to adapt coding parameters to reduce variance to provide a high quality encoded audio signal. This is, for example, implemented using a feedback loop that controls the deviation each time. Thus, the coding parameters are adapted until the deviation falls below a predetermined threshold. According to an embodiment, the threshold may be defined as 5%, 10%, 15%, 20%, or 25% deviation. Alternatively, the adaptation by the encoder is performed using a neural network (eg, looping preemptively). Using a neural network, a directional loudness map for an encoded version of a particular input audio signal can be estimated without being directly determined by an audio encoder or audio data analysis module. Thus, very fast high fidelity audio coding can be realized.

According to an embodiment, the audio encoder is configured to activate and deactivate a joint coding tool (which, for example, jointly encodes two or more of or derived audio signals) (for example, to decide on/off M/S (center/ side signal)) depending on one or more directional loudness maps that represent loudness information associated with a plurality of different directions of one or more signals to be encoded. To enable or disable the joint coding tool, the audio encoder can be configured to determine the contribution of the directional loudness map of each signal, or each pair of possible signals, to the overall directional loudness map of the overall scene. According to an embodiment, a contribution above a threshold (e.g., a contribution of at least 10%, or at least 20%, or at least 30%, or at least 50%, indicates whether it is reasonable to apply joint coding of the input audio signals.For example, the threshold value may be relatively low for this use case (for example, lower than in other use cases) in order to primarily filter out inappropriate pairs.Based on directional loudness maps, an audio encoder can check whether the joint encoding of the signals will lead to a more efficient encoding and/or high resolution encoding.

According to an embodiment, an audio encoder is configured to determine one or more parameters of a joint coding tool (which, for example, jointly encodes two or more of or derived from the input audio signals) depending on one or more directional loudness maps that represent information about loudness associated with a plurality of different directions of the one or more signals to be coded (eg, to control the smoothing of frequency-dependent prediction coefficients; eg, to set parameters of the "intense stereo" joint coding tool). One or more directional loudness information maps contain, for example, loudness information in given directions and time frames. Thus, for example, the audio encoder is configured to determine one or more parameters for the current time frame based on the loudness information of previous time frames. Based on directional loudness maps, masking effects can be analyzed very efficiently and can be indicated by one or more parameters, whereby frequency-dependent prediction coefficients can be determined based on one or more parameters such that the predicted sample values are close to the original sample values. (associated with the signal to be encoded). Thus, it is possible for the encoder to determine frequency dependent prediction coefficients representing an approximation of the masking threshold rather than the signal to be coded. In addition, directional loudness maps, for example, are based on a psychoacoustic model, whereby the determination of frequency-dependent prediction coefficients based on one or more parameters is further improved and can result in very accurate prediction. Alternatively, the parameters of the joint coding tool determine, for example, which signal or pair of signals should be jointly encoded by the audio encoder. The audio encoder is, for example, configured to justify determining one or more parameters on the contributions of each directional loudness map associated with the signal or signal pair to be encoded to the overall directional loudness map. Thus, for example, one or more parameters indicate the individual signals and/or pairs of signals with the highest contribution or a contribution equal to or above a threshold value (see, for example, the definition of a threshold value above). Based on one or more parameters, the audio encoder is, for example, configured to jointly encode the signals indicated by the one or more parameters. Alternatively, for example, pairs of signals having high proximity/similarity in the respective directional loudness map may be indicated by one or more joint coding tool parameters. Selected pairs of signals are, for example, jointly presented by downmixing. In this way, the number of bits required for encoding is minimized or reduced because the downmix signal or the difference signal of the signals to be jointly encoded is very small.

In accordance with an embodiment, an audio encoder is configured to determine or evaluate the effect of a variation of one or more control parameters that control the provision of one or more encoded audio signals on a directional loudness map of one or more encoded signals and adjust one or more control parameters depending on the determination or impact assessments. The influence of the control parameters on the directional loudness map of one or more of the encoded signals may include a measure for induced noise (e.g., the control parameters relative to the quantization position can be adjusted) by audio encoder coding, a measure for audio data distortion, and/or a measure for a drop in listener quality of experience. According to an embodiment, the control parameters may be represented by encoding parameters, or the encoding parameters may comprise control parameters.

According to an embodiment, an audio encoder is configured to derive a direction component (e.g., direction information) used to derive one or more directional loudness maps using metadata representing speaker position information associated with audio input signals (this concept can also be used in other audio encoders). The direction component, for example, is represented herein by the first various directions described herein, which are, for example, associated with different channels or speakers associated with audio input signals. According to an embodiment based on the direction component, the resulting one or more directional loudness maps can be associated with an input audio signal and/or a pair of signals from the input audio signals with the same direction component. Thus, for example, a directional volume map may have an index L, and an input audio signal may have an index L, where L indicates the left channel or left speaker signal. Alternatively, the direction component may be represented by a vector such as (1, 3) that indicates the combination of the first channel and third channel audio input signals. Thus, a directional loudness map with index (1, 3) can be associated with a pair of signals. According to an embodiment, each channel may be associated with a separate speaker.

An embodiment according to the present invention relates to an audio encoder for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The audio encoder is configured to provide one or more encoded (e.g., quantized and then lossless coded) audio signals (e.g., coded representations in the spectral domain) based on two or more input audio signals (e.g., a left channel signal and a right channel signal) or based on two or more signals derived therefrom using joint encoding of the two or more signals to be jointly encoded (eg, using a center signal or downmix signal and a side signal or difference signal). Additionally, the audio encoder is configured to select signals to be jointly encoded from a plurality of possible signals or from a plurality of pairs of possible signals (for example, from two or more input audio signals or from two or more signals derived from them) depending on directional loudness maps that represent information about loudness associated with a plurality of different directions (e.g., pan direction) of possible signals or pairs of possible signals (e.g., depending on the contributions of individual candidate loudness directional maps to an overall directional loudness map, e.g., associated with multiple audio input signals (e.g., with each signal from one or more input audio signals), or depending on the contributions of the directional loudness maps of pairs of possible signals to a common directional loudness map (eg, associated with all input audio signals)).

According to an embodiment, the audio encoder may be configured to enable and disable joint coding. Thus, for example, if the audio content contains only one audio input, then joint coding is disabled and it is enabled only if the audio content contains two or more audio inputs. Thus, with the audio encoder, it is possible to encode monaural audio content, stereo audio content, and/or audio content containing three or more input audio signals (ie, multi-channel audio content). According to an embodiment, an audio encoder provides each input audio signal with a separate encoded audio signal as an output signal (e.g., suitable for audio content containing only one single input audio signal), or provides one combined output signal (e.g., jointly coded signals) containing two or more encoded audio signals from two or more input audio signals.

Embodiments of this audio encoder are based on the idea that the rationale for joint coding on directional loudness maps is efficient and improves coding fidelity. The use of directional loudness maps is advantageous because they can indicate the listener's perception of the audio content and thereby improve the audio quality of the encoded audio content, especially in a co-coding context. For example, it is possible to optimize the choice of signal pairs to be jointly encoded by analyzing directional loudness maps. Analysis of directional loudness maps provides, for example, information about signals or pairs of signals that can be neglected (for example, signals that have only a small effect on the listener's perception), resulting in a small number of bits needed for an audio encoder-encoded audio content (for example, containing two or more coded signals). This means, for example, that signals with a low contribution of their respective directional loudness map to the overall directional loudness map can be neglected. Alternatively, the analysis may indicate signals that have high similarity (eg, signals with similar directional loudness maps), whereby, for example, through joint coding, optimized difference signals can be obtained.

In accordance with an embodiment, the audio encoder is configured to select signals to be jointly encoded from a plurality of possible signals or from a plurality of pairs of possible signals depending on the contributions of the individual directional loudness maps of the possible signals to the overall directional loudness map or depending on the contributions of the directional loudness maps of pairs of possible signals to a common directional loudness map (eg, associated with multiple audio inputs (eg, each signal from one or more audio inputs)) (or associated with a common (audio)scene, eg, represented by the audio inputs). The overall directional loudness map represents, for example, loudness information associated with different directions (for example, audio components) of an audio scene represented (or to be represented, for example, after rendering at the decoder side) by input audio signals (perhaps in combination with knowledge or ancillary information regarding the positions of loudspeakers and/or knowledge or ancillary information describing the positions of sound objects).

According to an embodiment, the audio encoder is configured to determine the contribution of pairs of possible signals to the overall directional loudness map. Additionally, the audio encoder is configured to select one or more pairs of possible signals having the highest contribution to the overall directional loudness map for joint coding, or the audio encoder is configured to select one or more pairs of possible signals whose contribution to the overall directional loudness map is greater than a given threshold. values (eg, the contribution is at least 60%, 70%, 80%, or 90%), for joint coding. With regard to the highest contribution, it is possible that only one pair of candidate signals has the highest contribution, but it is also possible that more than one pair of candidate signals have the same contribution, which is the highest contribution, or more than one pair of candidate signals have similar contributions with small deviations from the highest contribution. Thus, an audio encoder, for example, is configured to select more than one signal or pair of signals for joint coding. With the features described in this embodiment, it is possible to find relevant signal pairs for improved joint coding and discard signals or signal pairs that do not greatly affect the listener's perception of the encoded audio content.

In accordance with an embodiment, the audio encoder is configured to determine separate directional loudness maps of two or more possible signals (eg, directional loudness maps associated with pairs of signals). Additionally, the audio encoder is configured to compare individual directional loudness maps of two or more possible signals and select two or more of the possible signals for co-coding depending on the result of the comparison (e.g., such that the possible signals (e.g., signal pairs, signal triples, quads signals, etc.) whose individual loudness maps contain the maximum similarity or similarity that is above the similarity threshold is chosen for joint coding). Thus, for example, only few bits are wasted or no bits are wasted for the difference signal (eg, side channel relative to the center channel) to maintain a high quality of the encoded audio content.

According to an embodiment, an audio encoder is configured to determine an overall directional loudness map using downmixing of the input audio signals and/or using binauralization of the input audio signals. Downmixing or binauralization takes into account, for example, directions (eg, channel or speaker associations for the respective audio inputs). The overall directional loudness map may be associated with loudness information corresponding to the audio scene created by all audio inputs.

An embodiment according to the present invention relates to an audio encoder for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The audio encoder is configured to provide one or more encoded (e.g., quantized and then lossless encoded) audio signals (e.g., coded representations in the spectral domain) based on two or more input audio signals (e.g., a left channel signal and a right channel signal) or based on two or more signals received from them. Additionally, the audio encoder is configured to determine a general directional loudness map (e.g., a target scene directional loudness map) based on input audio signals and/or determine one or more separate directional loudness maps associated with individual input audio signals (or associated with two or more input audio signals, such as signal pairs). In addition, the audio encoder is configured to encode the overall directional loudness map and/or one or more individual directional loudness maps as side information.

Thus, for example, if the audio content contains only one input audio signal, the audio encoder is configured to encode only that signal, along with a corresponding separate directional loudness map. If the audio content contains two or more input audio signals, the audio encoder is, for example, configured to encode all or at least some (e.g., one single signal and one signal pair of three audio input signals) signals separately, along with a corresponding directional loudness map (e.g., with separate directional loudness maps of individual encoded signals and/or with directional loudness maps corresponding to pairs of signals or other combinations of more than two signals, and/or with common directional loudness maps associated with all input audio signals). According to an embodiment, an audio encoder is configured to encode all or at least some of the signals, resulting in one encoded audio signal, for example, together with a common directional loudness map as output (for example, one combined output signal (for example, jointly encoded signals) containing, for example, two or more encoded audio signals from two or more input audio signals). Thus, with the audio encoder, it is possible to encode monaural audio content, stereo audio content, and/or audio content containing three or more input audio signals (ie, multi-channel audio content).

Embodiments of this audio encoder are based on the idea that there is an advantage in defining and encoding one or more directional loudness maps since they can indicate the listener's perception of the audio content and thereby improve the audio quality of the encoded audio content. According to an embodiment, one or more directional loudness maps may be used by an encoder to improve coding, for example by adapting coding parameters based on the one or more directional loudness maps. Thus, encoding one or more directional loudness maps is of particular advantage because they can provide information regarding the impact of the encoding. By using one or more directional loudness maps as ancillary information in the encoded audio content provided by the audio encoder, very accurate decoding can be achieved since the encoding information is provided by the audio decoder (eg, in the data stream).

According to an embodiment, an audio encoder is configured to determine an overall directional loudness map based on input audio signals, wherein the overall directional loudness map represents loudness information associated with different directions (e.g., audio components) of an audio scene presented (or to be presented, for example, after rendering at the decoder side) via input audio signals (possibly combined with knowledge or ancillary information regarding speaker positions and/or knowledge or ancillary information describing the positions of audio objects). The different audio scene directions are, for example, the second different directions described herein.

According to an embodiment, the audio encoder is configured to encode the overall directional loudness map as multiple (eg, scalar) values associated with different directions (and preferably multiple frequency intervals or frequency bands). If the overall directional loudness map is encoded as a set of values, the value associated with a certain direction may contain information about the loudness of a set of frequency intervals or frequency bands. Alternatively, the audio encoder is configured to encode the overall directional loudness map using a center position value (e.g., describing the pan angle or index at which the maximum of the overall directional loudness map for a particular frequency interval or frequency band is observed) and gradient information (e.g., one or more scalars describing the gradients of the values of the overall directional loudness map in the angular direction or pan index direction). The encoding of the overall directional loudness map using the center position value and gradient information can be performed for various defined frequency intervals or frequency bands. Thus, for example, an overall directional loudness map may contain center position value information and gradient information for more than one frequency interval or frequency band. Alternatively, the audio encoder is configured to encode the overall directional loudness map as a polynomial representation, or the audio encoder is configured to encode the overall directional loudness map as a spline representation. Encoding the overall directional loudness map as a polynomial representation or a spline representation is an economical encoding. While these features are described in relation to a general directional loudness map, this encoding may also be performed on individual directional loudness maps (eg, individual signals, pairs of signals, and/or groups of three or more signals). Thus, with these features, the directional loudness maps are encoded very efficiently and the information on which the encoding is based is provided.

In accordance with an embodiment, an audio encoder is configured to encode (and, for example, transmit or insert into an encoded audio representation) a single (e.g., only one) downmix signal derived from a plurality of input audio signals and a common directional loudness map. Alternatively, the audio encoder is configured to encode (and also, for example, transmit or insert into the encoded audio representation) a plurality of signals (for example, input audio signals or signals derived from them) and encode (and also, for example, transmit or insert into the encoded audio representation ) individual directional loudness maps of the plurality of signals that are encoded (eg, directional loudness maps of individual signals and/or pairs of signals and/or groups of three or more signals). Alternatively, the audio encoder is configured to encode (and, for example, transmit or insert into the encoded audio representation) a general directional loudness map, a plurality of signals (for example, input audio signals or signals derived from them), and parameters describing (for example, corresponding) contributions signals that are encoded in a common directional loudness map. According to an embodiment, the parameters describing the contributions may be represented by scalars. Thus, it is possible for an audio decoder receiving an encoded audio representation (e.g., audio content or a data stream containing encoded signals, an overall directional loudness map, and parameters) to recreate individual directional loudness maps of the signals based on the overall directional loudness map and parameters describing the contributions of the signals.

An embodiment according to the present invention relates to an audio decoder for decoding encoded audio content. The audio decoder is configured to receive an encoded representation of the one or more audio signals and provide a decoded representation of the one or more audio signals (eg, using AAC-like decoding or using decoding of entropy-encoded spectral values). Further, the audio decoder is configured to receive encoded directional loudness map information and decode the encoded directional loudness map information to obtain one or more (eg, decoded) directional loudness maps. Additionally, the audio decoder is configured to recreate the audio scene using a decoded representation of one or more audio signals and using one or more directional loudness maps. The audio content may contain an encoded representation of one or more audio signals and encoded information about directional loudness maps. The encoded directional loudness map information may comprise directional loudness maps of individual signals, pairs of signals, and/or groups of three or more signals.

An embodiment of this audio decoder is based on the idea that determining and decoding one or more directional loudness maps is advantageous because they can indicate the listener's perception of the audio content and thereby improve the audio quality of the decoded audio content. The audio decoder, for example, is configured to determine a high quality prediction signal based on one or more directional loudness maps, whereby differential decoding (or joint decoding) can be improved. According to an embodiment, directional loudness maps determine loudness information for different directions in an audio scene over time. Loudness information for a certain direction at a certain point in time or in a certain time frame may contain information about the loudness of different audio signals or one audio signal, for example, in different frequency intervals or frequency bands. Thus, for example, the provision of a decoded representation of one or more audio signals by an audio decoder can be improved, for example, by adapting the decoding of the encoded representation of one or more audio signals based on the decoded directional loudness maps. Thus, the reconstructed audio scene is optimized because the decoded representation of the one or more audio signals can achieve a minimum deviation from the original audio signal based on the analysis of the one or more directional loudness maps, resulting in a high quality audio scene. In accordance with an embodiment, an audio decoder may be configured to use one or more directional loudness maps to adapt decoding parameters to provide an efficient, high fidelity decoded representation of one or more audio signals.

In accordance with an embodiment, an audio decoder is configured to obtain output signals such that one or more directional loudness maps associated with the output signals are approximately equal to or equal to one or more target directional loudness maps. One or more target directional loudness maps are based on one or more decoded directional loudness maps or equal to one or more decoded directional loudness maps. The audio decoder, for example, is configured to use suitable scaling or combining one or more decoded audio signals to produce output signals. Target directional loudness maps, for example, are considered directional loudness reference maps. According to an embodiment, directional loudness target maps may represent loudness information of one or more audio signals before encoding and decoding the audio signals. Alternatively, the target directional loudness maps may represent loudness information associated with an encoded representation of one or more audio signals (eg, one or more decoded directional loudness maps). The audio decoder receives, for example, encoding parameters used for encoding to provide encoded audio content. The audio decoder is, for example, configured to determine decoding parameters based on the encoding parameters for scaling one or more decoded directional loudness maps to determine one or more target directional loudness maps. It is also possible that the audio decoder comprises an audio data analysis module that is configured to determine target directional loudness maps based on the decoded directional loudness maps and one or more decoded audio signals, wherein, for example, the decoded directional loudness maps are scaled based on the one or more decoded audio signals. Since one or more target directional loudness maps may be associated with an optimal or optimized audio scene implemented by audio signals, it is useful to minimize the variance between one or more directional loudness maps associated with output signals and one or more target directional loudness maps. According to an embodiment, this deviation can be minimized by the audio decoder by adapting the decoding parameters or adapting the parameters relative to the audio scene reconstruction. Thus, this feature controls the quality of the output signals, for example by means of a feedback loop analyzing one or more directional loudness maps associated with the output signals. The audio decoder, for example, is configured to determine one or more directional loudness maps of output signals (eg, the audio decoder includes an audio data analysis module described herein for determining directional loudness maps). Thus, the audio decoder provides output signals that are associated with directional loudness maps that are approximately equal to or equal to the target directional loudness maps.

In accordance with an embodiment, an audio decoder is configured to receive one (eg, only one) encoded downmix signal (eg, derived from multiple audio inputs) and a common directional loudness map; or a plurality of encoded audio signals (eg, encoder input audio signals or signals derived therefrom) and individual directional loudness maps of the plurality of encoded signals; or an overall directional loudness map, a plurality of encoded audio signals (eg, input audio signals received by the audio encoder or signals derived from them), and parameters describing (eg, appropriate) contributions of the encoded audio signals to the overall directional loudness map. The audio decoder is configured to provide output signals based on this.

An embodiment according to the present invention relates to a format converter for converting an audio content format that represents an audio scene (eg, a spatial audio scene) from a first format to a second format. The first format may contain, for example, a first number of channels or input audio signals and side information or spatial side information adapted to the first number of channels or input audio signals, and the second format may contain, for example, a second number of channels or audio output signals, which may differ from the first number of channels or input audio signals, and side information or spatial side information adapted to the second number of channels or output audio signals. In addition, the format converter is configured to provide a representation of the audio content in the second format based on the representation of the audio content in the first format. Additionally, the format converter is configured to adjust the complexity of the format conversion (for example, by passing one or more first format input audio signals whose contribution to the directional loudness map is below a threshold during the format conversion process) depending on the contributions of the first format input audio signals (for example, one or more audio signals, one or more downmix signals, one or more difference signals, etc.) into an overall directional loudness map of the audio scene (wherein the overall directional loudness map, for example, can be described by the first format side information received by the format converter) . Thus, for example, the contributions of the individual directional loudness maps associated with individual audio input signals to the overall directional loudness map of the audio scene are analyzed to adjust the complexity of the format conversion. Alternatively, this adjustment can be made by the format converter depending on the contributions of the directional loudness maps corresponding to combinations of input audio signals (e.g., signal pairs, center signal, side signal, downmix signal, difference signal, difference signal, and/or groups of three or more signals) into the overall directional loudness map of the audio scene.

Format converter embodiments are based on the idea that format conversion of audio content based on one or more directional loudness maps is advantageous because they can indicate the perception of the audio content by the listener, and thereby realizing high quality audio content in the second format, and reducing format conversion complexity depending on directional volume maps. With the help of contributions, it is possible to obtain information about the signals relevant for the high-quality perception of the audio data of the format-converted audio content. Thus, the audio content in the second format, for example, contains fewer signals (eg, only relevant signals according to directional loudness maps) than the audio content in the first format, with almost the same quality of the audio data.

In accordance with an embodiment, the format converter is configured to receive information about directional loudness maps and, based on this, obtain a general directional loudness map (e.g., a decoded audio scene; e.g., audio content in the first format) and/or one or more directional loudness maps. Information about directional loudness maps (i.e., one or more directional loudness maps associated with individual audio content signals, or associated with signal pairs or a combination of three or more audio content signals) may represent audio content in the first format, may be part of the audio content in the first format , or may be determined by a format converter based on the audio content in the first format (eg, the audio parsing module described herein; for example, the format converter comprises an audio parsing module). According to an embodiment, the format converter is also configured to determine information about directional loudness maps of the audio content in the second format. Thus, for example, directional loudness maps before and after format conversion can be compared to reduce perceived degradation in quality due to format conversion. This is, for example, implemented by minimizing the deviation between the directional loudness map before and after format conversion.

According to an embodiment, the format converter is configured to derive an overall directional loudness map (eg, decoded audio scene) from one or more (eg, decoded) directional loudness maps (eg, associated with signals in the first format).

In accordance with an embodiment, the format converter is configured to calculate or estimate the contribution of a certain input audio signal (eg, a signal in the first format) to the overall directional loudness map of the audio scene. The format converter is configured to decide whether a certain input audio signal should be considered in format conversion depending on the calculation or evaluation of the contribution (for example, by comparing the calculated or estimated contribution with a given absolute or relative threshold value). If the contribution is, for example, equal to or greater than an absolute or relative threshold value, the corresponding signal may be considered relevant, and thus the format converter may be configured to decide to consider this signal. This can be thought of as adjusting the complexity with a format converter, since not all signals in the first format are necessarily converted to the second format. The predetermined threshold may represent a contribution of at least 2%, or at least 5%, or at least 10%, or at least 20%, or at least 30%. For example, this means excluding inaudible and/or irrelevant channels (or almost inaudible and/or irrelevant channels), i.e. the threshold should be lower (eg when compared to other use cases), eg 5%, 10%, 20%, 30%.

An embodiment according to the present invention relates to an audio decoder for decoding encoded audio content. The audio decoder is configured to receive an encoded representation of the one or more audio signals and provide a decoded representation of the one or more audio signals (eg, using AAC-like decoding or using decoding of entropy-encoded spectral values). In addition, the audio decoder is configured to reconstruct an audio scene using a decoded representation of one or more audio signals and adjust the decoding complexity depending on the contributions of the encoded signals (e.g., one or more audio signals, one or more downmix signals, one or more difference signals, etc.). .d.) into the overall directional loudness map of the decoded audio scene.

Embodiments of this audio decoder are based on the idea that adjusting the decoding complexity based on one or more directional loudness maps is advantageous because they can indicate the listener's perception of the audio content and thereby simultaneously realize decoding complexity reduction and audio quality improvement of the audio content decoder. Thus, for example, the audio decoder is configured to decide, based on contributions, which encoded audio content signals are to be decoded and used to recreate the audio scene by the audio decoder. This means, for example, that the encoded representation of one or more audio signals contains fewer audio signals (eg, only relevant audio signals according to directional loudness maps) than the decoded representation of one or more audio signals, with almost the same quality of the audio data.

According to an embodiment, an audio decoder is configured to receive encoded directional loudness map information and decode the encoded directional loudness map information, obtain an overall directional loudness map (e.g., a decoded audio scene or, e.g., a target directional loudness map of a decoded audio scene), and/or one or more (decoded) directional loudness maps. According to an embodiment, the format converter is configured to determine or receive directional loudness map information of encoded audio content (eg, received) and decoded audio content (eg, determined). Thus, for example, directional loudness maps before and after decoding can be compared to reduce perceived quality degradation due to decoding and/or previous encoding (eg, performed by the audio encoder described herein). This is, for example, implemented by minimizing the deviation between the directional loudness map before and after format conversion.

According to an embodiment, an audio decoder is configured to derive an overall directional loudness map (eg, a decoded audio scene or, eg, as a target directional loudness map of a decoded audio scene) from one or more (eg, decoded) directional loudness maps.

In accordance with an embodiment, the audio decoder is configured to calculate or estimate the contribution of a particular encoded signal to the overall directional loudness map of the decoded audio scene. Alternatively, the audio decoder is configured to calculate the contribution of a particular encoded signal to the overall directional loudness map of the encoded audio scene. The audio decoder is configured to decide whether to decode a certain encoded signal, depending on the calculation or assessment of the contribution (for example, by comparing the calculated or estimated contribution with a given absolute or relative threshold value). The predetermined threshold may represent a contribution of at least 60%, 70%, 80%, or 90%. To maintain good quality, thresholds should be lower for cases where processing power is very limited (e.g. for a mobile device), for this range eg 10%, 20%, 40%, 60% may be appropriate. In other words, in some preferred embodiments, the specified threshold should represent a contribution of at least 5%, or at least 10%, or at least 20%, or at least 40%, or at least 60%.

An embodiment according to the present invention relates to a renderer (eg, a binaural renderer, or a soundbar renderer, or a loudspeaker renderer) for converting audio content for listening. In accordance with an embodiment, a renderer for distributing audio content represented using a first number of input audio channels and ancillary information describing desired spatial characteristics, such as placement of audio objects or relationships between audio channels, into a representation containing a certain number of channels that, regardless of the first number input audio channels (eg, greater than the first number of input audio channels or less than the first number of input audio channels). The rendering module is configured to recreate an audio scene based on one or more input audio signals (or, for example, based on two or more input audio signals). In addition, the renderer is configured to adjust the rendering complexity (e.g., by skipping one or more input audio signals whose contribution to the directional loudness map is below a threshold during the rendering process) depending on the contributions of the input audio signals (e.g., one or more audio signals, one or more downmix signals, one or more difference signals, etc.) into an overall directional loudness map of the listenable audio scene. The overall directional loudness map, for example, can be described by ancillary information received by the renderer.

In accordance with an embodiment, the renderer is configured to obtain (e.g., receive or self-determine) information about directional loudness maps and, based on this, obtain a general directional loudness map (e.g., a decoded audio scene) and/or one or more directional loudness maps.

According to an embodiment, the renderer is configured to derive an overall directional loudness map (eg, decoded audio scene) from one or more (or two or more) (eg, decoded or self-derived) directional loudness maps.

In accordance with an embodiment, the renderer is configured to calculate or estimate the contribution of a particular input audio signal to the overall directional loudness map of the audio scene. In addition, the renderer is configured to decide whether a certain input audio signal should be considered when rendering, depending on the calculation or assessment of the contribution (for example, by comparing the calculated or estimated contribution with a given absolute or relative threshold value)

An embodiment according to the present invention relates to a method for analyzing an audio signal. The method comprises deriving a plurality of spectral domain (eg, time-frequency domain) weighted representations (eg, "directional signals") based on one or more spectral domain (eg, time-frequency domain) representations of two or more input audio signals. The values of one or more representations in the spectral domain are weighted depending on the different directions (e.g., pan directions) (e.g., represented by weighting factors) of the audio components (e.g., spectral intervals or spectral bands) (e.g., melodic tones of instruments or a singer) in two or over input audio signals to get many weighted representations in the spectral domain (eg "directional signals"). Additionally, the method comprises obtaining loudness information (e.g., one or more "directional loudness maps") associated with different directions (e.g., pan directions) based on a plurality of weighted representations in the spectral domain (e.g., "directional signals") as a result analysis.

An embodiment according to the present invention relates to a method for evaluating the similarity of audio signals. The method comprises obtaining first loudness information (eg, a map of directional loudness; eg, combined loudness values) associated with different directions (eg, panning) based on a first set of two or more input audio signals. The method further comprises comparing the first loudness information with second (e.g., corresponding) loudness information (e.g., reference loudness information; of two or more reference audio signals to obtain similarity information (e.g., a "model output variable" (MOV)) describing the similarity between a first set of two or more input audio signals and a set of two or more reference audio signals (or representing, for example, quality of the first set of two or more input audio signals when compared to a set of two or more reference audio signals).

An embodiment according to the present invention relates to a method for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The method comprises providing one or more encoded (e.g., quantized and then lossless encoded) audio signals (e.g., encoded spectral domain representations) based on one or more input audio signals (e.g., a left channel signal and a right channel signal) or one or more received of these signals (for example, the center signal, or the downmix signal and the side signal, or the difference signal). In addition, the method comprises adapting the generation of one or more encoded audio signals depending on one or more directional loudness maps that represent loudness information associated with a plurality of different directions (eg, pan directions) of one or more signals to be encoded. Adaptation of the generation of one or more encoded audio signals, for example, is performed depending on the contributions of individual directional loudness maps (eg, associated with a single signal, a pair of signals, or a group of three or more signals) of one or more signals to be quantized to an overall directional loudness map, for example associated with multiple audio inputs (eg, each signal from one or more audio inputs)).

An embodiment according to the present invention relates to a method for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The method comprises providing one or more encoded (e.g., quantized and then lossless encoded) audio signals (e.g., encoded spectral domain representations) based on two or more input audio signals (e.g., a left channel signal and a right channel signal), or based on two or more more signals derived from them using joint encoding of two or more signals to be jointly encoded (eg, using a center signal or downmix signal and a side signal or difference signal). In addition, the method comprises selecting signals to be jointly encoded from a plurality of possible signals or from a plurality of pairs of possible signals (for example, from two or more input audio signals or from two or more signals derived from them) depending on directional loudness maps that represent information about loudness associated with a plurality of different directions (eg, panning directions) of possible signals or pairs of possible signals. According to an embodiment, the signals to be jointly encoded are selected depending on the contributions of the individual directional loudness maps of the possible signals to the overall directional loudness map associated with multiple audio inputs (e.g., each signal from one or more audio inputs), or depending on the contributions directional loudness maps of pairs of possible signals into a common directional loudness map.

An embodiment according to the present invention relates to a method for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The method comprises providing one or more encoded (e.g., quantized and then lossless coded) audio signals (e.g., encoded spectral domain representations) based on two or more input audio signals (e.g., a left channel signal and a right channel signal), or based on two or more more signals received from them. In addition, the method comprises determining an overall directional loudness map (e.g., a target scene directional loudness map) based on the input audio signals and/or determining one or more separate directional loudness maps associated with the individual audio input signals (and/or determining one or more directional loudness maps). volume associated with pairs of input audio signals). Additionally, the method comprises encoding a common directional loudness map and/or one or more individual directional loudness maps as ancillary information.

An embodiment according to the present invention relates to a method for decoding encoded audio content. The method comprises receiving an encoded representation of the one or more audio signals and providing a decoded representation of the one or more audio signals (eg, using AAC-like decoding or using decoding of entropy-encoded spectral values). Further, the method comprises receiving encoded directional loudness map information and decoding the encoded directional loudness map information to obtain one or more (eg, decoded) directional loudness maps. Additionally, the method comprises recreating an audio scene using a decoded representation of one or more audio signals and using one or more directional loudness maps.

An embodiment according to the present invention relates to a method for converting an audio content format that represents an audio scene (eg, a spatial audio scene) from a first format to a second format. The first format may contain, for example, a first number of channels or input audio signals and side information or spatial side information adapted to the first number of channels or input audio signals, and the second format may contain, for example, a second number of channels or audio output signals, which may differ from the first number of channels or input audio signals, and side information or spatial side information adapted to the second number of channels or output audio signals. The method comprises providing a representation of the audio content in a second format based on the representation of the audio content in the first format, and adjusting the format conversion complexity (e.g., by passing one or more input audio signals of the first format whose contribution to the directional loudness map is below a threshold during the format conversion) depending on from the contributions of the input audio signals of the first format (eg, one or more audio signals, one or more downmix signals, one or more difference signals, etc.) to the overall directional loudness map of the audio scene. The overall directional loudness map, for example, can be described by the ancillary information of the audio content in the first format received by the format converter.

An embodiment according to the present invention relates to a method that comprises receiving an encoded representation of one or more audio signals and providing a decoded representation of one or more audio signals (e.g., using AAC-like decoding or using decoding of entropy-coded spectral values). The method comprises restoring an audio scene using a decoded representation of one or more audio signals. In addition, the method comprises adjusting the decoding complexity depending on the contributions of the encoded signals (eg, one or more audio signals, one or more downmix signals, one or more difference signals, etc.) to the overall directional loudness map of the decoded audio scene.

An embodiment according to the present invention relates to a method for rendering audio content. According to an embodiment, the present invention relates to a method for upmixing audio content represented using a first number of input audio channels and ancillary information describing desired spatial characteristics, such as placement of audio objects or relationships between audio channels, into a representation containing more channels than than the first number of input audio channels. The method comprises recreating an audio scene based on one or more input audio signals (or based on two or more input audio signals). In addition, the method comprises adjusting the rendering complexity (e.g., by skipping one or more input audio signals whose contribution to the directional loudness map is below a threshold during rendering) depending on the contributions of the input audio signals (e.g., one or more audio signals, one or more downmixed signals, one or more difference signals, etc.) into an overall directional loudness map of the listenable audio scene. The overall directional loudness map, for example, can be described by ancillary information received by the renderer.

An embodiment in accordance with the present invention relates to a computer program having program code for performing the method described herein when executed on a computer.

An embodiment in accordance with the present invention relates to an encoded representation of audio data (eg, an audio stream or data stream) comprising an encoded representation of one or more audio signals and encoded information about directional loudness maps.

The methods described above are based on the same considerations as the audio parsing module, audio similarity estimator, audio encoder, audio decoder, format converter and/or renderer described above. Meanwhile, the methods can be supplemented with all the features and functionality that are also described in relation to the audio analysis module, the audio similarity estimator, the audio encoder, the audio decoder, the format converter and/or the renderer.

Brief description of the drawings

The drawings are not necessarily drawn to scale; instead, emphasis is usually placed on illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings.

Fig. 1 shows a block diagram of an audio data analysis module according to an embodiment of the present invention;

Fig. 2 shows a detailed block diagram of an audio data analysis module according to an embodiment of the present invention;

Fig. 3a shows a block diagram of an audio data analysis module using the first pan index approach, in accordance with an embodiment of the present invention;

Fig. 3b shows a block diagram of an audio data analysis module using a second panning index approach, in accordance with an embodiment of the present invention;

Fig. 4a shows a block diagram of an audio data analysis module using the first histogram approach, in accordance with an embodiment of the present invention;

Fig. 4b shows a block diagram of an audio data analysis module using a second histogram approach, in accordance with an embodiment of the present invention;

Fig. 5 shows diagrams of spectral domain representations to be analyzed by the audio data analysis module and results of directional analysis, loudness calculation for each frequency interval, and loudness calculation for each direction by the audio data analysis module according to an embodiment of the present invention;

Fig. 6 shows schematic histograms of two signals for directional analysis by an audio data analysis module according to an embodiment of the present invention;

Fig. 7a shows matrices with one non-zero scaling factor for each time-frequency bin associated with a direction for scaling performed by an audio data analysis module, in accordance with an embodiment of the present invention;

Fig. 7b shows matrices with multiple non-zero scaling factors for each time-frequency bin associated with a direction for scaling performed by an audio data analysis module, in accordance with an embodiment of the present invention;

Fig. 7c shows a schematic diagram of a printed circuit board with a first conductive track, a second conductive track after processing according to an embodiment of the present invention;

Fig. 8 shows a block diagram of an audio similarity estimator according to an embodiment of the present invention;

Fig. 9 shows a block diagram of an audio similarity estimator for stereo analysis according to an embodiment of the present invention;

Fig. 10a shows a color plot of a directional loudness reference map that can be used by an audio similarity estimator, in accordance with an embodiment of the present invention;

Fig. 10b shows a color plot of a directional loudness map to be analyzed by an audio similarity estimator according to an embodiment of the present invention;

Fig. 10c shows a color graph of a difference directional loudness map determined by an audio similarity estimator according to an embodiment of the present invention;

Fig. 11 shows a block diagram of an audio encoder according to an embodiment of the present invention;

Fig. 12 shows a block diagram of an audio encoder capable of adapting quantization parameters, in accordance with an embodiment of the present invention;

Fig. 13 shows a block diagram of an audio encoder capable of selecting signals to be encoded, in accordance with an embodiment of the present invention;

Fig. 14 shows a schematic drawing illustrating the determination of the contributions of individual directional loudness maps of possible signals to the overall directional loudness map performed by an audio encoder, in accordance with an embodiment of the present invention;

Fig. 15 shows a block diagram of an audio encoder capable of encoding directional loudness information as side information, according to an embodiment of the present invention;

Fig. 16 shows a block diagram of an audio decoder according to an embodiment of the present invention;

Fig. 17 shows a block diagram of an audio decoder capable of adapting decoding parameters according to an embodiment of the present invention;

Fig. 18 shows a block diagram of a format converter according to an embodiment of the present invention;

Fig. 19 shows a block diagram of an audio decoder capable of adjusting decoding complexity according to an embodiment of the present invention;

Fig. 20 shows a block diagram of a renderer according to an embodiment of the present invention;

Fig. 21 shows a flowchart of an audio signal analysis method according to an embodiment of the present invention;

Fig. 22 shows a flowchart of an audio similarity estimation method according to an embodiment of the present invention;

Fig. 23 shows a flowchart of a method for encoding input audio content containing one or more input audio signals, in accordance with an embodiment of the present invention;

Fig. 24 shows a flowchart of an audio joint coding method according to an embodiment of the present invention;

Fig. 25 shows a flowchart of a method for encoding one or more directional loudness maps as ancillary information in accordance with an embodiment of the present invention;

Fig. 26 shows a flowchart of a method for decoding encoded audio content according to an embodiment of the present invention;

Fig. 27 shows a flowchart of a method for converting an audio content format that represents an audio scene from a first format to a second format, according to an embodiment of the present invention;

Fig. 28 shows a flowchart of a method for decoding encoded audio content and adjusting decoding difficulty according to an embodiment of the present invention; And

Fig. 29 shows a flowchart of an audio content rendering method according to an embodiment of the present invention.

Implementation of the invention

Same or equivalent elements are elements with the same or equivalent functionality. In the following description, they are designated by equal or equivalent reference numerals, even if they occur in different drawings.

In the following description, many details are set forth in order to provide a more complete explanation of the embodiments of the present invention. However, those skilled in the art will appreciate that embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail so as not to obscure the embodiments of the present invention. In addition, the features of the various embodiments described below may be combined with each other, unless specifically indicated otherwise.

например, для i={L; R; DM} или для i

[1; I].In FIG. 1 shows a block diagram of an audio analysis module 100 that is configured to obtain a spectral domain representation 110 ₁ of a first input audio signal, such as X _{L, b} (m, k), and a spectral domain representation 110 ₂ of a second audio input signal, such as X _{R, b} (m, k). Thus, for example, the audio analysis module 100 receives spectral domain representations 110 ₁ , 110 ₂ as input 110 to be analyzed. This means, for example, that the first input audio signal and the second input audio signal are converted into spectral domain representations 110 ₁ , 110 ₂ by an external instrument or device and then provided to the audio analysis module 100 . Alternatively, spectral domain representations 110 ₁ , 110 ₂ may be determined by audio analysis module 100, as will be described with respect to FIG. 2. According to an embodiment, spectral domain representations 110 can be represented as

for example, for i={L; R; DM} or for i

[1; I].

(m, k), ассоциированной со спектральными полосами (например, спектральными интервалами k во временном кадре m) представлений 110₁, 110₂ в спектральной области. Информация 122 о направлении представляет, например, различные направления звуковых компонентов, содержащихся в двух или более входных аудиосигналах. Таким образом, информация 122 о направлении может иметь отношение к направлению, с которого слушатель услышит компонент, содержащийся в двух входных аудиосигналах. В соответствии с вариантом осуществления информация о направлении может представлять индексы панорамирования. Таким образом, например, информация 122 о направлении содержит первое направление, указывающее певца в помещении для прослушивания, и дополнительные направления, соответствующие различным музыкальным инструментам частотной полосы в аудиосцене. Информация 122 о направлении, например, определяется модулем 100 анализа аудиоданных посредством анализа отношений уровней между представлениями 110₁, 110₂ в спектральной области для всех частотных интервалов или групп частот (например, для всех спектральных интервалов k или спектральных полос b). Примеры для модуля 120 определения информации о направленности описаны относительно фиг. 5-7b.In accordance with an embodiment, the representations 110 ₁ , 110 ₂ in the spectral domain are fed to the direction information determination module 120 to obtain direction information 122, for example,

(m, k) associated with spectral bands (eg, spectral intervals k in time frame m) of representations 110 ₁ , 110 ₂ in the spectral domain. Direction information 122 represents, for example, different directions of audio components contained in two or more input audio signals. Thus, direction information 122 may relate to the direction from which the listener will hear the component contained in the two input audio signals. According to an embodiment, the direction information may represent panning indices. Thus, for example, direction information 122 includes a first direction indicating a singer in the listening room and additional directions corresponding to different frequency band musical instruments in the audio scene. The direction information 122 is, for example, determined by the audio data analysis module 100 by analyzing the level relationships between the representations 110 ₁ , 110 ₂ in the spectral domain for all frequency intervals or frequency groups (eg, for all spectral intervals k or spectral bands b). Examples for the directivity information determining module 120 are described with respect to FIG. 5-7b.

According to an embodiment, the audio data analysis module 100 is configured to obtain directional information 122 based on amplitude pan analysis of the audio content; and/or based on the analysis of the phase relationship, and/or time delay, and/or correlation between the audio content of two or more input audio signals; and/or based on the identification of extended (eg, decorrelated and/or panned) sources. The audio content may comprise input audio signals and/or spectral domain representations 110 of the input audio signals.

и

) в информацию 142 о громкости. В соответствии с вариантом осуществления первые вклады 132₁, ассоциированные с представлением 110₁ в спектральной области первого входного аудиосигнала, определяются блоком 130 определения вкладов в зависимости от информации 122 о направлении, и вторые вклады 132₂, ассоциированные с представлением 110₂ в спектральной области второго входного аудиосигнала, определяются блоком 130 определения вкладов в зависимости от информации 122 о направлении. В соответствии с вариантом осуществления информация 122 о направлении содержит различные направления (например, извлеченные значения направлений

(m, k)). Вклады 132 содержат, например, информацию о громкости для заданных направлений

в зависимости от информации 122 о направлении. В соответствии с вариантом осуществления вклады 132 определяют информацию об уровне спектральных полос, направление которых

(m, k) (соответствующее информации 122 о направлении) равняется заданным направлениям

, и/или масштабированную информацию об уровне спектральных полос, направление которых

(m, k) граничит с заданным направлением

.Based on the direction information 122 and spectral domain representations 110 ₁ , 110 ₂ , the audio analysis module 100 is configured to determine contributions 132 (eg,

And

) to loudness information 142. According to an embodiment, the first contributions 132 ₁ associated with the representation 110 ₁ in the spectral domain of the first input audio signal are determined by the contribution determiner 130 depending on the direction information 122, and the second contributions 132 ₂ associated with the representation 110 ₂ in the spectral domain of the second of the input audio signal are determined by the contribution determination block 130 depending on the direction information 122 . According to an embodiment, direction information 122 contains various directions (e.g., extracted direction values

(m, k)). Contributions 132 contain, for example, loudness information for given directions

depending on the information 122 about the direction. In accordance with an embodiment, the contributions 132 define information about the level of spectral bands whose direction

(m, k) (corresponding to the direction information 122) equals the given directions

, and/or scaled information about the level of spectral bands, the direction of which

(m, k) borders on the given direction

.

определяются в зависимости от значений в спектральной области (например,

как

и

как

в обозначении [13]) входных аудиосигналов.According to an embodiment, the extracted direction values

are determined depending on the values in the spectral region (for example,

How

And

How

in notation [13]) of audio input signals.

) для множества различных оценочных диапазонов направлений

(j

[1; J] для J заданных направлений)), ассоциированных с различными направлениями

(например, заданными направлениями) как результат анализа посредством модуля 100 анализа аудиоданных, модуль 100 анализа аудиоданных выполнен с возможностью объединения вкладов 132₁ (например,

), соответствующих представлению 110₁ в спектральной области первого входного аудиосигнала, и вкладов 132₂ (например,

), соответствующих представлению 110₂ в спектральной области второго входного аудиосигнала, чтобы принять объединенный сигнал как информацию 142 о громкости, например, двух или более каналов (например, первый канал ассоциирован с первым входным аудиосигналом и представлен индексом L, и второй канал ассоциирован со вторым входным аудиосигналом и представлен индексом R). Тес самым получается информация 142 о громкости, которая определяет громкость с течением времени и для каждого из различных направлений

. Это выполняется, например, посредством блока 140 определения информации о громкости.To obtain loudness information 142 (for example, L(m,

) for many different estimated ranges of directions

(j

[1; J] for J given directions)) associated with different directions

(for example, given directions) as a result of the analysis by the audio data analysis module 100, the audio data analysis module 100 is configured to combine the contributions 132 ₁ (for example,

) corresponding to the representation 110 ₁ in the spectral domain of the first input audio signal, and contributions 132 ₂ (for example,

) corresponding to the spectral domain representation 110 ₂ of the second input audio signal to receive the combined signal as loudness information 142 of, for example, two or more channels (for example, the first channel is associated with the first input audio signal and is represented by index L, and the second channel is associated with the second input audio signal and is represented by index R). This results in loudness information 142 which determines the loudness over time and for each of the various directions.

. This is done, for example, by the loudness information determination unit 140 .

In FIG. 2 shows an audio parsing module 100, which may include features and/or functionality as described with respect to the audio parsing module 100 of FIG. 1. According to an embodiment, audio analysis module 100 receives a first audio input x _L 112 ₁ and a second audio input x _R 112 ₂ . Index L is associated with the left side and index R is associated with the right side. The indexes may be associated with a loudspeaker (eg, with positioning of loudspeakers). According to an embodiment, the indexes may be represented by numbers indicating the channel associated with the input audio signal.

According to an embodiment, the first input audio signal 112₁ and/or second audio input signal 112₂may represent a time domain signal that may be converted by time domain to spectral domain conversion module 114 to obtain a spectral domain representation 110 from the corresponding input audio signal. In other words, the time-to-spectral domain conversion module 114 may decompose two or more input audio signals 112₁, 112₂ (e.g. x_L, x_R, x_i) into the Short Time Fourier Transform (STFT) region to obtain two or more transformed audio signals 115₁, 115₂ (e.g. X'_L,X'_R,X'_i). If the first input audio signal is 112₁ and/or second audio input 112₂ is a representation 110 in the spectral domain, the time-domain-to-spectral-domain conversion module 114 may be omitted.

Optionally, input audio signals 112 or converted audio signals 115 are processed by transform module 116 using an ear model to obtain a spectral domain representation 110 from the corresponding input audio signal 112 ₁ and 112 ₂ . The spectral intervals of the signal to be processed, eg 112 or 115, are grouped into spectral bands, eg based on a model for the perception of spectral bands by the human ear, and then the spectral bands can be weighted based on the outer ear and/or middle ear model. Thus, an optimized representation 110 in the spectral domain of the input audio signals 112 can be determined using the ear model processing module 116.

In accordance with an embodiment, the spectral domain representation 110 ₁ of the first input audio signal 112 ₁ , such as X _{L, b} (m, k), is associated with information about the level of the first input audio signal 112 ₁ (for example, denoted by the index L) and various spectral bands ( for example, marked with index b). For each spectral band b, the spectral domain representation 110 ₁ represents, for example, level information for time frames m and for all spectral slots k of the corresponding spectral band b.

In accordance with an embodiment, the spectral domain representation 110 ₂ of the second audio input signal 112 ₂ , such as X _{R, b} (m, k), is associated with information about the level of the second audio input signal 112 ₂ (for example, denoted by the index R) and various spectral bands ( for example, marked with index b). For each spectral band b, the spectral domain representation 110 ₂ represents, for example, level information for time frames m and for all spectral slots k of the corresponding spectral band b.

(m, k). Информация 125 о направлении панорамирования представляет, например, индексы панорамирования, соответствующие компонентам сигнала (например, компонентам первого входного аудиосигнала 112₁ и второго входного аудиосигнала 112₂, панорамированным в некотором направлении). В соответствии с вариантом осуществления входные аудиосигналы 112 ассоциированы с разными указанными направлениями, например, посредством индекса L для левой стороны и индекса R для правой стороны. Индекс панорамирования определяет, например, направление между двумя или более входными аудиосигналами 112 или направление в направлении входного аудиосигнала 112. Таким образом, например, в случае двухканального сигнала, как показано на фиг. 2, информация 125 о направлении панорамирования может содержать индексы панорамирования, соответствующие компонентам сигнала, панорамированным полностью в левую сторону или в правую сторону, или в каком-либо промежуточном направлении.Based on the representation 110 ₁ in the spectral domain of the first input audio signal 112 and the representation 110 ₂ in the spectral domain of the second input audio signal, determination of the information 120 about the direction can be performed by the audio analysis module 100 . Using the direction analysis module 124, pan direction information 125 can be determined, for example,

(m, k). The pan direction information 125 represents, for example, pan indices corresponding to signal components (eg, components of the first input audio signal 112 ₁ and the second input audio signal 112 ₂ panned in a certain direction). In accordance with an embodiment, the input audio signals 112 are associated with different indicated directions, for example, through an index L for the left side and an index R for the right side. The pan index determines, for example, the direction between two or more input audio signals 112, or the direction in the direction of the input audio signal 112. Thus, for example, in the case of a two-channel signal, as shown in FIG. 2, the pan direction information 125 may comprise pan indices corresponding to signal components panned all the way to the left or to the right, or in some direction in between.

для j

[1; i]. Зависящее от направления взвешивание 127 определяет, например, масштабный коэффициент в зависимости от направлений

(m, k), извлеченный из информации 125 о направлении панорамирования. Зависящее от направления взвешивание 127 определяется для множества заданных направлений

. В соответствии с вариантом осуществления зависящее от направления взвешивание 127 определяет функции для каждого заданного направления. Функции зависят, например, от направлений

(m, k), извлеченных из информации 125 о направлении панорамирования. Масштабный коэффициент зависит, например, от расстояния между направлениями

(m, k), извлеченных из информации 125 о направлении панорамирования, и заданным направлением

. Масштабные коэффициенты, т.е. зависящее от направления взвешивание 127, могут быть определены для каждого спектрального интервала и/или для каждого временного шага/временного кадра.According to an embodiment, based on the pan direction information 125, the audio analysis module 100 is configured to perform a scale factor determination 126 to determine direction-dependent weighting 127, for example,

for j

[1; i]. The direction-dependent weighting 127 determines, for example, the scaling factor depending on the directions

(m, k) extracted from the pan direction information 125 . The direction dependent weighting 127 is determined for a plurality of given directions

. In accordance with an embodiment, direction dependent weighting 127 determines functions for each given direction. Functions depend, for example, on directions

(m, k) extracted from the pan direction information 125 . The scaling factor depends, for example, on the distance between directions

(m, k) extracted from the pan direction information 125 and the given direction

. Scale factors, i.e. direction-dependent weighting 127 may be determined for each spectral interval and/or for each time step/time frame.

(m, k) и соответствующими заданными значениями направлений

.In accordance with an embodiment, the direction-dependent weighting 127 uses a Gaussian function, whereby the direction-dependent weighting decreases as the deviation between the corresponding extracted direction values increases.

(m, k) and corresponding given direction values

.

, ассоциированного с заданным направлением (например, представленным индексом

), временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k, в соответствии с

, где

- заданное значение (которое управляет, например, шириной гауссова окна); где

обозначает извлеченные значения направлений, ассоциированные со временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k; и где

- (например, заданное) значение направления, которое обозначает (или которое ассоциировано) заданное направление (например, имеющее индекс направления j).According to an embodiment, the audio data analysis module 100 is configured to obtain a direction dependent weighting 127

, associated with a given direction (for example, represented by the index

), a time (or time frame) denoted by time index m, and a spectral interval denoted by spectral interval index k, according to

, Where

- set value (which controls, for example, the width of the Gaussian window); Where

denotes the extracted direction values associated with a time (or time frame) denoted by time index m and a spectral interval denoted by spectral slot index k; and where

- (eg, given) direction value that denotes (or is associated with) a given direction (eg, having a direction index j).

According to an embodiment, the audio data analysis module 100 is configured to determine direction information comprising pan direction information 125 and/or direction-dependent weighting 127 by using the direction information determination module 120 . This direction information is obtained based on the audio content of two or more audio input signals 112, for example.

для разных

(j

[1; J] или j = {L; R; DM})). Другими словами, представление 110₁ в спектральной области первого входного аудиосигнала и представление 110₂ в спектральной области второго входного аудиосигнала взвешиваются для каждого заданного направления

индивидуально. Таким образом, например, взвешенное представление 135₁ в спектральной области, например,

первого входного аудиосигнала может содержать только компоненты сигнала первого входного аудиосигнала 112, соответствующего заданному направлению

, или дополнительно взвешенные (например, уменьшенные) компоненты сигнала первого входного аудиосигнала 112₁, ассоциированные с соседними заданными направлениями. Таким образом значения одного или более представлений 110 в спектральной области (например,

) взвешиваются в зависимости от различных направлений (например, направлений панорамирования

) (например, представленных весовыми коэффициентами

) звуковых компонентов.In accordance with an embodiment, the audio data analysis module 100 includes a scaling module 134 and/or a combining module 136 to determine 130 contributions. Using a scaler 134, direction-dependent weighting 127 is applied to one or more spectral domain representations 110 of two or more input audio signals 112 to obtain weighted spectral domain representations 135 (e.g.,

for different

(j

[1; J] or j = {L; R; DM})). In other words, the spectral domain representation 110 ₁ of the first audio input signal and the spectral domain representation 110 ₂ of the second audio input signal are weighted for each given direction.

individually. Thus, for example, a weighted representation of 135 ₁ in the spectral domain, for example,

of the first input audio signal may contain only the signal components of the first input audio signal 112 corresponding to the specified direction

, or additionally weighted (eg, reduced) signal components of the first input audio signal 112 ₁ associated with adjacent predetermined directions. Thus, the values of one or more representations 110 in the spectral domain (for example,

) are weighted depending on different directions (e.g. pan directions

) (for example, represented by weight coefficients

) audio components.

(m, k) которых отклоняются от заданного направления

, взвешиваются таким образом, чтобы они имели меньшее влияние, чем компоненты сигнала, извлеченные значения направлений

(m, k) которых равны заданному направлению

. Другими словами, в зависящем от направления взвешивании 127 для первого заданного направления компоненты сигнала

, ассоциированные с первым заданным направлением

, усиливаются по отношению к компонентам сигнала, ассоциированным с другими направлениями, в первом взвешенном представлении в спектральной области

, соответствующем первому заданному направлению

.In accordance with an embodiment, the scale factor determination module 126 is configured to determine a direction dependent weighting 127 such that, for each given direction signal components, the extracted direction values

(m, k) which deviate from the given direction

, are weighted so that they have less influence than the signal components, the extracted direction values

(m, k) which are equal to the given direction

. In other words, in the direction dependent weighting 127 for the first given direction of the signal component

, associated with the first given direction

, are amplified with respect to signal components associated with other directions, in the first weighted representation in the spectral domain

corresponding to the first given direction

.

, ассоциированных с входным аудиосигналом (например, 110₁ для i=1 или 110₂ для i=2) или объединением входных аудиосигналов (например, с объединением двух входных аудиосигналов 110₁ и 110₂ для i=1,2), обозначенных индексом i, спектральной полосой, обозначенной индексом b, (например, заданным) направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k, в соответствии с

где

обозначает представление 110 в спектральной области, ассоциированное с входным аудиосигналом 112 или объединением входных аудиосигналов 112, обозначенным индексом i (например, i=L, или i=R, или i=DM, или i представлен номером, указывающим канал), спектральной полосой, обозначенной индексом b, временем (или временным кадром), обозначенным временным индексом m. и спектральным интервалом, обозначенным индексом спектрального интервала k; и где

обозначает зависящее от направления взвешивание 127, ассоциированное с (например, заданным) направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k.In accordance with an embodiment, the audio data analysis module 100 is configured to obtain weighted representations 135 in the spectral domain

associated with the input audio signal (for example, 110₁ for i=1 or 110₂ for i=2) or by combining the input audio signals (for example, combining two input audio signals 110₁ And 110₂ for i=1,2) denoted by index i, spectral band denoted by index b, (for example, given) direction denoted by index

, a time (or time frame) denoted by time index m, and a spectral interval denoted by spectral interval index k, according to

Where

denotes a representation 110 in the spectral domain associated with an input audio signal 112 or a combination of input audio signals 112, denoted by index i (for example, i=L or i=R or i=DM or i is represented by a number indicating a channel), a spectral band, indicated by index b, time (or time frame) indicated by time index m. and a spectral interval, denoted by the index of the spectral interval k; and where

denotes a direction dependent weighting 127 associated with the (eg given) direction indicated by the index

, a time (or time frame) denoted by time index m, and a spectral interval denoted by spectral slot index k.

Additional or alternative functionality of scaler 134 is described with respect to FIG. 6-7b.

. Таким образом, с помощью модуля 136 объединения взвешенные представления 135 в спектральной области всех каналов (в случае фиг. 2 первого входного аудиосигнала 112₁ и второго входного аудиосигнала 112₂) соответствующие заданному направлению

, объединяются в один сигнал. Это выполняется, например, для всех заданных направлений

(для j

[1; i]). В соответствии с вариантом осуществления взвешенное объединенное представление 137 в спектральной области ассоциировано с разными частотными полосами b.In accordance with an embodiment, the weighted spectral domain representations 135 ₁ of the first input audio signal and the weighted spectral domain representations 135 ₂ of the second audio input signal are combined by a combiner 136 to obtain a weighted combined spectral domain representation 137

. Thus, using the combiner 136, the weighted representations 135 in the spectral domain of all channels (in the case of FIG. 2 of the first input audio signal 112 ₁ and the second input audio signal 112 ₂ ) corresponding to a given direction

are combined into one signal. This is done, for example, for all given directions

(for j

[1; i]). According to an embodiment, the weighted combined spectral domain representation 137 is associated with different frequency bands b.

. Таким образом, полученные значения 145 громкости частотных полос больше не зависят от отдельных спектральных интервалов k.Based on the weighted combined representation 137 in the spectral domain, loudness information determination 140 is performed to obtain loudness information 142 as an analysis result. According to an embodiment, the loudness information definition 140 comprises a loudness definition 144 in frequency bands and a loudness definition 146 across all frequency bands. In accordance with an embodiment, the frequency band loudness determination 144 is configured to determine the frequency band loudness values 145 for each spectral band b based on the weighted combined representations 137 in the spectral domain. In other words, determining 144 loudness in frequency bands determines the loudness in each spectral band depending on the given directions.

. Thus, the loudness values 145 obtained for the frequency bands no longer depend on the individual spectral intervals k.

) по спектральным значениям частотной полосы (или по спектральным интервалам (k) частотной полосы (b)), и применять возведение в степень с показателем между 0 и 1/2 (и предпочтительно меньше чем 1/3 или ¼) к среднему значению квадратов спектральных значений, чтобы определить значения 145 громкости частотных полос (например,

) (например, ассоциированных с соответствующей частотной полосой (b)).In accordance with an embodiment, the audio data analysis module is configured to calculate the average of the squares of the spectral values of the weighted combined representations 137 in the spectral domain (for example,

) over the spectral values of the frequency band (or over the spectral intervals (k) of the frequency band (b)), and apply exponentiation with an exponent between 0 and 1/2 (and preferably less than 1/3 or ¼) to the average of the squares of the spectral values to determine the loudness values 145 of the frequency bands (for example,

) (eg associated with the corresponding frequency band (b)).

, ассоциированных со спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, в соответствии с

где K_b обозначает количество спектральных интервалов в частотной полосе, имеющей индекс частотной полосы b; где k - бегущая переменная и обозначает спектральные интервалы в частотной полосе, имеющей индекс частотной полосы b; где b обозначает спектральную полосу; и где

обозначает взвешенное объединенное представление 137 в спектральной области, ассоциированное со спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, временем (или временным кадром), обозначенным временным индексом m, и спектральным интервалом, обозначенным индексом спектрального интервала k.According to an embodiment, the audio data analysis module is configured to obtain loudness values of 145 frequency bands

, associated with the spectral band, denoted by the index b, the direction, denoted by the index

, the time (or time frame) denoted by time index m, according to

where K _b denotes the number of spectral intervals in the frequency band having the frequency band index b; where k is a running variable and denotes spectral intervals in a frequency band having a frequency band index b; where b denotes the spectral band; and where

denotes the weighted combined representation 137 in the spectral domain associated with the spectral band indicated by the index b, the direction indicated by the index

, a time (or time frame) denoted by time index m, and a spectral slot denoted by spectral slot index k.

.In determining 146 loudness information across all frequency bands, the loudness values 145 of the frequency bands are, for example, averaged over all spectral bands to provide loudness information 142 dependent on a given direction and at least one time frame m. According to an embodiment, the loudness information 142 may represent the overall loudness caused by the input audio signals 112 in various directions in the listening room. In accordance with an embodiment, loudness information 142 may be associated with combined loudness values associated with various defined or predetermined directions.

.

), ассоциированных с направлением, обозначенным индексом

, и временем (или временным кадром), обозначенным временным индексом m, в соответствии с

где B обозначает общее количество спектральных полос b, и где

обозначает значения 145 громкости частотных полос, ассоциированных со спектральной полосой, обозначенной индексом b, направлением, обозначенным индексом

, и временем [или временным кадром], обозначенным временным индексом m.The audio data analysis module as set forth in any one of 1 to 17, wherein the audio data analysis module is configured to obtain a plurality of combined loudness values L(m,

) associated with the direction indicated by the index

, and the time (or time frame) denoted by time index m, according to

where B denotes the total number of spectral bands b, and where

denotes the loudness values 145 of the frequency bands associated with the spectral band indicated by the index b, the direction indicated by the index

, and time [or time frame] denoted by time index m.

In FIG. 1 and FIG. 2, the audio data analysis module 100 is configured to analyze the spectral domain representation 110 of two input audio signals, but the audio data analysis module 100 is also configured to analyze more than two spectral domain representations 110.

In FIG. 3a-4b show various implementations of the audio analysis module 100. The audio data analysis module shown in FIG. 1-4b is not limited to features and functionality shown for one implementation, but may also contain features and functionality from other implementations of the audio data parsing module shown in various FIGS. 1-4b.

In FIG. 3a and fig. 3b shows two different approaches for determining loudness information 142 based on determining the pan index by the audio analysis module 100 .

The audio data analysis module 100 shown in FIG. 3a is similar or equivalent to the audio analysis module 100 shown in FIG. 2. Two or more input signals 112 are converted to time-frequency signals 110 by time-frequency decomposition 113. According to an embodiment, time-frequency decomposition 113 may comprise time domain to spectral domain transformation and/or ear model processing.

Based on the time-frequency signals, direction information is determined 120 . Determination 120 information about the direction contains, for example, directional analysis 124 and the definition of 126 window functions. In the contribution determiner 130, directional signals 132 are obtained, for example, by separating the time-frequency signals 110 into directional signals by applying direction-dependent window functions 127 to the time-frequency signals 110. Based on the directional signals 132, loudness calculation 140 is performed to obtain information 142 about loudness as a result of the analysis. Loudness information 142 may include a directional loudness map.

The audio data analysis module 100 in FIG. 3b is different from the audio analysis unit 100 in FIG. 3a by calculating 140 the volume. In accordance with FIG. 3b, the loudness calculation 140 is performed before the calculation of the directional signals of the time-frequency signals 110. Thus, for example, in accordance with FIG. 3b, the loudness values 141 of the frequency bands are computed directly from the time-frequency signals 110. By applying a direction dependent window function 127 to the loudness values 141 of the frequency bands 141, directional loudness information 142 can be obtained as an analysis result.

Fig. 4a and FIG. 4b show an audio analysis module 100 which, according to an embodiment, is configured to determine loudness information 142 using a histogram approach. According to an embodiment, audio data analysis module 100 is configured to use time-frequency decomposition 113 to determine time-frequency signals 110 based on two or more input signals 112.

In accordance with an embodiment based on the time-frequency signals 110, a loudness calculation 140 is performed to obtain a combined loudness value 145 for each time-frequency cell. The combined loudness value 145 is not associated with any direction information. The value of the combined loudness, for example, is associated with the loudness resulting from the imposition of the input signals 112 on the frequency-time cell.

In addition, the audio data analysis module 100 is configured to perform direction analysis 124 of the time-frequency signals 110 to obtain direction information 122 . In accordance with FIG. 4a, direction information 122 comprises one or more direction vectors with ratio values indicating time-frequency cells with the same level relationship between two or more input signals 112. This direction analysis 124, for example, is performed as described with respect to FIG. 5 or FIG. 6.

The audio data analysis module 100 in FIG. 4b is different from the audio analysis unit 100 shown in FIG. 4a in that, after the direction analysis 124, a directional blur 126 of the direction values 122 ₁ is optionally performed. With directional blur 126, also, time-frequency cells associated with directions adjacent to a given direction can be associated with a given direction, in which the obtained direction information 122 ₂ may further comprise a scale factor for these time-frequency cells to minimize the effect of in the given direction.

In FIG. 4a and FIG. 4b, audio data analysis module 100 is configured to accumulate 146 combined loudness values 145 in direction histogram bins based on direction information 122 associated with time-frequency bins.

More details about the audio analysis module 100 in FIG. 3a and fig. 3b is described further in the chapter "Generalized Steps for Computing a Directional Loudness Map" and in the chapter "Embodiments for Various Forms of Computing Loudness Maps Using Generalized Criteria Functions".

In FIG. 5 shows a spectral domain representation 110 ₁ of a first input audio signal and a spectral domain representation 110 ₂ of a second audio input signal to be analyzed by the audio analysis module described herein. An analysis 124 of the direction of representations 110 in the spectral domain results in information 122 about the direction. According to an embodiment, the direction information 122 represents a direction vector with relationship values between the spectral domain representation 110 ₁ of the first input audio signal and the spectral domain representation 110 ₂ of the second audio input signal. Thus, for example, frequency bins, eg, time-frequency bins of representations 110 in the spectral domain with the same level ratio, are associated with the same direction 125.

In accordance with an embodiment, loudness calculation 140 results in combined loudness values 145, for example, for each time-frequency cell. The combined volume values 145 are, for example, associated with a combination of a first audio input signal and a second audio input signal (eg, a combination of two or more audio input signals).

Based on the direction information 122 and the combined loudness values 145, combined loudness values 145 can be accumulated 146 over direction and time dependent histogram intervals. Thus, for example, all combined loudness values 145 associated with some direction are summed. According to direction information 122, directions are associated with time-frequency cells. With the accumulation 146, the result is a directional loudness histogram that can represent loudness information 142 as a result of the analysis of the audio data analysis module described here.

It is also possible that time-frequency cells corresponding to the same direction and/or adjacent directions in a different or adjacent time frame (e.g., in a previous or next time frame) may be associated with a direction in the current time step or time frame. This means, for example, that direction information 122 contains direction information for each frequency cell (or frequency slot) as a function of time. Thus, for example, direction information 122 is obtained for several time frames or for all time frames.

More details on the histogram approach shown in FIG. 5 will be described in the chapter "Variants for the implementation of various forms of calculation of loudness maps using generalized criterion functions", option 2.

In FIG. 6 shows the determination of 130 contributions based on the pan direction information performed by the audio analysis module described herein. In FIG. 6a shows a spectral domain representation of the first input audio signal, and FIG. 6b shows the spectral domain representation of the second input audio signal. In accordance with FIG. 6a1-6a3.1 and FIGS. 6b1 to 6b3.1, spectral intervals or spectral bands corresponding to the same pan direction are selected to calculate loudness information in that pan direction. Fig. 6a3.2 and fig. 6b3.2 show an alternative process in which not only the frequency slots or frequency bands corresponding to the pan direction are considered, but also other frequency slots or frequency groups that are weighted or scaled for less influence. More details regarding FIG. 6 is described in the chapter "Restoring Directional Signals Using the Windowing/Selection Function Derived from the Pan Index".

In accordance with an embodiment, direction information 122 may comprise scale factors associated with direction 121 and time-frequency bins 123, as shown in FIG. 7a and/or FIG. 7b. According to the embodiment in FIG. 7a and FIG. 7b, time-frequency cells 123 are shown for only one time step or time frame. Fig. 7a shows scaling factors that consider only time-frequency cells 123 that contribute in some (eg, given) direction 121, as eg described with respect to FIG. 6a1-6a3.1 and figs. 6b1-6b3.1. Alternatively, in FIG. 7b also considers neighboring directions, but these are scaled to reduce the influence of the respective time-frequency cell 123 in neighboring directions. In accordance with FIG. 7b, the time-frequency cell 123 is scaled such that its influence will decrease as the deviation from the associated direction increases. Instead, in FIG. 6a3.2 and fig. 6b3.2, all time-frequency bins corresponding to a different panning direction are scaled equally. Various calculations or weightings are possible. Depending on the scaling, the accuracy of the analysis result of the audio data analysis module can be improved.

)) и второй информации 142₂ о громкости (например, L₂(m,

)). Первая информация 142₁ о громкости ассоциирована с различными направлениями (например, с заданными направлениями панорамирования

) на основе первого множества из двух или более входных аудиосигналов 112a (например, x_L, x_R или x_i для i ϵ [1;n]), и вторая информация 142₂ о громкости ассоциирована с различными направлениями на основе второго множества из двух или более входных аудиосигналов, которые могут быть представлены множеством эталонных аудиосигналов 112b (например, x_2,R, x_2,L, x_2,i для i ϵ [1;n]). Первое множество входных аудиосигналов 112a и множество эталонных аудиосигналов 112b могут содержать n аудиосигналов, где n представляет целое число больше или равное 2. Каждый аудиосигнал первого множества входных аудиосигналов 112a и множества эталонных аудиосигналов 112b может быть ассоциировано с разными громкоговорителями, помещенными в разные положения в пространстве прослушивания. Первая информация 142₁ о громкости и вторая информация 142₂ о громкости могут представлять распределение громкости в пространстве прослушивания (например, в положениях громкоговорителей или между ними). В соответствии с вариантом осуществления первая информация 142₁ о громкости и вторая информация 142₂ о громкости содержат значения громкости для дискретных положений или направлений в пространстве прослушивания. Различные направления могут быть ассоциированы с направлениями панорамирования аудиосигналов, выделенными для одного множества аудиосигналов 112a или 112b в зависимости от того, какое множество соответствует информации о громкости, подлежащей вычислению.In FIG. 8 shows an embodiment of an audio similarity estimator 200 . Audio similarity estimator 200 is configured to obtain first loudness information 142 ₁ (for example, L ₁ (m,

)) and second loudness information 142 ₂ (for example, L ₂ (m,

)). The first volume information 142 ₁ is associated with different directions (for example, given pan directions

) based on a first set of two or more input audio signals 112a (e.g., x _L , x _R or x _i for i ϵ [1;n]), and second loudness information 142 ₂ is associated with different directions based on a second set of two or more input audio signals, which may be represented by a plurality of reference audio signals 112b (eg, x _2,R , x _2,L , x _2,i for i ϵ [1;n]). The first set of input audio signals 112a and the set of reference audio signals 112b may contain n audio signals, where n represents an integer greater than or equal to 2. Each audio signal of the first set of input audio signals 112a and the set of reference audio signals 112b may be associated with different speakers placed at different positions in space listening. The first loudness information 142 ₁ and the second loudness information 142 ₂ may represent the distribution of loudness in the listening space (eg, at or between speaker positions). According to an embodiment, the first loudness information 142 ₁ and the second loudness information 142 ₂ contain loudness values for discrete positions or directions in the listening space. Different directions may be associated with audio pan directions allocated to one set of audio signals 112a or 112b, depending on which set corresponds to the loudness information to be calculated.

First information 142₁ about volume and second information 142₂ about the loudness can be determined by determining 100 information about the loudness, which can be performed by the module 200 evaluating the similarity of audio data. According to an embodiment, the loudness information determination 100 may be performed by an audio analysis module. Thus, for example, the audio data similarity estimator 200 may comprise an audio data analysis module or receive first information 142₁ volume and/or other information 142₂ volume from an external audio analysis module. According to an embodiment, the audio data analysis module may comprise features and/or functionality as described with respect to the audio data analysis module in FIG. 1-4b. Alternatively, only the first information 142₁ about the loudness is determined by determining 100 information about the loudness, and the second information 142₂ about the loudness is received or obtained by the module 200 evaluating the similarity of audio data from the data bank with the reference information about the loudness. In accordance with an embodiment, the data bank may contain reference maps of loudness information for various speaker settings and/or speaker configurations, and/or various sets of reference audio signals 112b.

In accordance with an embodiment, the set of reference audio signals 112b may represent an ideal set of audio signals for an optimized listener experience of audio data in the listening space.

) до L₁(m,

)) и/или вторая информация 142₂ о громкости (например, вектор, содержащий элементы от L₂(m,

) до L₂(m,

)), могут содержать множество значений объединенной громкости, ассоциированных с соответствующими входными аудиосигналами (например, входными аудиосигналами, соответствующими первому множеству входных аудиосигналов 112a, или эталонными аудиосигналами, соответствующими множеству эталонных аудиосигналов 112b (и ассоциированными с соответствующими заданными направлениями)). Соответствующие заданные направления могут представлять индексы панорамирования. Поскольку каждый входной аудиосигнал, например, ассоциирован с громкоговорителем, соответствующие заданные направления могут восприниматься как равномерно распределенные положения между соответствующими громкоговорителями (например, между соседними громкоговорителями и/или другими парами громкоговорителей). Другими словами, модуль 200 оценки сходства аудиоданных выполнен с возможностью получения компонента направления (например, описанного здесь первого направления), используемого для получения информации 142₁ и/или 142₂ о громкости с другими направлениями (например, описанным здесь вторым направлением) с использованием метаданных, представляющих информацию о положении громкоговорителей, ассоциированных с входными аудиосигналами. Значения объединенной громкости первой информации 142₁ о громкости и/или второй информации 142₂ о громкости описывают громкость компонентов сигнала соответствующего множества входных аудиосигналов 112a и 112b, ассоциированных с соответствующими заданными направлениями. Первая информация 142₁ о громкости и/или вторая информация 142₂ о громкости ассоциированы с комбинациями множества взвешенных представлений в спектральной области, ассоциированных с соответствующим заданным направлением.According to an embodiment, the first loudness information 142 ₁ (for example, a vector containing elements from L ₁ (m,

) to L ₁ (m,

)) and/or second loudness information 142 ₂ (eg, a vector containing elements from L ₂ (m,

) to L ₂ (m,

)) may comprise a plurality of combined loudness values associated with respective input audio signals (e.g., audio input signals corresponding to the first set of audio input signals 112a, or reference audio signals corresponding to the set of reference audio signals 112b (and associated with the respective predetermined directions)). The respective predetermined directions may represent panning indices. Since each input audio signal is, for example, associated with a loudspeaker, the respective predetermined directions may be perceived as evenly spaced positions between the respective loudspeakers (eg, between adjacent loudspeakers and/or other pairs of loudspeakers). In other words, audio similarity estimator 200 is configured to obtain a direction component (eg, the first direction described here) used to obtain loudness information 142 ₁ and/or 142 ₂ with other directions (eg, the second direction described here) using metadata. , representing information about the position of the speakers associated with the input audio signals. The combined loudness values of the first loudness information 142 ₁ and/or the second loudness information 142 ₂ describe the loudness of the signal components of the respective plurality of input audio signals 112a and 112b associated with the respective predetermined directions. The first loudness information 142 ₁ and/or the second loudness information 142 ₂ are associated with combinations of a plurality of spectral domain weighted representations associated with the respective predetermined direction.

The audio similarity evaluator 200 is configured to compare the first loudness information 142 ₁ with the second loudness information 142 ₂ to obtain similarity information 210 describing the similarity between the first set of two or more input audio signals 112a and the set of two or more reference audio signals. 112b. This may be performed by the loudness information comparer 220 . Similarity information 210 may indicate the quality of the first set of input audio signals 112a. To further improve the perception prediction of the first set of input audio signals 112a based on the similarity information 210, only a subset of the frequency bands in the first loudness information 142 ₁ and/or the second loudness information 142 ₂ can be considered. According to an embodiment, the first loudness information 142 ₁ and/or the second loudness information 142 ₂ are only determined for frequency bands of 1.5 kHz and above. Thus, the compared loudness information 142 ₁ and 142 ₂ can be optimized based on the sensitivity of the human auditory system. Thus, the loudness information comparer 220 is configured to compare the loudness information 142 ₁ and 142 ₂ which contains only the loudness values of the relevant frequency bands. Relevant frequency bands may be associated with frequency bands corresponding to sensitivity (eg, human ear) above a given threshold for given level differences.

To obtain similarity information 210, for example, the difference between the second loudness information 142 ₂ and the first loudness information 142 ₁ is calculated.

This difference may represent difference loudness information and may already define similarity information 210. Alternatively, the difference loudness information is further processed to obtain similarity information 210. According to an embodiment, the audio data similarity estimator 220 is configured to determine a value that determines the magnitude of a multi-directional difference. This value may be a single scalar value representing the similarity information 210 . To obtain a scalar value, the loudness information comparer 220 may be configured to calculate a difference for sections or total duration of the first set of input audio signals 112a and/or a set of reference audio signals 112b and then average the obtained difference loudness information over all panning directions (e.g., over different directions with which the first loudness information 142 ₁ and/or the second loudness information 142 ₂ are associated) and over time to produce a single number called the model output variable (MOV).

In FIG. 9 shows an embodiment of an audio similarity estimator 200 for calculating similarity information 210 based on a stereo reference input signal 112b (REF) and a stereo signal 112a to be analyzed (eg, in this case, a signal under test (SUT)). According to an embodiment, the audio similarity estimator 200 may comprise features and/or functionality as described with respect to the audio similarity estimator in FIG. 8. The two stereo signals 112a and 112b may be processed by peripheral ear model 116 to obtain spectral domain representations 110a and 110b of stereo input signals 112a and 112b.

According to an embodiment, in a next step, the audio components of the stereo signals 112a and 112b can be analyzed for their direction information. Different pan directions 125 may be specified and may be combined with the window width 128 to obtain a direction dependent weighting 127 ₁ -127 ₇ . Based on the direction-dependent weighting 127 and spectral domain representation 110a and/or 110b of the respective stereo input signal 112a and/or 112b, pan index directional decomposition 130 can be performed to obtain contributions 132a and/or 132b. According to an embodiment, the contributions 132a and/or 132b are then processed, for example, by loudness calculation 144, to obtain loudness 145a and/or 145b for each frequency band and pan direction. According to an embodiment, ERB (Equivalent Rectangular Bandwidth) frequency averaging 146 of loudness signals 145b and/or 145a is performed to obtain directional loudness maps 142a and/or 142b for comparing 220 loudness information. The comparison 220 of the loudness information, for example, is configured to calculate a distance measure based on the two directional loudness maps 142a and 142b. The distance measure may represent a directional loudness map containing the differences between the two directional loudness maps 142a and 142b. In accordance with an embodiment, a single number, called the output variable of the MOV model, can be obtained as similarity information 210 by averaging a distance measure over all pan directions and time.

Fig. 10c shows a distance measure as described in FIG. 9 or similarity information as described in FIG. 8 represented by a directional loudness map 210 showing loudness differences between the directional loudness map 142b shown in FIG. 10a and the directional volume map 142a shown in FIG. 10b. The directional loudness maps shown in FIG. 10a-10c represent, for example, volume values over time and across panning directions. The directional loudness map shown in FIG. 10a may represent loudness values corresponding to an input signal with a reference value. This directional loudness map can be computed as described in FIG. 9 or via an audio data analysis module as described in FIG. 1-4b, or alternatively may be taken from a database. The directional loudness map shown in FIG. 10b corresponds to, for example, the stereo signal being tested, and may represent loudness information determined by the audio data analysis module as explained in FIG. 1-4b and FIG. 8 or 9.

In FIG. 11 shows an audio encoder 300 for encoding 310 audio input content 112 containing one or more audio input signals (eg x _i ). The input audio content 112 preferably contains a plurality of input audio signals such as stereo or multi-channel signals. Audio encoder 300 is configured to provide one or more encoded audio signals 320 based on one or more input audio signals 112 or based on one or more signals 110 obtained from one or more input audio signals 112 through optional processing 330. Thus, either one or more input audio signals 112, or one or more signals 110 derived therefrom, are encoded 310 by audio encoder 300. Processing 330 may include center/side processing, downmix or difference processing, time domain to spectral domain conversion, and/or ear model processing. Encoding 310 includes, for example, quantization and then lossless coding.

) для множества различных

), которые представляют информацию о громкости, ассоциированную с множеством различных направлений (например, заданных направлений или направлений одного или более сигналов 112, подлежащих кодированию). В соответствии с вариантом осуществления параметры кодирования содержат параметры квантования и/или другие параметры кодирования, такие как распределение битов и/или параметры, относящиеся к запрещению/разрешению кодирования 310.Audio encoder 300 is configured to adapt 340 encoding parameters depending on one or more directional loudness maps 142 (e.g., L _i (m,

) for many different

) that represent loudness information associated with a plurality of different directions (eg, given directions or directions of one or more signals 112 to be encoded). According to an embodiment, the coding parameters comprise quantization parameters and/or other coding parameters such as bit allocation and/or parameters related to coding inhibit/enable 310.

In accordance with an embodiment, audio encoder 300 is configured to perform determination 100 of loudness information to obtain a directional loudness map 142 based on input audio signal 112 or based on processed input audio signal 110. Thus, for example, audio encoder 300 may comprise an audio analysis module 100 as described with respect to FIG. 1-4b. Alternatively, the audio encoder 300 may receive a directional loudness map 142 from an external audio analysis module that performs loudness information determination 100 . According to an embodiment, audio encoder 300 may receive more than one directional loudness map 142 related to audio input signals 112 and/or processed audio input signals 110.

According to an embodiment, the audio encoder 300 can only receive one input audio signal 112. In this case, the directional loudness map 142 contains, for example, loudness values for only one direction. In accordance with an embodiment, the directional loudness map 142 may contain volume values equal to zero for directions other than the direction associated with the input audio signal 112. be performed adaptation 340 of the encoding parameters. Thus, for example, adaptation 340 of the coding parameters may comprise setting the coding parameters to standard coding parameters for mono signals.

If audio encoder 300 receives a stereo or multi-channel signal as input audio signal 112, directional loudness map 142 may contain loudness values for different directions (eg, non-zero). In the case of a stereo input signal, audio encoder 300 obtains, for example, one directional volume map 142 associated with two input audio signals 112. signal 112 is encoded by audio encoder 300, then, for example, a general directional loudness map 142 based on all channel signals and/or directional loudness maps and/or one or more directional loudness maps 142 based on signal pairs of multi-channel input audio signal 112 can be obtained by determining 100 volume information. Thus, for example, audio encoder 300 may be configured to adapt 340 coding parameters depending on the contributions of individual directional loudness maps 142, such as signal pairs, center signal, side signal, downmix signal, difference signal, and/or groups of three. or more signals into a common directional volume map 142, e.g., associated with multiple audio inputs, e.g., associated with all signals of multi-channel audio input 112 or processed multi-channel audio input 110.

Loudness information determination 100 as described with respect to FIG. 11 is illustrative and may be performed in the same or similar manner by all subsequent audio encoders or decoders.

)), которые представляют собой входной аудиосодержимое 112 и/или кодированную версию 320 входного аудиосодержимого 112, или которые основаны на них.In FIG. 12 shows an embodiment of an audio encoder 300, which may include features and/or functionality as described with respect to the audio encoder in FIG. 11. According to an embodiment, coding 310 may comprise quantization by quantizer 312 and coding by coding unit 314, such as entropy coding. Thus, for example, adapting encoding parameters 340 may comprise adapting quantization parameters 342 and adapting encoding parameters 344. The audio encoder 300 is configured to encode 310 audio input content 112 comprising, for example, two or more audio inputs, to provide encoded audio content 320 comprising, for example, encoded two or more audio inputs. This encoding 310 depends on, for example, a directional loudness map 142 or a plurality of directional loudness maps 142 (for example, L _i (m,

)) that are or are based on the input audio content 112 and/or the encoded version 320 of the input audio content 112.

According to an embodiment, the input audio content 112 may be directly encoded 310 or optionally processed 330 earlier. As already described above, audio encoder 300 may be encoded to determine the spectral domain representation 110 of one or more input audio signals of input audio content 112 through processing 330. Alternatively, processing 330 may include further processing steps to obtain one or more input audio content signals 112 that may be converted from the time domain to the spectral domain to obtain a representation 110 in the spectral domain. In accordance with an embodiment, the signals obtained by processing 330 may comprise, for example, a center signal or downmix signal and a side signal or difference signal.

According to an embodiment, signals of input audio content 112 or spectral domain representation 110 may be quantized by a quantizer 312 . Quantization module 312 uses, for example, one or more quantization parameters to obtain one or more quantized representations 313 in the spectral domain. These one or more quantized spectral domain representations 313 may be encoded by an encoder 314 to obtain one or more encoded audio signals of the encoded audio content 320.

To optimize encoding 310 by audio encoder 300, audio encoder 300 may be configured to adapt 342 quantization parameters. The quantization parameters, for example, contain scaling factors or parameters describing which quantization precision or which quantization steps are to be applied to which spectral intervals of the frequency bands of the one or more signals to be quantized. According to an embodiment, the quantization parameters describe, for example, the allocation of bits to different signals to be quantized and/or different frequency bands. Adaptation 342 of the quantization parameters can be understood as adapting the quantization accuracy and/or adapting the noise introduced by the encoder 300 and/or adapting the bit distribution between one or more signals 112/110 and/or parameters to be encoded by the audio encoder 300. In other words, the audio encoder 300 configured to adjust one or more quantization parameters to adapt the bit allocation for quantization accuracy adaptation and/or noise adaptation. Additionally, the quantization parameters and/or coding parameters may be encoded 310 by the audio encoder.

According to an embodiment, parameter encoding adaptation 340, such as quantization parameter adaptation 342 and encoding parameter adaptation 344, may be performed depending on one or more directional loudness maps 142 that represent loudness information associated with a plurality of different directions, pan directions, one or more signals 112/110 to be quantized. More specifically, the adaptation 340 may be performed depending on the contributions of the individual directional loudness maps 142 of one or more signals to be encoded to the overall directional loudness map 142. This can be done as described with respect to FIG. 11. Thus, for example, bit allocation adaptation, quantization accuracy adaptation, and/or noise adaptation may be performed depending on the contributions of individual directional loudness maps of one or more signals to be encoded 112/110 to the overall directional loudness map. This is done, for example, by adjusting one or more quantization parameters through adaptation 342.

According to an embodiment, audio encoder 300 is configured to determine an overall directional loudness map based on input audio signals 112 or spectral domain representation 110, whereby the overall directional loudness map represents loudness information associated with different directions, for example, audio components of an audio scene, represented by input audio content 112. Alternatively, the overall directional loudness map may represent loudness information associated with different directions of the audio scene to be represented, for example, after rendering at the decoder side. According to an embodiment, different directions can be obtained by determining 100 loudness information, possibly in combination with knowledge or ancillary information regarding speaker positions and/or knowledge or ancillary information describing the positions of sound objects. This knowledge or ancillary information may be derived from one or more of the signals 112/110 to be quantized, since these signals 112/110, for example, are associated in a fixed, signal-independent manner with different directions, or with different speakers, or with different audio objects. For example, the signal is associated with some channel, which can be interpreted as a direction from various directions (for example, the first directions described here). According to an embodiment, sound objects of one or more signals are panned in different directions or transformed to be heard in different directions, which can be obtained by defining 100 loudness information as object rendering information. This knowledge or ancillary information can be obtained by determining 100 loudness information for groups of two or more input audio signals of the input audio content 112 or a representation 110 in the spectral domain.

According to an embodiment, the signals 112/110 to be quantized may comprise components such as a center signal and a side signal of a stereo center/side coding, joint multi-signal coding of two or more input audio signals 112. Thus, the audio encoder 300 is configured to evaluate the aforementioned contributions of directional loudness maps 142 of one or more joint multi-signal coding differential signals to a common directional loudness map 142 and adjusting one or more coding parameters 340 depending on them.

According to an embodiment, the audio encoder 300 is configured to adapt the bit distribution between one or more signals 112/110 and/or parameters to be encoded and/or adapt the quantization accuracy of one or more signals 112/110 to be encoded and/or adapt noise introduced by the encoder 300 individually for different spectral intervals or individually for different frequency bands. This means, for example, that the adaptation 342 of the quantization parameters is performed such that the coding 310 is improved for individual spectral intervals or individual different frequency bands.

In accordance with an embodiment, audio encoder 300 is configured to adapt the bit allocation between one or more signals 112/110 and/or parameters to be encoded depending on an estimate of the spatial masking between the two or more signals to be encoded. The audio encoder is, for example, configured to estimate spatial concealment based on directional loudness maps 142 associated with two or more signals 112/110 to be encoded. Additionally or alternatively, the audio encoder is configured to evaluate the spatial masking or masking effect of the loudness contribution associated with the first direction of the first signal to be encoded on the loudness contribution associated with the second direction, which is different from the first direction of the second signal to be encoded. According to an embodiment, the loudness contribution associated with the first direction may represent, for example, loudness information of an audio object or audio component of input audio content signals, and the loudness contribution associated with the second direction may represent, for example, loudness information associated with another audio object or audio component of input audio content signals. Depending on the loudness information of the loudness contribution associated with the first direction and the loudness contribution associated with the second direction, and depending on the distance between the first direction and the second direction, a masking or spatial masking effect can be estimated. According to an embodiment, the masking effect decreases as the angle difference between the first direction and the second direction increases. Similarly, temporal masking may be evaluated.

According to an embodiment, quantization parameter adaptation 342 may be performed by audio encoder 300 to adapt the noise introduced by encoder 300 based on a directional loudness map achievable with encoded version 320 of input audio content 112. Thus, audio encoder 300 is, for example, configured to use deviation between the directional loudness map 142 that is associated with a particular unencoded 112/110 audio input (or two or more audio inputs) and the directional loudness map achievable with an encoded version 320 of a particular 112/110 audio input (or two or more audio inputs). audio signals), as a criterion for adapting the generation of a certain encoded audio signal or audio signals of the encoded audio content 320. This deviation may represent the quality of the encoding 310 of the encoder 300. Thus, the encoder 300 may be configured to adapt 340 encoding parameters such that the deviation is below a certain threshold. . Thus, a feedback loop 322 is implemented to improve encoding 310 by audio encoder 300 based on directional loudness maps 142 of encoded audio content 320 and directional loudness maps 142 of uncoded input audio content 112 or uncoded spectral domain representations 110. According to an embodiment, in the feedback loop 322, the encoded audio content 320 is decoded to perform loudness information determination 100 based on the decoded audio signals. Alternatively, it is also possible that the directional loudness maps 142 of the encoded audio content 320 are performed by a neural network-implemented feedforward (eg, predicted).

In accordance with an embodiment, an audio encoder is configured to adjust one or more quantization parameters through adaptation 342 to adapt the provision of one or more encoded audio signals of the encoded audio content 320.

In accordance with an embodiment, encoding parameter adaptation 340 may be performed to disable or enable encoding 310 and/or enable and disable a collaborative coding tool that is used by coding unit 314, for example. For example, this is done by adapting 344 coding parameters. In accordance with an embodiment, the adaptation 344 of the coding parameters may depend on the same criteria as the adaptation 342 of the quantization parameters. Thus, in accordance with an embodiment, the audio encoder 300 is configured to inhibit encoding 310 of a specific one of the signals to be coded, such as a difference signal, when the contributions of a particular directional loudness map 142 of the specified one of the signals to be coded (or, for example, when the contributions of the map 142 directional loudness of a pair of signals to be encoded or a group of three or more signals to be encoded) into a common directional loudness map are below the threshold value. Thus, audio encoder 300 is configured to efficiently encode 310 only relevant information.

In accordance with an embodiment, the joint coding tool of the coding unit 314, for example, is configured to jointly encode two or more of the input audio signals 112 or signals 110 derived from them, for example, to decide on/off M/S (center/side signal ). The encoding parameter adaptation 344 may be performed such that the joint encoding tool is enabled or disabled depending on one or more directional loudness maps 142 that represent loudness information associated with multiple different directions of one or more signals 112/110 to be coded. Alternatively or additionally, audio encoder 300 may be configured to define one or more joint coding tool parameters as coding parameters depending on one or more directional loudness maps 142 . Thus, with the adaptation 344 of the coding parameters, for example, it is possible to control the smoothing of the frequency-dependent prediction coefficients, for example, to set the parameters of the "intense stereo" joint coding tool.

In accordance with an embodiment, the quantization parameters and/or coding parameters may be considered as control parameters that may control the provision of one or more encoded audio signals 320. Thus, the audio encoder 300 is configured to determine or evaluate the effect of variation of one or more control parameters on the map 142 directional loudness of one or more encoded signals 320; and adjusting one or more control parameters depending on the impact determination or evaluation. This may be implemented by a feedback and/or feedforward loop 322 as described above.

In FIG. 13 shows an audio encoder 300 for encoding 310 audio input content 112 comprising one or more audio input signals 112 ₁ , 112 ₂ . Preferably, as shown in FIG. 13, the audio input content 112 comprises a plurality of audio input signals, such as two or more audio input signals 112 ₁ , 112 ₂ . In accordance with an embodiment, the input audio content 112 may comprise signals in the time domain or signals in the spectral domain. In some cases, input audio content signals 112 may be processed 330 by audio encoder 300 to determine candidate signals, such as first candidate signal 110 ₁ and/or second candidate signal 110 ₂ . Processing 330 may comprise, for example, a time domain to spectral domain conversion if the input audio signals 112 are time domain signals.

The audio encoder 300 is configured to select 350 signals to be jointly encoded 310 from a plurality of possible signals 110 or from a plurality of pairs of possible signals 110 depending on the directional volume maps 142. Directional volume maps 142 represent loudness information associated with a variety of different directions, such as pan directions, possible signals 110 or pairs of possible signals 110, and/or predetermined directions.

According to an embodiment, directional loudness maps 142 may be computed by determining loudness information 100 as described herein. Thus, the loudness information determination 100 may be implemented as described with respect to the audio encoder 300 described in FIG. 11 or FIG. 12. Directional volume maps 142 are based on candidate signals 110, with candidate signals representing input audio signals of input audio content 112, unless processing 330 is applied by audio encoder 300.

If the input audio content 112 contains only one input audio signal, that signal is selected by selecting 350 as to be encoded by the audio encoder 300, for example using entropy coding, to provide one encoded audio signal as the encoded audio content 320. In this case, for example, the audio encoder is configured to disabling joint coding 310 and switching to coding only one signal.

If the input audio content 112 contains two input audio signals 112₁ and 112₂, which can be described as X₁ and X₂, both signals 112₁ and 112₂ are selected 350 by the audio encoder 300 for joint encoding 310 to provide one or more encoded signals in the encoded audio content 320. Thus, the encoded audio content 320 optionally comprises a center signal and a side signal, or a downmix signal and a difference signal, or only one of these four signals.

If input audio content 112 contains three or more input audio signals, signal selection 350 is based on directional loudness maps 142 of possible signals 110. According to an embodiment, audio encoder 300 is configured to use signal selection 350 to select one pair of signals from a plurality of possible signals 110, for which, according to the directional volume map 142, efficient audio coding and high quality audio output can be realized. Alternatively, or additionally, it is also possible that the signal selection 350 selects three or more signals from the possible signals 110 to be jointly encoded 310. Alternatively, or additionally, the audio encoder 300 uses the signal selection 350 to select more than one pair of signals or a group of signals to be co-coded 310. The selection 350 of the signals 352 to be coded may depend on the contributions of the individual directional loudness maps 142 of combining two or more signals into a common directional loudness map. According to an embodiment, a common directional loudness map is associated with multiple selected audio input signals, or with each signal of input audio content 112. How this signal selection 350 can be performed by audio encoder 300 is described by way of example in FIG. 14 for input audio content 112 containing three input audio signals.

Thus, audio encoder 300 is configured to provide one or more encoded, such as quantized and then lossless encoded audio signals, such as encoded spectral domain representations, based on two or more input audio signals 112 ₁ , 112 ₂ , or based on two or more received of these, signals 110 ₁ , 110 ₂ using joint encoding 310 of two or more signals 352 to be jointly encoded.

According to an embodiment, the audio encoder 300, for example, is configured to determine separate directional loudness maps 142 of two or more candidate signals and compare separate directional loudness maps 142 of the two or more possible signals. Additionally, the audio encoder, for example, is configured to select two or more of the possible signals for joint coding depending on the result of the comparison, for example, in such a way that possible signals are selected for joint coding, the individual loudness maps of which contain the maximum similarity or similarity that is higher than the threshold similarity values. By means of this optimized selection, very efficient coding can be realized since the high similarity of the signals to be jointly encoded can lead to coding using only a small number of bits. This means, for example, that the downmix signal or the difference signal of a selected pair of candidate signals can be efficiently jointly encoded.

In FIG. 14 shows an embodiment of signal selection 350 that can be performed by any audio encoder 300 described here, such as audio encoder 300 in FIG. 13. An audio encoder may be configured to use signal selection 350 as shown in FIG. 14, or applying the described signal selection 350 to more than three input audio signals to select signals to be jointly encoded from a plurality of possible signals or from a plurality of pairs of possible signals depending on the contributions of the individual directional loudness maps of the possible signals to the overall directional loudness map 142b, or depending on the contributions of the directional loudness maps 142a ₁ -142a ₃ of pairs of possible signals to the overall directional loudness map 142b, as shown in FIG. 14.

In accordance with FIG. 14, for each possible pair of signals, a directional loudness map 142a ₁ -142a ₃ , for example, is obtained by signal selection 350, and a general directional loudness map 142b associated with all three input audio content signals is obtained by signal selector 350. Directional loudness maps 142, such as signal pair directional loudness maps 142a ₁ -142a ₃ and overall directional loudness map 142b, may be received from an audio analysis module, or may be determined by an audio encoder and provided to a signal selector 350 . According to an embodiment, the overall directional volume map 142b may represent the overall audio scene, such as represented by the input audio content, such as before being processed by the audio encoder. According to an embodiment, the overall directional loudness map 142b represents loudness information associated with different directions, for example, of the audio components of a presented or to be presented audio scene, for example, after rendering at the decoder side via audio input signals 112 ₁ -112 ₃ . A general directional loudness map, for example, is represented as DirLoudMap(1, 2, 3). In accordance with an embodiment, the overall directional loudness map 142b is determined by the audio encoder using downmix of the input audio signals 112 ₁ -112 ₃ or using binauralization of the input audio signals 112 ₁ -112 ₃ .

In FIG. 14 shows signal selection 350 for three channels CH1-CH3 respectively associated with first audio input 112 ₁ , second audio input 112 ₂ , or third audio input 112 ₃ . The first directional volume map 142a ₁ , such as DirLoudMap (1, 2), is based on the first audio input 112 ₁ and the second audio input 112 ₂ , the second directional volume map 142a ₂ , such as DirLoudMap (2, 3), is based on the second input audio signal 112 ₂ and a third audio input signal 112 _{3 ,} and a third directional volume map 142a ₃ , such as DirLoudMap(1, 3), is based on the first audio input signal 112 ₁ and the third audio input signal 112 ₃ .

In accordance with an embodiment, each directional loudness map 142 represents loudness information associated with different directions. The different directions are indicated in Fig. 14 with a line between L and R, where L is associated with audio component panning to the left side, and where R is associated with audio component panning to the right side. Thus, the various directions comprise a left side and a right side and directions or angles between the left and right side. The directional volume maps 142 shown in FIG. 14 are presented as charts, but alternatively it is also possible that directional loudness maps 142 can be represented by directional loudness histograms as shown in FIG. 5 or a matrix as shown in FIG. 10a-10c. It is clear that only the information associated with directional volume maps 142 is relevant to signal selection 350, and that the graphical representation is only intended to improve understanding.

In accordance with an embodiment, signal selection 350 is performed such that the contribution of pairs of possible signals to the overall directional loudness map 142b is determined. The relationship between the overall directional loudness map 142b and the directional loudness maps 142a ₁ -142a ₃ of the pairs of possible signals can be described by the formula

DirLoudMap(1,2,3) = a*DirLoudMap(1,2,3) + b*DirLoudMap(2,3) + c*DirLoudMap(1,3).

The contribution determined by the audio encoder through the use of signal selection may be represented by coefficients a, b and c.

According to an embodiment, the audio encoder is configured to select for joint encoding one or more pairs of possible signals 112 ₁ -112 ₃ having the highest contribution to the overall directional loudness map 142b. This means, for example, that by signal selection 350, a pair of possible signals is selected that is associated with the highest coefficient of the coefficients a, b, and c.

Alternatively, the audio encoder is configured to select for joint encoding one or more pairs of candidate signals 112 ₁ -112 ₃ having a contribution to the overall directional loudness map 142b that is greater than a predetermined threshold. This means, for example, that a predetermined threshold has been selected, and that each coefficient a, b, c is compared with a predetermined threshold to select each pair of signals associated with a coefficient that is greater than the predetermined threshold.

According to an embodiment, the contributions may range from 0% to 100%, which means, for example, for the coefficients a, b, and c, the range from 0 to 1. The 100% contribution, for example, is associated with a directional loudness map 142a that is exactly is equal to the overall directional volume map 142b. According to an embodiment, the predetermined threshold depends on how many input audio signals are included in the input audio content. According to an embodiment, the predetermined threshold may be defined as a contribution of at least 35%, or at least 50%, or at least 60%, or at least 75%.

According to an embodiment, the given threshold depends on how many signals are to be selected by signal selection 350 for joint coding. For example, if at least two pairs of signals are to be selected, then the two pairs of signals that are associated with the directional loudness maps 142a having the highest contribution to the overall directional loudness map 142b can be selected. This means, for example, that the 350 pair of signals with the highest contribution and the second highest contribution are selected.

It is advantageous to base the audio encoder's selection of signals to be encoded on directional loudness maps 142, since comparison of directional loudness maps can indicate the quality of the listener's perception of the encoded audio signals. According to an embodiment, signal selection 350 is performed by the audio encoder such that a pair of signals or pairs of signals is selected for which their directional loudness map 142a is most similar to the overall directional loudness map 142b. This can lead to a similar perception of the selected pair and pairs of possible signals compared to the perception of all input audio signals. Thus, the quality of the encoded audio content can be improved.

In FIG. 15 shows an embodiment of an audio encoder 300 for encoding 310 audio input content 112 comprising one or more audio input signals. Preferably, two or more input audio signals are encoded 310 by audio encoder 300. Audio encoder 300 is configured to provide one or more encoded audio signals 320 based on two or more input audio signals 112, or based on two or more signals 110 derived from them. Signal 110 may be obtained from input audio signal 112 through optional processing 330. According to an embodiment, optional processing 330 may include features and/or functionality as described in relation to other audio encoders 300 described here. .

Audio encoder 300 is configured to determine 100 an overall directional loudness map based on audio input signals 112 and/or determine 100 one or more individual directional loudness maps 142 associated with individual audio input signals 112. The overall directional loudness map can be represented as L(m,Ψ _0,j ), and individual directional loudness maps can be represented as L _i (m,Ψ _0,j ). In accordance with an embodiment, the overall directional loudness map may represent a target directional loudness map of a scene. In other words, a general directional loudness map may be associated with a desired directional loudness map for combining encoded audio signals. Additionally or alternatively, it is possible that 100 directional loudness maps L _i (m,Ψ _0,j ) of signal pairs or groups of three or more signals can be determined by audio encoder 300.

Audio encoder 300 is configured to encode 310 a general directional loudness map 142 and/or one or more individual directional loudness maps 142 and/or one or more directional loudness maps of signal pairs or groups of three or more input audio signals 112 as ancillary information. Thus, the encoded audio content 320 comprises encoded audio signals and encoded directional loudness maps. According to an embodiment, encoding 310 may be dependent on one or more directional loudness maps 142, whereby it is advantageous to also encode these directional loudness maps 142 to enable high quality decoding of encoded audio content 320. With directional loudness maps 142 as coded side information, initially the intended quality characteristic (eg, to be achieved by encoding 310 and/or by an audio decoder) is provided by the encoded audio content 320.

According to an embodiment, audio encoder 300 is configured to determine 100 an overall directional loudness map L(m,Ψ _0,j ) based on input audio signals 112, whereby the overall directional loudness map represents loudness information associated with different directions, for example, audio components of the audio scene represented by the input audio signals 112. Alternatively, the overall directional loudness map L(m,Ψ _0,j ) represents the loudness information associated with different directions, for example, the audio components of the audio scene, which will be presented, for example, after rendering on side of the decoder with the input audio signals. Loudness information determination 100 can optionally be performed by audio encoder 300 in conjunction with knowledge or ancillary information regarding speaker positions and/or knowledge or ancillary information describing the positions of audio objects in input audio signals 112.

According to an embodiment, loudness information determination 100 may be implemented as described with the other audio encoders 300 described herein.

Audio encoder 300, for example, is configured to encode 310 a general directional loudness map L(m,Ψ _0,j ) in the form of a set of values, such as scalars, associated with different directions. In accordance with an embodiment, the values are further associated with a plurality of frequency bands. Each value or values in the discrete directions of the overall directional loudness map may be encoded. This means, for example, that each color matrix value, as shown in FIG. 10a-10c, or values of different histogram bins as shown in FIG. 5, or directional loudness map curve values as shown in FIG. 14 are encoded for discrete directions.

Alternatively, audio encoder 300, for example, is configured to encode a general directional loudness map L(m,Ψ _0,j ) using a center position value and gradient information. The center position value describes, for example, the angle or direction in which the maximum of the overall directional loudness map is located for a given frequency band or frequency interval, or for a plurality of frequency intervals or frequency bands. The gradient information represents, for example, one or more scalars describing the gradients of the values of the overall directional loudness map in the angular direction. The gradient information scalars are, for example, the overall directional loudness map values for directions adjacent to the center position value. The center position value may represent a scalar value of the loudness information and/or a scalar value of the direction corresponding to the loudness value.

Alternatively, the audio encoder, for example, is configured to encode the overall directional loudness map L(m,Ψ _0,j ) as a polynomial representation or as a spline representation.

In accordance with an embodiment, the coding capabilities 310 described above for the overall directional loudness map L(m,Ψ _0,j ) can also be applied to individual directional loudness maps L _i (m,Ψ _0,j ) and/or directional loudness maps associated with pairs of signals or groups of three or more signals.

In accordance with an embodiment, audio encoder 300 is configured to encode a single downmix signal derived from a plurality of audio input signals 112 and a common directional loudness map L(m,Ψ _0,j ). Optionally, also the contribution of the directional loudness map associated with the downmixed signal is, for example, encoded with the common directional loudness map as side information.

Alternatively, the audio encoder 300 is, for example, configured to encode 310 a plurality of signals, such as input audio signals 112 or signals 110 derived therefrom, and encode 310 individual loudness maps L _i (m,Ψ _0,j ) of the plurality of signals 112/110 to be encoded 310 (for example, individual signals, pairs of signals, or groups of three or more signals). The encoded set of signals and the encoded individual directional loudness maps are, for example, transmitted to an encoded audio representation 320 or inserted into an encoded audio representation 320.

In accordance with an alternative embodiment, audio encoder 300 is configured to encode 310 a general directional loudness map L(m,Ψ _0,j ), a plurality of signals, such as input audio signals 112 or signals 110 derived from them, and parameters describing contributions, such as relative signal contributions that are encoded with a common directional loudness map. According to an embodiment, the parameters may be represented by parameters a, b, and c, as described in FIG. 14. Thus, for example, the audio encoder 300 is configured to encode 310 all of the information on which the encoding 310 is based to provide, for example, information for high quality decoding of the provided encoded audio content 320.

According to an embodiment, an audio encoder may comprise or combine individual features and/or functionality as described in relation to one or more audio encoders 300 described in FIG. 11-15.

In FIG. 16 shows an embodiment of an audio decoder 400 for decoding 410 encoded audio content 420. Encoded audio content 420 may include encoded representations 422 of one or more audio signals and encoded directional loudness map information 424.

Audio decoder 400 is configured to receive an encoded representation 422 of one or more audio signals and provide a decoded representation 412 of one or more audio signals. In addition, the audio decoder 400 is configured to receive encoded directional loudness map information 424 and decode 410 the encoded directional loudness map information 424 to obtain one or more decoded directional loudness maps 414. The decoded directional loudness maps 414 may contain features and/or functionality as described with respect to the directional loudness maps 142 described above.

According to an embodiment, decoding 410 may be performed by audio decoder 400 using AAC-like decoding, or using entropy-coded spectral value decoding, or using entropy-coded loudness decoding.

Audio decoder 400 is configured to reconstruct 430 an audio scene using a decoded representation 412 of one or more audio signals and using one or more directional volume maps 414 . Based on the recreation 430 by the audio decoder 400, the decoded audio content 432, such as a multi-channel representation, can be determined.

According to an embodiment, the directional loudness map 414 may represent a target directional loudness map to be achieved by the decoded audio content 432. Thus, with the directional loudness map 414, the recreation of the audio scene 430 can be optimized and result in a high quality listener experience of the decoded audio content 432. This is based on the idea that the directional loudness map 414 can indicate a desired listening experience.

In FIG. 17 shows the encoder 400 shown in FIG. 16 with the optional decoding parameter adaptation feature 440. According to an embodiment, the decoded audio content may comprise output signals 432 that represent, for example, signals in the time domain or signals in the spectral domain. Audio decoder 400, for example, is configured to obtain output signals 432 such that one or more directional loudness maps associated with output signals 432 are approximately equal to or equal to one or more target directional loudness maps. One or more target directional loudness maps are based on one or more decoded directional loudness maps 414, or equal to one or more decoded directional loudness maps 414. In some cases, audio decoder 400 is configured to use suitable scaling or combining one or more decoded directional loudness maps 414 to determine the target directional loudness map or maps.

According to an embodiment, one or more directional loudness maps associated with output signals 432 may be determined by audio decoder 400. Audio decoder 400 includes, for example, an audio parsing module for determining one or more directional loudness maps associated with output signals 432, or is configured capable of receiving from the external audio analysis module 100 one or more directional volume maps associated with the output signals 432.

In accordance with an embodiment, audio decoder 400 is configured to compare one or more directional loudness maps associated with output signals 432 and decoded directional loudness maps 414; or comparing one or more directional loudness maps associated with output signals 432 with a directional loudness map obtained from decoded directional loudness map 414 and adapting 440 decoding parameters or recreating 430 based on this comparison. According to an embodiment, audio decoder 400 is configured to adapt 440 decoding parameters or adapt reconstruction 430 such that the deviation between one or more directional loudness maps associated with output signals 432 and one or more target directional loudness maps is below a predetermined threshold. This may be a feedback loop, whereby decoding 410 and/or reconstruction 430 are adapted such that one or more directional loudness maps associated with output signals 432 approximate one or more target directional loudness maps by at least 75%. , or at least 80%, or at least 85%, or at least 90%, or at least 95%.

According to an embodiment, the audio decoder 400 is configured to receive the encoded downmix signal as an encoded representation 422 of one or more audio signals and the overall directional loudness map as encoded directional loudness map information 424. An encoded downmix signal, for example, is obtained based on a plurality of input audio signals. Alternatively, the audio decoder 400 is configured to receive a plurality of encoded audio signals as an encoded representation 422 of one or more audio signals and individual directional loudness maps of the plurality of encoded signals as encoded directional loudness map information 424. The encoded audio signal is, for example, input audio signals encoded by an encoder or signals derived from input audio signals encoded by an encoder. Alternatively, the audio decoder 400 is configured to receive the overall directional loudness map as encoded directional loudness map information 424, a plurality of encoded audio signals as an encoded representation 422 of one or more audio signals, and further parameters describing the contributions of the encoded audio signals to the overall directional loudness map. Thus, the encoded audio content 420 may further comprise parameters, and the audio decoder 400 may be configured to use these parameters to improve the adaptation 440 of the decoding parameters and/or improve the reconstruction 430 of the audio scene.

Audio decoder 400 is configured to provide output signals 432 based on one of the previously mentioned encoded audio content 420.

In FIG. 18 shows an embodiment of a format converter 500 for format conversion 510 of audio content 520 that represents an audio scene. Format converter 500 receives, for example, audio content 520 in a first format, and converts 510 audio content 520 to audio content 530 in a second format. In other words, the format converter 500 is configured to provide a presentation 530 of the audio content in the second format based on the presentation 520 of the audio content in the first format. According to an embodiment, audio content 520 and/or audio content 530 may represent a spatial audio scene.

The first format may, for example, comprise a first number of channels or input audio signals and side information or spatial side information adapted to the first number of channels or input audio signals. The second format, for example, may contain a second number of channels or audio outputs, which may be different from the first number of channels or audio inputs, and side information or spatial side information adapted to the second number of channels or audio outputs. The audio content 520 in the first format comprises, for example, one or more audio signals, one or more downmix signals, one or more difference signals, one or more center signals, one or more side signals, and/or one or more other signals.

The format converter 500 is configured to adjust 540 the complexity of the format conversion 510 depending on the contributions of the first format input audio signals to the overall audio scene directional volume map 142 . The audio content 520 contains, for example, the input audio signals of the first format. The contributions may directly represent the contributions of the first format audio inputs to the overall audio scene directional loudness map 142, or may be the contributions of individual first format audio input directional loudness maps to the overall directional loudness map 142, or may be the contributions of the first format directional loudness maps of pairs of audio input signals format into a common map 142 directional volume. In accordance with an embodiment, the contributions may be calculated by the format converter 500 as described in FIG. 13 or FIG. 14. According to an embodiment, the overall directional loudness map 142, for example, can be described using the first format side information received by the format converter 500. Alternatively, the format converter 500 is configured to determine an overall directional loudness map 142 based on the input audio signals of the audio content 520. In some cases, the format converter 500 includes an audio parsing module, as described with respect to FIG. 1-4b to calculate the overall directional loudness map 142, or the format converter 500 is configured to receive the overall directional loudness map 142 from an external audio analysis module as described with respect to FIG. 1-4b.

The audio content 520 in the first format may contain information about directional loudness maps of the input audio signals in the first format. Based on the information about the directional loudness maps, the format converter 500, for example, is configured to obtain a common directional loudness map 142 and/or one or more directional loudness maps. One or more directional loudness maps may be directional loudness maps of each input audio signal in the first format and/or directional loudness maps of groups or pairs of signals in the first format. Format converter 500, for example, is configured to derive a general directional loudness map 142 from one or more directional loudness maps or directional loudness map information.

Complexity adjustment 540, for example, is performed to check whether it is possible to skip one or more first format audio inputs whose contribution to the directional loudness map is below a threshold. In other words, the format converter 500, for example, is configured to calculate or evaluate the contribution of a certain input audio signal to the overall audio scene directional loudness map 142 and decide whether to consider the certain input audio signal to the format transformation 510, depending on the calculation or estimation of the contribution. The calculated or estimated contribution is, for example, compared by the format converter 500 against a predetermined absolute or relative threshold value.

The contributions of the first format audio input signals to the overall directional loudness map 142 may indicate the relevance of the corresponding input audio signal to the perceptual quality of the audio content 530 in the second format. Thus, for example, only audio signals in the first high-relevance format are subjected to format conversion 510 . This can result in high quality audio content 530 in the second format.

In FIG. 19 shows an audio decoder 400 for decoding 410 encoded audio content 420. Audio decoder 400 is configured to receive an encoded representation 420 of one or more audio signals and provide a decoded representation 412 of one or more audio signals. Decoding 410 uses, for example, AAC-like decoding or decoding of entropy-coded spectral values. Audio decoder 400 is configured to reconstruct 430 an audio scene using a decoded representation 412 of one or more audio signals. The audio decoder 400 is configured to adjust 440 the complexity of the decoding depending on the contributions of the encoded signals to the overall directional loudness map 142 of the decoded audio scene 434.

Decoding complexity adjustment 440 may be performed by audio decoder 400 in a manner similar to complexity adjustment 540 of format converter 500 in FIG. 18.

According to an embodiment, audio decoder 400 is configured to receive encoded directional loudness map information, such as extracted from encoded audio content 420. The encoded directional loudness map information can be decoded 410 by audio decoder 400 to determine decoded directional loudness information 414. Based on the decoded directional loudness information 414, an overall directional loudness map of one or more encoded audio content audio signals 420 and/or one or more individual directional loudness maps of one or more encoded audio content audio signals 420 can be derived. An overall directional loudness map of one or more encoded audio content audio signals 420, for example, are derived from one or more separate directional loudness maps.

The overall directional loudness map 142 of the decoded audio scene 434 may be computed by directional loudness map determination 100, which may optionally be performed by audio decoder 400. In accordance with an embodiment, audio decoder 400 includes an audio data analysis module as described with respect to FIG. 1-4b to perform the directional loudness map determination 100, or the audio decoder 400 may send the decoded audio scene 434 to an external audio analysis module and receive a general directional loudness map 142 of the decoded audio scene 434 from the external audio analysis module.

In accordance with an embodiment, the audio decoder 400 is configured to calculate or estimate the contribution of a certain encoded signal to the overall directional loudness map 142 of the decoded audio scene and decide whether to decode 410 the certain encoded signal depending on the calculation or estimate of the contribution. Thus, for example, the overall directional loudness map of one or more audio signals of the encoded audio content 420 may be compared to the overall directional loudness map of the decoded audio scene 434. The determination of the contributions may be performed as described above (eg, as described with respect to FIG. 13 or FIG. 14 ) or similar.

Alternatively, the audio decoder 400 is configured to calculate or estimate the contribution of a certain encoded signal to the decoded overall directional loudness map 414 of the encoded audio scene and decide whether to decode 410 the certain encoded signal depending on the calculation or estimate of the contribution.

Complexity adjustment 440, for example, is performed to check if it is possible to skip one or more encoded representations of one or more input audio signals whose contribution to the directional loudness map is below a threshold.

Additionally or alternatively, decoding complexity adjustment 440 may be configured to adapt decoding parameters based on contributions.

Additionally or alternatively, the decoding complexity adjustment 440 can be configured to compare the decoded directional loudness maps 414 with the overall directional loudness map of the decoded audio scene 434 (e.g., the overall directional loudness map of the decoded audio scene 434 is the target directional loudness map) to adapt the decoding parameters.

In FIG. 20 shows an embodiment of a rendering module 600 (listening transforms). The renderer 600 is, for example, a binaural renderer, or a soundbar renderer, or a loudspeaker renderer. With the help of the renderer 600, the audio content 620 is rendered to obtain the listener-transformed audio content 630. The audio content 620 may contain one or more input audio signals 622. The renderer 600 uses, for example, one or more input audio signals 622 to recreate 640 the audio scene. Preferably, the rendering 640 performed by the renderer 600 is based on two or more input audio signals 622. According to an embodiment, the input audio signal 622 may comprise one or more audio signals, one or more downmix signals, one or more difference signals, other audio signals, and /or additional information.

In accordance with an embodiment, to recreate 640 an audio scene, renderer 600 is configured to analyze one or more input audio signals 622 to optimize rendering to obtain the desired audio scene. Thus, for example, the renderer 600 is configured to modify the spatial arrangement of the audio objects of the audio content 620. This means, for example, that the renderer 600 can recreate 640 a new audio scene. The new audio scene contains, for example, rearranged audio objects compared to the original audio content audio scene 620. This means, for example, that a guitarist and/or singer and/or other audio objects are placed in the new audio scene at different spatial locations compared to the original audio scene.

Additionally or alternatively, the number of audio channels or the ratio between audio channels is converted by the renderer 600 for listening. Thus, for example, the renderer 600 may convert audio content 620 containing a multi-channel signal to, for example, a two-channel signal. For example, this is desirable if only two speakers are available to present audio content 620.

In accordance with an embodiment, rendering is performed by renderer 600 such that the new audio scene exhibits minor deviations from the original audio scene.

The renderer 600 is configured to adjust 650 the rendering complexity depending on the contributions of the input audio signals 622 to the overall directional loudness map 142 of the listener-transformed audio scene 642. In accordance with an embodiment, the listener-transformed audio scene 642 may be the new audio scene described above. According to an embodiment, the audio content 620 may include a general directional loudness map 142 as ancillary information. This general directional loudness map 142, received as ancillary information by the renderer 600, may indicate a desired audio scene for the listener-transformed audio content 630. from block 640 recreation. According to an embodiment, the renderer 600 may comprise a directional volume map definition 100, or receive a generic directional volume map 142 of an external directional volume map definition 100. According to an embodiment, directional loudness map determination 100 may be performed by an audio analysis module as described above.

According to an embodiment, rendering complexity adjustment 650 is, for example, performed by skipping one or more audio input signals 622. Audio input signals 622 to be skipped, for example, are signals whose contribution to directional loudness map 142 is below a threshold. Thus, only the relevant input audio signals are converted by the renderer 600 for listening.

In accordance with an embodiment, the renderer 600 is configured to calculate or evaluate the contribution of a particular input audio signal 622 to the overall directional loudness map 142 of an audio scene, such as a listening-transformed audio scene 642. In addition, the renderer 600 is configured to decide whether to consider a specific input audio signal when rendering, depending on the calculation or evaluation of the contribution. Thus, for example, the calculated or estimated contribution is compared to a predetermined absolute or relative threshold value.

для различных

(j

[1; J]); «направленных сигналов») на основе одного или более представлений (например,

, для i = {L; R}; или

) в спектральной области (например, в частотно-временной области) двух или более входных аудиосигналов (x_L, x_R, x_i). Значения одного или более представлений в спектральной области (например)

взвешиваются 1200 в зависимости от различных направлений (например, направления панорамирования)

(например, представленных весовыми коэффициентами)

звуковых компонентов (например, спектральных интервалов или спектральных полос) (например, мелодических тонов инструментов или певца) в двух или более входных аудиосигналах, чтобы получить множество взвешенных представлений в спектральной области (

для разных

(j

[1; J]); «направленных сигналов»). Кроме того, способ содержит получение 1300 информации о громкости (например, L(m,

) для множества различных

; например, «карты направленной громкости»), ассоциированной с разными направлениями (например, направлением панорамирования)

на основе множества взвешенных представлений в спектральной области (

для различных

(j

[1; J]); «направленных сигналов») в качестве результата анализа.In FIG. 21 shows a method 1000 for analyzing an audio signal. The method comprises obtaining 1100 a plurality of weighted spectral domain (e.g., time-frequency domain) representations (

for various

(j

[1; J]); "directional signals") based on one or more representations (for example,

, for i = {L; R}; or

) in the spectral domain (eg, time-frequency domain) of two or more input audio signals (x _L , x _R , x _i ). The values of one or more representations in the spectral domain (for example)

weighted 1200 depending on different directions (like pan direction)

(e.g. represented by weights)

audio components (e.g. spectral intervals or spectral bands) (e.g. melodic tones of instruments or a singer) in two or more input audio signals to obtain a set of weighted representations in the spectral domain (

for different

(j

[1; J]); "directed signals"). In addition, the method comprises obtaining 1300 loudness information (for example, L(m,

) for many different

; e.g. "directional volume maps") associated with different directions (e.g. pan direction)

based on a set of weighted representations in the spectral domain (

for various

(j

[1; J]); "directional signals") as a result of the analysis.

); карты направленной громкости; значения объединенной громкости), ассоциированной с различным направлениями (например, панорамирования) (например)

, на основе первого множества из двух или более входных аудиосигналов (x_R, x_L, x_i) и сравнение 2200 первой информация о громкости (L₁(m,

)) со второй (например, соответствующей) информацией о громкости (L₂(m,

)); эталонной информацией о громкости; эталонной картой направленной громкости; эталонным значением объединенной громкости), ассоциированной с различными направлениями панорамирования (например)

и с множеством из двух или более эталонных аудиосигналов (x_2,R, x_2,L, x_2,i), для получения 2300 информации о сходстве (например, «выходной переменной модели» (MOV)) описывающей сходство между первым множеством из двух или более входных аудиосигналов (x_R, x_L, x_i) и множеством из двух или более эталонных аудиосигналов (x_2,R, x_2,L, x_2,i) (или представляющей качество первого множества из двух или более входных аудиосигналов при сравнении с множеством из двух или более эталонных аудиосигналов).In FIG. 22 shows a method 2000 for evaluating the similarity of audio signals. The method includes obtaining 2100 first information about the loudness (L ₁ (m,

); directional volume maps; combined volume values) associated with different directions (e.g. panning) (e.g.)

, based on a first set of two or more input audio signals (x _R , x _L , x _i ) and comparing 2200 the first loudness information (L ₁ (m,

)) with second (for example, corresponding) loudness information (L ₂ (m,

)); reference loudness information; directional loudness reference card; reference value of the combined loudness) associated with different pan directions (for example)

and with a set of two or more reference audio signals (x _2,R , x _2,L , x _2,i ), to obtain 2300 similarity information (eg, a "model output variable" (MOV)) describing the similarity between the first set of two or more input audio signals (x _R , x _L , x _i ) and a set of two or more reference audio signals (x _2,R , x _2,L , x _2,i ) (or representing the quality of the first set of two or more input audio signals when compared to a set of two or more reference audio signals).

In FIG. 23 shows a method 3000 for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The method comprises providing 3100 one or more encoded (e.g., quantized and then lossless coded) audio signals (e.g., encoded spectral domain representations) based on one or more input audio signals (e.g., a left channel signal and a right channel signal), or one or more signals derived from them (for example, the center signal, or the downmix signal and the side signal, or the difference signal). Additionally, method 3000 comprises adapting 3200 to provide one or more encoded audio signals in dependence on one or more directional loudness maps that represent loudness information associated with a plurality of different directions (e.g., pan direction) of one or more signals to be encoded (e.g., depending on from the contributions of individual directional loudness maps of one or more signals to be quantized to a common directional loudness map, eg, associated with multiple audio inputs (eg, each signal from one or more audio inputs)).

In FIG. 24 shows a method 4000 for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The method comprises providing 4100 one or more encoded (e.g., quantized and then lossless coded) audio signals (e.g., encoded spectral domain representations) based on two or more input audio signals (e.g., a left channel signal and a right channel signal), or based on two or more signals derived therefrom using joint encoding of the two or more signals to be jointly encoded (eg, using a center signal or a downmix signal and a side signal or a difference signal). In addition, method 4000 comprises selecting 4200 signals to be jointly encoded from a plurality of possible signals or from a plurality of pairs of possible signals (e.g., from two or more input audio signals or from two or more signals derived from them) depending on directional loudness maps, which represent loudness information associated with a plurality of different directions (e.g., pan direction) of possible signals or pairs of possible signals (e.g., depending on the contributions of individual candidate signal directional loudness maps to an overall directional loudness map, e.g., associated with multiple audio input signals ( for example, with each signal from one or more input audio signals), or depending on the contributions of the directional loudness maps of pairs of possible signals to the overall directional loudness map).

In FIG. 25 shows a method 5000 for encoding input audio content containing one or more input audio signals (preferably a plurality of input audio signals). The method comprises providing 5100 one or more encoded (e.g., quantized and then lossless encoded) audio signals (e.g., encoded spectral domain representations) based on two or more input audio signals (e.g., a left channel signal and a right channel signal), or based on two or more signals received from them. Additionally, the method 5000 comprises determining 5200 an overall directional loudness map (e.g., a target scene directional loudness map) based on the input audio signals and/or determining one or more separate directional loudness maps associated with the individual input audio signals, and encoding 5300 the overall directional loudness map and/ or one or more separate directional loudness maps as ancillary information.

In FIG. 26 shows a method 6000 for decoding encoded audio content, comprising receiving 6100 an encoded representation of one or more audio signals and providing 6200 a decoded representation of one or more audio signals (eg, using AAC-like decoding or using decoding of entropy-coded spectral values). The method 6000 comprises receiving 6300 the encoded directional loudness map information and decoding 6400 the encoded directional loudness map information to obtain 6500 one or more (decoded) directional loudness maps. Additionally, method 6000 comprises restoring 6600 an audio scene using a decoded representation of one or more audio signals and using one or more directional loudness maps.

In FIG. 27 shows a method 7000 for converting 7100 an audio content format that represents an audio scene (e.g., a spatial audio scene) from a first format to a second format (wherein the first format, for example, may contain a first number of channels or input audio signals and ancillary information or spatial ancillary information adapted to the first number of channels or input audio signals, and the second format, for example, may contain a second number of channels or output audio signals, which may differ from the first number of channels or input audio signals, and ancillary information or spatial assistance information adapted to the second number of channels or audio output signals ). The method 7000 comprises providing a representation of the audio content in a second format based on the representation of the audio content in the first format, and adjusting 7200 the complexity of the format conversion (e.g., by passing one or more first format audio inputs whose contribution to the directional loudness map is below a threshold during the format conversion process) depending on the contributions of the input audio signals of the first format (for example, one or more audio signals, one or more downmix signals, one or more difference signals, etc.) to the overall directional loudness map of the audio scene (where the overall directional loudness map, for example, can be described by the first format side information received by the format converter).

In FIG. 28 shows a method 8000 for decoding encoded audio content, comprising receiving 8100 an encoded representation of one or more audio signals and providing 8200 a decoded representation of one or more audio signals (eg, using AAC-like decoding or using decoding of entropy-coded spectral values). Method 8000 comprises reconstructing 8300 an audio scene using a decoded representation of one or more audio signals. Additionally, the method 8000 comprises adjusting 8400 the complexity of the decoding depending on the contributions of the encoded signals (eg, one or more audio signals, one or more downmix signals, one or more difference signals, etc.) to the overall directional loudness map of the decoded audio scene.

In FIG. 29 shows a method 9000 for rendering audio content (e.g., for upmixing audio content represented using a first number of input audio channels and ancillary information describing desired spatial characteristics, such as placement of audio objects or relationship between audio channels, into a representation containing a number of channels that is greater than than the first number of input audio channels) comprising reconstruction 9100 of an audio scene based on one or more input audio signals (or based on two or more input audio signals). Method 9000 comprises adjusting 9200 rendering complexity (e.g., by skipping one or more input audio signals whose contributions to the directional loudness map are below a threshold during rendering) depending on input audio signal contributions (e.g., one or more audio signals, one or more mixed with downsampling of signals, one or more difference signals, etc.) into a general directional loudness map of the audio scene converted for listening (and the general directional loudness map, for example, can be described by auxiliary information received by the renderer).

Remarks

Various embodiments of the invention and aspects will be described next in the chapter "Objective evaluation of spatial sound quality using directional loudness maps", in the chapter "Using directional loudness for audio coding and objective quality measurement", in the chapter "Directional loudness for audio coding", in the chapter " General Steps for Computing a Directional Loudness Map (DirLoudMap)", in the chapter "Example: Restoring Directional Signals with a Windowing/Selection Function Derived from a Pan Index", and in the chapter "Embodiments for Various Forms of Computing Loudness Maps Using Generalized Estimator Functions" .

In addition, additional embodiments will be defined by the appended claims.

It should be noted that any embodiments defined in the claims may be supplemented by any of the details (features and functionality) described in the above chapters.

In addition, the embodiments described in the above chapters may be used alone, and may also be supplemented by any of the features in another chapter or by any feature included in the claims.

In addition, it should be noted that the individual aspects described in this document may be used individually or in combination. Thus, details can be added to each of said individual aspects without adding details to another of said aspects.

It should also be noted that the present disclosure describes, explicitly or implicitly, features used in an audio encoder (a device for providing an encoded representation of an input audio signal) and an audio decoder (a device for providing a decoded representation of an audio signal based on the encoded representation). Thus, any of the features described herein may be used in the context of an audio encoder and in the context of an audio decoder.

In addition, any features and functionality related to the method disclosed herein may also be used in a device (capable of implementing such functionality). In addition, any features and functionality disclosed herein in relation to the device can also be used in the corresponding method. In other words, the methods disclosed herein may be supplemented by any of the features and functionality described in relation to devices.

In addition, any of the features and functionality described herein may be implemented in hardware or software, or using a combination of hardware and software, as will be described in the "Implementation Alternatives" section.

Implementation alternatives

Although some aspects have been described in the context of a device, it is clear that these aspects also represent a description of the corresponding method, in which the block or device corresponds to a method step or a feature of a method step. Likewise, the aspects described in the context of a method step also provide a description of the corresponding block or element or feature of the corresponding device. Some or all of the steps of the method may be executed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important steps of the method may be performed by such a device.

Depending on some implementation requirements, embodiments of the invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium such as floppy disk, digital versatile disc (DVD), Blu-ray disc, compact disc (CD), read only memory (ROM), programmable read only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory having electronically readable signals stored thereon that interact (or are capable of interacting) with a programmable computer system, resulting in the execution of the corresponding method. Thus, the digital storage medium may be machine readable.

Some embodiments in accordance with the invention include a storage medium having electronically readable control signals that are capable of interacting with a programmable computer system, resulting in one of the methods described here.

Typically, embodiments of the present invention may be implemented as a computer program product with program code, the program code being able to function to perform one of the methods when the computer program product is running on the computer. The program code may, for example, be stored on a computer-readable medium.

Other embodiments include a computer program for performing one of the methods described herein, stored on a computer-readable medium.

In other words, an embodiment of the method of the invention is thus a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

An additional embodiment of the methods of the invention is thus a storage medium (either a digital storage medium or a computer-readable medium) containing a computer program recorded thereon for performing one of the methods described herein. The storage medium, digital storage medium, or recorded data medium is typically tangible and/or non-transitory.

An additional embodiment of the method of the invention is thus a stream of data or a sequence of signals representing a computer program for performing one of the methods described herein. The data stream or signal sequence, for example, can be configured to be transmitted over a data connection, such as the Internet.

An additional embodiment comprises a processing means, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

An additional embodiment comprises a computer having a computer program installed thereon for performing one of the methods described herein.

A further embodiment in accordance with the invention comprises an apparatus or system capable of transferring (eg, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver, for example, may be a computer, mobile device, storage device, or the like. The device or system, for example, may include a file server for transferring a computer program to the receiver.

In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a user-programmable gate array may interface with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware device.

The apparatus described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

The device described herein, or any components of the device described herein, may be implemented at least in part in hardware and/or software.

The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

The methods described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and/or software.

The embodiments described above are merely illustrative of the principles of the present invention. It is intended that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Thus, the invention is intended to be limited only by the scope of the following patent claims, and not by the specific details provided through the description and explanation of the embodiments presented herein.

Objective evaluation of surround sound quality using directional loudness maps

2. Abstract

In this work, a feature is introduced, for example, extracted from stereo/binaural audio signals, to measure the perceived quality degradation in processed spatial audio scenes. This function may be based on a simplified model assuming a stereo mix created by directional signals positioned using an amplitude level panning technique. We calculate, for example, the corresponding loudness in the stereo image for each directional signal in the Short Time Fourier Transform (STFT) domain to compare the reference signal and the degraded version and obtain a measure of distortion aimed at describing the perceived impairment metrics obtained in the listening tests.

This measure has been tested against an extensive database of listening tests with stereo signals processed by state of the art perceptual audio codecs using non-waveform preserving techniques such as bandwidth extension and joint stereo coding, known for causing difficulties for existing quality predictors [1] , [2]. The results show that the resulting distortion measure can be included as an addition to existing automated perceptual quality estimation algorithms to improve the prediction of spatially encoded audio signals.

Keywords - spatial sound, objective quality assessment, PEAQ, panning index.

1. Introduction

We propose a simple feature aimed at describing the deterioration of the perceived stereo sound image, for example, based on the change in loudness in areas that have a common panning index [13]. For example, these are the time-frequency regions of a binaural signal that have the same ratio of intensity levels between the left and right channels, and therefore correspond to a certain perceived direction in the horizontal plane of the sound image.

The use of directional loudness measurements in the context of audio scene analysis for rendering audio data of complex virtual environments is also proposed in [14], while current work is focused on a general objective assessment of the quality of spatial audio coding.

The perceived distortion of the stereo image can be reflected as changes in the directional loudness map of a certain granularity corresponding to the number of pan index values to be evaluated as a parameter.

2. Way

According to an embodiment, a reference signal (REF) and a signal under test (SUT) are processed in parallel to extract features that aim to describe - in comparison - the deterioration in perceived audio quality caused by operations performed to obtain the SUT signal.

отсчета и наложением

, задающими временное разрешение 21 мс при частоте дискретизации F_S=48 кГц. Частотные интервалы преобразованного сигнала затем, например, группируются для учета частотной селективности передней части ушного лабиринта человека по шкале ERB [15] в общей сложности в

подмножествах частотных интервалов или частотных полосах. Затем каждая частотная полоса может быть взвешена посредством значения, полученного из объединенной линейной передаточной функции, которая моделирует внешнее и среднее ухо, как описано в [3].Both binaural signals can first be processed by the Peripheral Ear Model block. Each input signal is, for example, decomposed into the STFT transform domain using a Hann window with block size

countdown and overlay

, specifying a time resolution of 21 ms at a sampling rate F _S= 48 kHz. The frequency intervals of the converted signal are then, for example, grouped to take into account the frequency selectivity of the anterior part of the human ear labyrinth on the ERB scale [15] in total

subsets of frequency intervals or frequency bands. Each frequency band can then be weighted by the value obtained from the combined linear transfer function that models the outer and middle ear as described in [3].

в каждом временном кадре

и частотном интервале

, и для каждого канала

и каждой группы частот

с разной шириной

, выраженной в частотных интервалах.Then the peripheral model issues signals

in every time frame

and frequency interval

, and for each channel

and each frequency group

with different width

expressed in frequency intervals.

2.1. Directional loudness calculation (e.g., performed by the audio analysis module and/or audio similarity estimator described herein)

может интерпретироваться как

, где j ϵ [1; J]. Следующая концепция основана на методе, представленном в [13], в котором мера сходства между левым и правым каналами бинаурального сигнала в области преобразования STFT может быть использована для извлечения частотно-временных областей, занятых каждым источником в стереофонической записи, на основе их обозначенных коэффициентов панорамирования в процессе микширования.According to an embodiment, the directional loudness calculation can be performed for different directions such that, for example, a certain panning direction

can be interpreted as

, where j ϵ [1; J]. The following concept is based on the method presented in [13], in which a similarity measure between the left and right channels of a binaural signal in the STFT transform domain can be used to extract the time-frequency regions occupied by each source in a stereo recording based on their designated pan coefficients. during the mixing process.

частотно-временная (T/F) ячейка

может быть восстановлена из входного сигнала, соответствующего определённому направлению панорамирования

, посредством умножения входного сигнала на оконную функцию

:Considering the output signal of the peripheral model

time-frequency (T/F) cell

can be recovered from the input signal corresponding to the specific pan direction

, by multiplying the input signal by the window function

:

(1)

(1)

в пределах значения допуска. Функция оконной обработки может быть определена как гауссово окно, центрированное в требуемом направлении панорамирования:The reconstructed signal will have input signal time-frequency components that match the pan direction.

within the tolerance value. The windowing function can be defined as a Gaussian window centered in the desired panning direction:

(2)

(2)

- индекс панорамирования, вычисляемый в [13] с определенной поддержкой

соответствующих сигналов, панорамированных полностью влево или вправо, соответственно. Действительно,

может содержать частотные интервалы, значения которых в левом и правом каналах приведут к тому, что функция

будет иметь значение

или вблизи него. Все другие компоненты могут быть ослаблены в соответствии с гауссовой функцией. Значение

представляет ширину окна и, таким образом, упомянутую окрестность для каждого направления панорамирования. Значение

=0,006 было выбрано, например, для отношения сигнал/помеха (SIR) в -60 дБ [13]. Факультативно множество из 22 равномерно расположенных направлений панорамирования в пределах

выбирается эмпирически для значений

. Для каждого восстановленного сигнала вычисление громкости [16] в каждой частотной полосе ERB и в зависимости от направления панорамирования выражается, например, следующим образом:Where

- panning index calculated in [13] with a certain support

corresponding signals panned all the way to the left or right, respectively. Really,

may contain frequency intervals whose values in the left and right channels will cause the function

will matter

or near it. All other components can be attenuated according to a Gaussian function. Meaning

represents the width of the window and thus the referred neighborhood for each panning direction. Meaning

=0.006 was chosen, for example, for a signal-to-noise ratio (SIR) of -60 dB [13]. Optional set of 22 evenly spaced pan directions within

is chosen empirically for the values

. For each reconstructed signal, the loudness calculation [16] in each ERB frequency band and depending on the panning direction is expressed, for example, as follows:

(3)

(3)

- суммарный сигнал каналов

. Затем громкость усредняется, например, по всем частотным полосам ERB, чтобы обеспечить карту направленной громкости, определенную в области панорамирования

по временному кадру

:Where

- total channel signal

. The loudness is then averaged over, for example, all ERB frequency bands to provide a directional loudness map defined in the pan area

by time frame

:

(4)

(4)

, соответствующие частотам от 1,34 кГц до

.For further refinement, Equation 4 can only be computed considering a subset of the ERBs corresponding to frequency domains of 1.5 kHz and above to accommodate the sensitivity of the human auditory system to level differences in this domain, according to duplex theory [17]. In accordance with the embodiment, bands are used

corresponding to frequencies from 1.34 kHz to

.

As a step, for example, the directional loudness-to-time maps of the reference signal and the SUT signal are subtracted, and then the absolute value of the difference is averaged over all pan and time directions to obtain a single number, following the terminology in [3] called the Model Output Variable (MOV). This number, effectively expressing the distortion between the directional loudness maps of the reference and test signals, is expected to predict the associated subjective degradation reported in listening tests.

In FIG. 9 shows a flow chart for the proposed MOV (Model Output Value) calculation. In FIG. 10a-10c show an example of applying the directional loudness map concept to a pair of reference (REF) and degraded (SUT) signals and their absolute difference (DIFF). In FIG. 10a to 10c show an example recording of a 5 second violin solo panned to the left. Clearer areas on the maps represent, for example, louder content. The degraded signal (SUT) represents the temporal collapse of the panning direction of the acoustic event from left to center between times 2 and 2.5 seconds, and again between 3 and 3.5 seconds.

3. Description of the experiment

), оценки абсолютной погрешности (

) и количества выбросов (

), как описано в [3].To test and confirm the usefulness of the proposed MOV value, a regression experiment similar to that described in [18] was performed in which the MOV values were calculated for pairs of reference and test signals in the database and compared with their respective subjective quality scores from the listening test. The predictive performance of a system using this MOV value is evaluated in terms of correlation with subjective data (

), absolute error estimates (

) and the amount of emissions (

), as described in [3].

The database used for the experiment corresponds to part of the Unified Voice and Audio Data Coding (USAC) verification test, set 2 [19], which contains stereo signals encoded at bit rates ranging from 16 to 24 kbps using joint stereo tools [12 ] and broadening the frequency band along with their qualitative assessment on the MUSHRA scale. Speech elements have been excluded because the proposed MOV value is not expected to describe the root cause of speech distortion. A total of 88 items (eg, average length 8 seconds) remained in the database for the experiment.

To account for possible monophonic/timbre distortion in the database, the results of the implementation of the standard PEAQ (extended version) called Objective Difference Score (ODG) and POLQA called Mean Opinion Score (MOS) were taken as additional MOV values complementing directional loudness distortion (DirLoudDist; for example, DLD) described in the previous section. All MOV values can be normalized and adapted to give a score of 0 to indicate the best quality and 1 to indicate the worst quality possible. The listening test results were scaled accordingly.

,

и

были рассмотрены в качестве показателей эффективности.One random portion of the available database content (60%, 53 items) was reserved for regression model training using multivariate adaptive regression splines (MARS) [8] mapping MOV values to subjective item scores. The rest (35 points) was used to test the performance of the trained regression model. To exclude the influence of the training procedure from the overall analysis of the effectiveness of the MOV values, the training/testing cycle was conducted, for example, 500 times with random training/testing elements, and the average values for

,

And

were considered as performance indicators.

4. Results and discussion

MOV Set(N) R AES v MOS+ODG(2) 0.77 2.63 12 MOS+ODG+CHOI(5) 0.77 2.39 eleven MOS+ODG+EITDD(3) 0.82 2.0 eleven MOS+ODG+SEO(6) 0.88 1.65 7 MOS+ODG+DirLoudDist(3) 0.88 1.69 8

Table 1: Average performance values for 500 train/validate cycles (eg testing) of a regression model with different sets of MOV values. CHOI represents 3 binaural MOV values as calculated in [20], EITDD corresponds to the high frequency envelope of the ITD distortion MOV value as calculated in [1]. SEO matches 4 binaural MOV values from [1], including EITDD. DirLoudDist is the suggested MOV value. The number in parentheses represents the total number of MOV values used (optional).

Table 1 shows the average performance values (correlation, absolute error estimate, number of outliers) for the experiment described in section 3. In addition to the proposed MOV value, the methods for objective evaluation of spatially encoded audio signals proposed in [20] and [ 1]. Both implementations being compared use the classic interaural landmark distortions mentioned in the introduction: IACC distortion (IACCD), ILD distortion (ILDD), and ITDD.

As mentioned, baseline performance is given by ODG and MOS, which alone achieve R=0.66, but represent a combined R=0.77 as shown in Table 1. This confirms that the features are complementary in the evaluation of monophonic distortion.

Based on the work of Choi et. al. [20] Adding three binaural distortions (CHOIs in Table 1) to two monophonic quality measures (constituting up to five joint MOV values) does not provide a further improvement in the system in terms of prediction performance for the dataset used.

In [1], some additional optional refinements of the model for the mentioned features were introduced in terms of localization in the lateral plane and the possibility of detecting landmark distortions. In addition, for example, a new MOV value has been included that takes into account the high-frequency distortion of the Envelope Interaural Time Difference (EITDD) [21]. A set of these four binaural MOVs (labeled SEO in Table 1) plus two monaural descriptors (for a total of six MOVs) greatly improves system performance for the current dataset.

Given the contribution to the improvement of EITDD, it can be assumed that the time-frequency energy envelopes used in joint stereo methods [12] represent an important aspect of the overall perception of quality.

However, the reported MOV value based on directional loudness map distortions (DirLoudDist) correlates even better with perceived quality degradation than EITDD and even achieves similar performance as a combination of all binaural MOV values [1] when using one additional MOV value for two monaural quality descriptors instead of four. Using fewer features for the same performance reduces the risk of overlearning and indicates their higher perceptual significance.

The maximum average correlation with subjective estimates for the database of 0.88 indicates that there is still room for improvement.

In accordance with an embodiment, the proposed feature is based on the model described here, which assumes a simplified description of stereo signals, in which sound objects are localized only in the lateral plane using ILDS, which is usually the case in studio production audio content [13]. For ITD distortion, typically present when encoding recordings with multiple microphones or more natural sounds, the model must either be extended or augmented with a suitable measure of ITD distortion.

5. Conclusions and directions for further research

In accordance with an embodiment, a distortion metric has been introduced that describes changes in the presentation of an audio scene based on the loudness of events corresponding to a particular panning direction. A significant increase in performance in relation to only monaural quality prediction shows the effectiveness of the proposed method. This approach also offers a possible alternative or addition to quality measurement for low bit rate spatial audio coding when established distortion measurements based on classical binaural landmarks do not work satisfactorily, perhaps due to the waveform-preserving nature of the audio data processing involved in the processing.

Performance measurements show that there are still areas for improvement to a more complete model that also includes audio distortion based on effects not related to channel level differences. Further research also includes exploring how the model can describe temporal instability/modulation in a stereo image as reported in [12], as opposed to static distortion.

Literature

[1] Jeong-Hun Seo, Sang Bae Chon, Keong-Mo Sung, and Inyong Choi, “Perceptual objective quality evaluation method for high quality multichannel audio codecs,” J. Audio Eng. Soc , vol. 61, no. 7/8, pp. 535-545, 2013.

[2] M. Scha¨fer, M. Bahram, and P. Vary, “An extension of the PEAQ measure by a binaural hearing model,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , May 2013, pp. 8164-8168.

[3] ITU-R Rec. BS.1387, Method for objective measurements of perceived audio quality , ITU-T Rec. BS.1387, Geneva, Switzerland, 2001.

[4] ITU-T Rec. P.863, “Perceptual objective listening quality assessment,” Tech. Rep., International Telecommunication Union, Geneva, Switzerland, 2014.

[5] Sven Ka¨mpf, Judith Liebetrau, Sebastian Schneider, and Thomas Sporer, “Standardization of PEAQ-MC: Extension of ITU-R BS.1387-1 to Multichannel Audio,” in Audio Engineering Society Conference: 40th International Conference: Spatial Audio: Sense the Sound of Space , Oct 2010.

[6] K Ulovec and M Smutny, “Perceived audio quality analysis in digital audio broadcasting plus system based on PEAQ,” Radioengineering , vol. 27, pp. 342-352, Apr. 2018.

[7] C. Faller and F. Baumgarte, “Binaural cue coding-Part II: Schemes and applications,” IEEE Transactions on Speech and Audio Processing , vol. 11, no. 6, pp. 520-531, Nov 2003.

[8] Jan-Hendrik Fleßner, Rainer Huber, and Stephan D. Ewert, “Assessment and prediction of binaural aspects of audio quality,” J. Audio Eng. Soc , vol. 65, no. 11, pp. 929-942, 2017.

[9] Marko Takanen and Gae¨tan Lorho, “A binaural auditory model for the evaluation of reproduced stereophonic sound,” in Audio Engineering Society Conference: 45th International Conference: Applications of Time-Frequency Processing in Audio , Mar 2012.

[10] Robert Conetta, Tim Brookes, Francis Rumsey, Slawomir Zielinski, Martin Dewhirst, Philip Jackson, Søren Bech, David Meares, and Sunish George, “Spatial audio quality perception (part 2): A linear regression model,” J. Audio Eng. Soc , vol. 62, no. 12, pp. 847-860, 2015.

[11] ITU-R Rec. BS.1534-3, “Method for the subjective assessment of intermediate quality levels of coding systems,” Tech. Rep., International Telecommunication Union, Geneva, Switzerland, Oct. 2015.

[12] Frank Baumgarte and Christof Faller, “Why binaural cue coding is better than intensity stereo coding,” in Audio Engineering Society Convention 112 , Apr 2002.

[13] C. Avendano, “Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications,” in 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics , Oct 2003, pp. 55-58.

[14] Nicolas Tsingos, Emmanuel Gallo, and George Drettakis, “Perceptual audio rendering of complex virtual environments,” in ACM SIGGRAPH 2004 Papers , New York, NY, USA, 2004, SIGGRAPH '04, pp. 249-258, ACM.

[15] BCJ Moore and BR Glasberg, “A revision of Zwicker's loudness model,” Acustica United with Acta Acustica:the Journal of the European Acoustics Association , vol. 82, no. 2, pp. 335-345, 1996.

[16] E. Zwicker, “U¨ber psychologische und methodische Grundlagen der Lautheit [On the psychological and methodological bases of loudness],” Acustica , vol. 8, pp. 237-258, 1958.

[17] Ewan A. Macpherson and John C. Middlebrooks, “Listener weighting of cues for lateral angle: The duplex theory of sound localization revisited,” The Journal of the Acoustical Society of America , vol. 111, no. 5, pp. 2219-2236, 2002.

[18] Pablo Delgado, Ju¨rgen Herre, Armin Taghipour, and Nadja Schinkel-Bielefeld, “Energy aware modeling of interchannel level difference distortion impact on spatial audio perception,” in Audio Engineering Society Conference: 2018 AES International Conference on Spatial Reproduction - Aesthetics and Science , Jul 2018.

[19] ISO/IEC JTC1/SC29/WG11, “USAC verification test report N12232,” Tech. Rep., International Organization for Standardization, 2011.

[20] Inyong Choi, Barbara G. Shinn-Cunningham, Sang Bae Chon, and Koeng-Mo Sung, “Objective measurement of perceived auditory quality in multichannel audio compression coding systems,” J. Audio Eng. Soc , vol. 56, no. 1/2, pp. 3-17, 2008

[21] ER Hafter and Raymond Dye, “Detection of interaural differences of time in trains of high-frequency clicks as a function of interclick interval and number,” The Journal of the Acoustical Society of America , vol. 73, pp. 644-51, 03 1983.

Using Directional Loudness for Audio Coding and Objective Quality Measurement

See the chapter titled "Objective Surround Quality with Directional Loudness Maps" for more information.

Description: (for example, the description of Fig. 9)

A feature extracted from, for example, stereo/binaural audio signals in a spatial (stereo) audio scene is presented. The feature is based, for example, on a simplified stereo mixing model that extracts the panning directions of events in a stereo image. The associated loudness in the stereo image for each panning direction in the Short Time Fourier Transform (STFT) region can be calculated. The feature is optionally computed for the reference and encoded signal and then compared to obtain a measure of distortion to describe the perceived impairment score reported in the listening test. The results show improved robustness to low bit rate, non-waveform-preserving parametric techniques, such as joint stereo and bandwidth extension, compared to existing techniques. The results show improved robustness to low bit rates, non-waveform-preserving parametric techniques, such as joint stereo and bandwidth extension, compared to existing techniques. This can be combined in standardized objective quality assessment systems such as PEAQ or POLQA (PEAQ - Objective Measures of Perceived Audio Quality; POLQA - Perceived Objective Listening Quality Analysis).

Terminology:

- Signal: For example, a stereo signal representing objects, downmixes, differences, etc.

- Directional Loudness Map (DirLoudMap): eg derived from each signal. Represents, for example, the volume in the time-frequency (T/F) domain associated with each panning direction in the audio scene. It can be derived from more than two signals by using binaural rendering (HRTF (hearing aid transfer function) / BRIR (binaural room impulse response)).

Applications (embodiments):

1. Automatic quality assessment (option 1) :

- As described in the chapter "Objectively Evaluating Surround Quality Using Directional Loudness Maps"

2. Bit allocation based on directional loudness (embodiment 2) in the audio encoder, based on the ratio (contribution) of the DirLoudMaps of the individual signals in the overall DirLoudMap.

- optional variation 1 (independent stereo pairs): audio signals as loudspeakers or objects.

- optional variation 2 (downmix/difference pairs): the contribution of the DirLoudMap of the downmixed signal and the difference DirLoudMap to the overall DirLoudMap. The "contribution amount" in the audio scene for the bit allocation criteria.

1. An audio encoder that jointly encodes two or more channels, such as those resulting in each one or more downmix signals and difference signals, in which the contribution of each difference signal to the overall directional loudness map is determined, for example, from a fixed decoding rule (e.g. , MS-Stereo) or by evaluating the inverse joint coding process from joint coding parameters (eg rotation in MCT). Based on the contribution of the difference signal to the overall DirLoudMap, the bit rate allocation between the downmix signal and the difference signal is adapted, for example, by controlling the signal quantization precision or by directly discarding difference signals whose contribution is below a threshold. Possible criteria for "contribution", for example, is the average ratio or the ratio in the maximum relative contribution of the direction.

- Issue: Combining and evaluating the contribution of an individual DirLoudMap to the resulting/total loudness map.

3. (Embodiment 3) On the decoder side, directional loudness can help the decoder make the next informed decision.

- Scaling/Format Conversion Complexity: Each audio signal can be included or excluded during the decoding process based on its contribution to the overall DirLoudMap (passed as a separate parameter or estimated based on other parameters) and thereby change the rendering complexity for different applications/format transformations. This allows for reduced complexity decoding when only limited resources are available (ie, a multi-channel signal being converted for listening on a mobile device).

- Because the resulting DirLoudMap may depend on the playback target setting, this ensures that the most important/essential signals for a particular scenario are reproduced, thus having an advantage over non-spatial approaches such as a simple signal/object priority level.

4. To make a joint encoding decision (Embodiment 4) (eg, description of FIG. 14)

- Determine the contribution of the directional loudness map of each signal or each pair of possible signals to the contribution of the DirLoudMap map of the overall scene.

1. optional variation 1) Select a pair of signals with the highest contribution to the overall loudness map

2. optional variation 2) Select a pair of signals where the signals have high proximity/similarity in their respective DirLoudMap => can be jointly represented by downmixing

- Since cascaded joint coding of signals is possible, a DirLoudMap of eg a downmix signal does not necessarily correspond to a point source from one direction (eg a single speaker), hence the contribution to a DirLoudMap eg is estimated based on the joint coding parameters.

- The DirLoudMap of the overall scene can be computed through some kind of downmix or binauralization that takes into account signal directions.

5. Parametric Audio Codec (Embodiment 5) Based on Directional Loudness

- transmits, for example, a map of the directional volume of the scene --> is transmitted as auxiliary information in parametric form

1. "PCM style" = quantized values across directions

2. center position + linear gradients for left/right side

3. polynomial or spline representation

- transmits, for example, one signal, or fewer signals, or efficient transmission,

1. optional modification 1) passes parameterized target scene map DirLoudMap + 1 downmix channel

2. optional modification 2) transmits multiple signals, each with an associated DirLoudMap

3. optional modification 3) passes a common target DirLoudMap and several signals plus a parametric relative contribution to the common DirLoudMap

- synthesizes , for example, a complete audio scene based on the transmitted signal, based on the directional volume map of the scene.

Directional volume for audio coding

Introduction and definitions

DirLoudMap=Directional Loudness Map

Implementation for DirLoudMap calculation:

a) perform time-frequency decomposition (+ grouping into critical frequency bands (CB)) (e.g. with filter bank, STFT transform, ...)

b) perform a direction analysis function for each time-frequency cell

c) enter/accumulate the result of point b) in the histogram of the DirLoudMap map optionally (if necessary):

d) Summarize the output over critical frequency bands (CB) to provide a broadband DirLoudMap

DirLoudMap map layer implementation/direction analysis function:

- Level 1 (optional): Contribution directions to maps according to the spatial position of signal reproduction (channels/objects) - (without knowledge of signal content used). Uses the direction analysis function considering only the playback direction of the channel/object +/- the playback direction of the L1 expansion window of the channel/object +/- the expansion window (can be wideband, i.e. the same for all frequencies),

- Level 2 (optional): Map input directions according to the spatial position of signal reproduction (channels/features) plus *dynamic* channel/feature signal content function (direction analysis function) of different complexity levels.

Allows you to identify

optional L2a): panned phantom sources (-> pan index) [level], or optional L2b) delay level+time panned phantom sources [level and time], or optional L2c) extended (decorrelated) panned phantom sources (even more advanced)

Applications for Perceptual Audio Coding

Embodiment A) per-channel/object masking - no joint coding tools -> goal: encoder quantization noise control (such that the original and encoded/decoded DirLoudMap deviate less than a certain threshold, i.e. a target criterion in the area of DirLoudMap maps ),

Embodiment B) per channel/object concealment - joint coding tools (e.g. M/S+prediction, MCT)

-> target: control of encoder quantization noise in instrument-processed signals (e.g. M or rotated "sum" signal) to meet target criterion in DirLoudMap map area

Example for B)

1) calculate the overall DirLoudMap based on all signals

2) apply collaborative coding tools

3) determine the contribution of the signals processed by the tool (for example, "sum" and "difference") to the DirLoudMap map, taking into account the decoding function (for example, panning by rotation / prediction)

4) control quantization through

a) taking into account the influence of quantization noise on the DirLoudMap

b) accounting for zero quantization values of signal parts in the DirLoudMap map

Embodiment C) Controlling the application (eg, MS on/off) and/or parameters (eg, prediction coefficient) of the joint coding tools

purpose: control the encoder/decoder parameters of the collaborative encoding tools to match the target criteria in the DirLoudMap map area

Examples for C)

- manage decision making on/off M/S based on DirLoudMap map

- control the smoothing of frequency-dependent prediction coefficients based on the effect of changing parameters on the DirLoudMap map

(for cheaper differential parameter coding)

(= managing the trade-off between ancillary information and prediction accuracy)

Option D) define parameters (on/off, ILD, ...) of *parametric* joint coding tools (e.g. stereo intensity)

-> target: Control parameter of parametric collaborative encoding tool to match target criteria in DirLoudMap map area

Embodiment E) Parametric encoder/decoder system transmitting a DirLoudMap as ancillary information (rather than traditional spatial landmarks e.g. ILD, ITD/IPD, ICC, ...)

-> Encoder determines parameters based on analysis of DirLoudMap, generates downmixed signal(s) and parameters (bitstream), e.g. overall DirLoudMap+contribution of each signal to DirLoudMap

-> The decoder synthesizes the transmitted DirLoudMap by suitable means

Embodiment F) Reducing Decoder/Renderer/Format Converter Complexity

Determine the contribution of each signal to the overall DirLoudMap (perhaps based on the passed auxiliary information) to determine the "importance" of each signal. In applications with limited computing power, skip decoding/rendering of signals whose contribution to the DirLoudMap is below a threshold.

General Steps for Computing a Directional Loudness Map (DirLoudMap)

This is for example valid for any implementation: (eg description of Fig. 3a and/or Fig. 4a)

a) Perform time-frequency decomposition of multiple input audio signals.

- optional: grouping of spectral components into processing frequency bands relative to the frequency resolution of the human auditory system (HAS)

- optional: weighting according to HAS sensitivity in different frequency domains (e.g. outer/middle ear transfer function)

-> result: time-frequency bins (e.g. spectral domain representations, spectral bands, spectral intervals, …)

For (FOR) several (for example, each) frequency bands (contours):

или направление панорамирования

).b) Calculate, for example, a directional analytic function on the frequency-time cells of several audio input channels-> result: direction d (for example, the direction

or pan direction

).

c) Calculate, for example, the loudness on the frequency-time cells of several input audio channels

-> result: volume L

- Loudness calculation can be just energy, or more specifically energy (or Zwicker model: alpha=0.25-0.27)

d.a) for example, enter/accumulate contribution l to DirLoudMap under direction d

- Optional: Expanding (panning index: windowing) l distributions between adjacent directions

end for (END FOR)

optional (if necessary for the application): compute a broadband DirLoudMap

d.b) generalize the DirLoudMap over multiple (avoid: over all) frequency bands to provide a broadband DirLoudMap indicating "activity" of audio data depending on direction/space

Example: Restoring Directional Signals with a Windowing/Selection Function Obtained from a Pan Index (Description of FIG. 6)

The signals of the left (see Fig. 6a; red) and right (see Fig. 6b; blue) channels, for example, are shown in Figs. 6a and FIG. 6b. Bands may be DFT (Discrete Fourier Transform) intervals of the entire spectrum, critical frequency bands (groups of frequency intervals), or DFT intervals within a critical frequency band, and so on.

.The criterion function is arbitrarily defined as:

.

The criterion, for example, is "panning direction according to level". For example, the level of each or more FFT intervals.

(одна оконная функция на каждое направление панорамирования

).a) From the criterion function, we can extract a window/weighting function that selects the appropriate frequency intervals/spectral groups/components and reconstructs the directional signals. So the input spectrum (e.g. L and R) will be multiplied by different window functions

(one window function per pan direction

).

(т.е. отношениями уровней между L и R).b) From the evaluation function we get different directions associated with different values

(i.e. the ratio of levels between L and R).

To recover signals using method a)

(содержит полосы, только имеющие соотношение

. Это направленный сигнал (см. фиг. 6a1 и фиг. 6b1).Example 1) Pan center direction,

(contains bands only having the ratio

. This is a directional signal (see Fig. 6a1 and Fig. 6b1).

(содержит только полосы, которые имеют соотношение

. Это направленный сигнал (см. фиг. 6a2 и рис. 6b2).Example 2) The panning direction is slightly shifted to the left,

(contains only bands that have the ratio

. This is a directional signal (see Fig. 6a2 and Fig. 6b2).

(содержит только полосы, которые имеют соотношение

, это направленный сигнал (см. фиг. 6a3.1 и фиг. 6b3.1).Example 3) The panning direction is slightly shifted to the right,

(contains only bands that have the ratio

, this is a directional signal (see Fig. 6a3.1 and Fig. 6b3.1).

или громкость для каждой критической частотной полосы

. Для разных применений возможны различные критерии.Criteria function can be arbitrarily defined as the level of each DFT interval, the energy for a group of DFT intervals (critical frequency band)

or loudness for each critical frequency band

. Different criteria are possible for different applications.

Weigh-in (optional)

Note : Not to be confused with outer/middle ear transfer function weighting (peripheral model), which weights e.g. critical bands.

используется допустимый диапазон и вес менее важных значений, которые отклоняются от

т.е. “извлечь все полосы, которые удовлетворяют соотношению 4/3 и передать их с весовым коэффициентом 1, находящиеся вблизи значения взвесить с коэффициентом меньше 1 -> для этого может использоваться гауссова функция. В упомянутых выше примерах направленные сигналы имели бы больше интервалов, взвешенных не со значением 1, а с меньшими значениями. Weighting: optional instead of exact value

the allowable range and weight of less important values that deviate from

those. “extract all bands that satisfy the 4/3 ratio and pass them with a weight factor of 1 that are close to the weight value with a factor less than 1 -> a gaussian function can be used for this. In the examples mentioned above, directional signals would have more intervals weighted not with a value of 1, but with smaller values.

Motivation: The weighting allows for a "smoother" transition between different directional signals, the separation is not as sharp as there is some "leakage" among the various directional signals.

For example 3), it may look as shown in FIG. 6a3.2 and fig. 6b3.2.

Embodiments of Various Forms of Computing Loudness Maps Using Generalized Evaluation Functions

Optional 1: pan index approach (see Fig. 3a and Fig. 3b):

может быть собрана карта «значений» для этой функции во времени. Так называемая «карта направленной громкости» может быть построена в соответствии с одним из следующих примеров.For (all) different

a map of "values" for that function over time can be collected. A so-called "directional loudness map" can be constructed according to one of the following examples.

, таким образом, направленные сигналы, например, состоят из отдельных интервалов DFT. Затем, например, с использованием вычисления энергии в каждой критической полосе (группе интервалов DFT) для каждого направленного сигнала, и затем подъема этих энергий для каждой критической полосы с экспонентой 0,25 или подобной. -> аналогично главе «Объективная оценка качества пространственного звучания с использованием карт направленной громкости».- Example 1) using the criterion function "panning direction according to the level of the individual FFT intervals"

, thus directional signals , for example, consist of individual DFT intervals . Then, for example, using the calculation of the energy in each critical band (group of DFT intervals) for each directional signal, and then lifting these energies for each critical band with an exponent of 0.25 or similar. -> similar to the chapter "Objective assessment of the quality of spatial sound using directional loudness maps".

- Example 2) Instead of windowing the amplitude spectrum, windowing the loudness spectrum can be performed. Directional signals will be already in the volume area.

. Тогда направленные сигналы будут состоять из участков целых критических полос, которые подчиняются значениям, заданным посредством

.- Example 3) using directly the criteria function "panning direction according to the loudness of each critical band"

. Then the directional signals will consist of sections of entire critical bands , which obey the values given by

.

направленный сигнал может представлять собой:For example, for

directional signal can be:

Y= 1*critical_band_1+0.2*critical_band_2+0.001*critical_band_3.

and different combinations apply for other directions/directional pan signals. It should be noted that when weighting is used, different pan directions may contain the same critical bands, but most likely with different values of the weight coefficient. If no weighting is applied , the directional signals are mutually exclusive .

Optional 2: histogram approach (see Fig. 4b):

This is a more general description of overall directional loudness. It does not necessarily use a panning index (i.e., it is not necessary to reconstruct "directional signals" via spectrum windowing to calculate loudness). The frequency spectrum of the overall loudness is "distributed" according to the "analyzed direction" in the respective frequency domain. The direction analysis can be based on level difference , time difference , or take another form.

For each time frame (see Fig. 5):

будет задано, например, величиной значений, заданных множеству

. Например, это величина интервалов, доступных для группировки случаев

при оценке

во временном кадре. Например, значения накапливаются и сглаживаются по времени, возможно с «коэффициентом забывания»

:Histogram resolution

will be given, for example, by the magnitude of the values given to the set

. For example, this is the amount of intervals available for grouping cases

when evaluating

in a temporary frame. For example, values are accumulated and smoothed over time, possibly with a "forgetting factor"

:

,

,

where n is the index of the time frame.

Claims (81)

1. Audio encoder (300) for encoding (310) input audio content (112) containing one or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b), moreover, the audio encoder (300) is configured to provide one or more encoded audio signals (320) based on one or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) or one or more signals derived from them (110 , 110 ₁ , 110 ₂ , 110a, 110b); wherein the audio encoder (300) is configured to adapt (340) the encoding parameters depending on one or more directional loudness maps that represent loudness information (142, 142 ₁ , 142 ₂ , 142a, 142b) associated with a plurality of different directions (121 ), from one or more signals to be encoded. 2. Audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to adapt (340) the bit distribution between one or more signals and/or parameters to be encoded depending on the contributions of individual directional loudness maps of one or more signals and/or parameters to be encoded into a general map (142, 142 ₁ , 142 ₂ , 142a, 142b) of directional loudness. 3. The audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to prohibit encoding (310) of a certain one of the signals to be encoded when the contributions of an individual directional loudness map of a certain one of the signals to be encoded to the overall directional loudness map are below a threshold values. 4. The audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to adapt (342) the quantization accuracy of one or more signals to be encoded depending on the contributions of individual directional loudness maps of one or more signals to be encoded to the overall directional loudness map . 5. The audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to quantize (312) representations (110, 110 ₁ , 110 ₂ , 110a, 110b) in the spectral domain of one or more input audio signals (112, 112 _{1 . _ _ _} spectral region; wherein the audio encoder (300) is configured to adjust (342) one or more quantization parameters depending on one or more directional loudness maps that represent loudness information (142, 142 ₁ , 142 ₂ , 142a, 142b) associated with a plurality of different directions (121) of one or more signals to be quantized to adapt the generation of one or more encoded audio signals (320); And wherein the audio encoder (300) is configured to encode one or more quantized representations (313) in the spectral domain to obtain one or more encoded audio signals (320). 6. The audio encoder (300) of claim 5, wherein the audio encoder (300) is configured to adjust (342) one or more quantization parameters depending on the contributions of the individual directional loudness maps of the one or more signals to be quantized to the overall directional loudness map. 7. The audio encoder (300) of claim 5, wherein the audio encoder (300) is configured to determine an overall directional loudness map based on the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) such that the overall map directional loudness represents loudness information (142, 142 ₁ , 142 ₂ , 142a, 142b) associated with different directions (121) of the audio scene represented by the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b). 8. The audio encoder (300) of claim 5, wherein the one or more signals to be quantized are associated with different directions (121), or are associated with different speakers, or are associated with different audio objects. 9. The audio encoder (300) of claim 5, wherein the signals to be quantized comprise joint multi-signal coding components of two or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b). 10. The audio encoder (300) of claim 5, wherein the audio encoder (300) is configured to estimate the contribution of the joint multi-signal coding difference signal to the overall directional loudness map and control (342) one or more quantization parameters depending on it. 11. Audio encoder (300) according to claim 1, moreover, the audio encoder (300) is configured to adapt (340) the distribution of bits between one or more signals and/or parameters to be encoded separately for different spectral intervals or separately for different frequency bands; and/or moreover, the audio encoder (300) is configured to adapt (342) the quantization accuracy of one or more signals to be encoded separately for different spectral intervals or separately for different frequency bands. 12. Audio encoder (300) according to claim 1, moreover, the audio encoder (300) is configured to adapt (340) the distribution of bits between one or more signals and/or parameters to be encoded depending on the estimate of the spatial masking between two or more signals to be encoded, wherein the audio encoder (300) is configured to estimate spatial concealment based on directional loudness maps associated with two or more signals to be encoded. 13. The audio encoder (300) of claim 12, wherein the audio encoder (300) is configured to estimate the effect of masking the loudness contribution (132, 132 ₁ , 132 ₂ , 135 ₁ , 135 ₂ ) associated with the first direction of the first signal to be encoded on the loudness contribution (132, 132 ₁ , 132 ₂ , 135 ₁ , 135 ₂ ) associated with the second direction of the second signal to be encoded. 14. Audio encoder (300) according to claim 1, wherein the audio encoder (300) comprises an audio data analysis module (100), moreover, the audio data analysis module (100) is configured to obtain representations (110, 110 ₁ , 110 ₂ , 110a, 110b) in the spectral region of two or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b); moreover, the audio data analysis module (100) is configured to obtain information (122, 122 ₁ , 122 ₂ , 125, 127) about the direction associated with the spectral bands of representations (110, 110 ₁ , 110 ₂ , 110a, 110b) in the spectral region; moreover, the audio data analysis module (100) is configured to obtain information (142, 142 ₁ , 142 ₂ , 142a, 142b) about the loudness associated with different directions (121) as an analysis result; where the contributions (132, 132 ₁ , 132 ₂ , 135 ₁ , 135 ₂ ) to the information (142, 142 ₁ , 142 ₂ , 142a, 142b) about loudness are determined depending on the information (122, 122 ₁ , 122 ₂ , 125, 127) about direction; moreover, information (142, 142 ₁ , 142 ₂ , 142a, 142b) about loudness associated with different directions (121) forms a map of directional loudness. 15. Audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to adapt (340) the noise introduced by the encoder depending on one or more directional loudness maps. 16. Audio encoder (300) according to claim 15, moreover, the audio encoder (300) is configured to use the deviation between the directional loudness map, which is associated with a certain unencoded input audio signal, and the directional loudness map, achievable by the encoded version of the certain input audio signal, as a criterion for adapting the generation of a certain encoded audio signal. 17. Audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to activate and deactivate the joint coding tool depending on one or more directional loudness maps that represent loudness information (142, 142 ₁ , 142 ₂ , 142a, 142b) associated with a plurality of different directions (121 ) one or more signals to be encoded. 18. Audio encoder (300) according to claim 1, wherein the audio encoder (300) is configured to determine one or more joint coding tool parameters depending on one or more directional loudness maps that represent loudness information (142, 142 ₁ , 142 ₂ , 142a, 142b) associated with a plurality of different directions (121) one or more signals to be coded. 19. Audio encoder (300) according to claim 1, moreover, the audio encoder (300) is configured to determine or evaluate the effect of a variation of one or more control parameters that control the provision of one or more encoded audio signals (320) on a directional loudness map of one or more encoded signals and adjust one or more control parameters depending on the definition or impact assessments. 20. Audio encoder (300) according to claim 1, moreover, the audio encoder (300) is configured to obtain a direction component used to obtain one or more directional loudness maps using metadata representing information about the position of the speakers associated with the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b). 21. Audio encoder (300) for encoding (310) input audio content (112) containing one or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b), moreover, the audio encoder (300) is configured to provide one or more encoded audio signals (320) based on two or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) or based on two or more signals derived from them (110, 110 ₁ , 110 ₂ , 110a, 110b) using joint coding (310) of two or more signals to be jointly encoded; moreover, the audio encoder (300) is configured to select (350) signals to be jointly encoded from a plurality of possible signals (110, 110 ₁ , 110 ₂ ) or from a plurality of pairs of possible signals (110, 110 ₁ , 110 ₂ ) depending on the directional loudness maps , which represent information (142, 142 ₁ , 142 ₂ , 142a, 142b) associated with a plurality of different directions (121) of possible signals (110, 110 ₁ , 110 ₂ ) or pairs of possible signals (110, 110 ₁ , 110 ₂ ) . 22. Audio encoder (300) according to claim 21, moreover, the audio encoder (300) is configured to select (350) the signals to be jointly encoded from a plurality of possible signals (110, 110 ₁ , 110 ₂ ) or from a plurality of pairs of possible signals (110, 110 ₁ , 110 ₂ ) depending on the contributions of individual maps directional loudness of possible signals (110, 110 ₁ , 110 ₂ ) into the general directional loudness map or depending on the contributions of the directional loudness maps of pairs of possible signals (110, 110 ₁ , 110 ₂ ) into the general directional loudness map. 23. Audio encoder (300) according to claim 21, moreover, the audio encoder (300) is configured to determine the contribution of pairs of possible signals (110, 110 ₁ , 110 ₂ ) to the overall directional loudness map; And wherein the audio encoder (300) is configured to select one or more pairs of possible signals (110, 110 ₁ , 110 ₂ ) having the highest contribution to the overall directional loudness map for joint encoding (310), or moreover, the audio encoder (300) is configured to select one or more pairs of possible signals (110, 110 ₁ , 110 ₂ ), whose contribution to the overall directional loudness map is greater than a given threshold value, for joint encoding (310). 24. Audio encoder (300) according to claim 21, moreover, the audio encoder (300) is configured to determine separate directional loudness maps of two or more possible signals (110, 110 ₁ , 110 ₂ ), and moreover, the audio encoder (300) is configured to compare individual directional loudness maps of two or more possible signals (110, 110 ₁ , 110 ₂ ), and moreover, the audio encoder (300) is configured to select (350) two or more of the possible signals (110, 110 ₁ , 110 ₂ ) for joint encoding (310) depending on the result of the comparison. 25. Audio encoder (300) according to claim 21, moreover, the audio encoder (300) is configured to determine the overall directional loudness map using downmixing of the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) or using binauralization of the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b). 26. Audio encoder (300) for encoding (310) input audio content (112) containing one or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b), moreover, the audio encoder (300) is configured to provide one or more encoded audio signals (320) based on two or more input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) or based on two or more signals derived from them (110, 110 ₁ , 110 ₂ , 110a, 110b); wherein the audio encoder (300) is configured to determine a common directional loudness map based on the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) and/or determine one or more separate directional loudness maps associated with individual input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b); And wherein the audio encoder (300) is configured to encode the overall directional loudness map and/or one or more individual directional loudness maps as side information. 27. Audio encoder (300) according to claim 26, wherein the audio encoder (300) is configured to determine an overall directional loudness map based on the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) such that the overall directional loudness map represents information (142, 142 ₁ , 142 ₂ , 142a, 142b) about the loudness associated with different directions (121) of the audio scene represented by the input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b). 28. Audio encoder (300) according to claim 26, moreover, the audio encoder (300) is configured to encode the overall directional loudness map as a set of values associated with different directions (121); or wherein the audio encoder (300) is configured to encode an overall directional loudness map using a center position value and gradient information; or wherein the audio encoder (300) is configured to encode the overall directional loudness map as a polynomial representation; or wherein the audio encoder (300) is configured to encode the overall directional loudness map as a spline representation. 29. Audio encoder (300) according to claim 26, wherein the audio encoder (300) is configured to encode a single downmix signal derived from a plurality of input audio signals (112, 112 ₁ , 112 ₂ , 112 ₃ , 112a, 112b) and a common directional loudness map; or moreover, the audio encoder (300) is configured to encode a plurality of signals and encode individual directional loudness maps of a plurality of encoded signals; or wherein the audio encoder (300) is configured to encode the overall directional loudness map, a plurality of signals and parameters describing the contributions of the encoded messages to the overall directional loudness map. 30. A method (3000) for encoding input audio content containing one or more input audio signals, the method comprising: providing (3100) one or more encoded audio signals based on one or more input audio signals or one or more signals derived from them; And the method comprising adapting (3200) generating one or more encoded audio signals depending on one or more directional loudness maps that represent loudness information associated with a plurality of different directions of one or more signals to be encoded. 31. A method (4000) for encoding input audio content containing one or more input audio signals, the method comprising: providing (4100) one or more encoded audio signals based on two or more input audio signals or based on two or more signals derived from them, using joint encoding of the two or more signals to be jointly encoded; And the method comprising selecting (4200) the signals to be jointly encoded from a plurality of possible signals or from a plurality of pairs of candidate signals depending on directional loudness maps that represent loudness information associated with a plurality of different directions of candidate signals or pairs of candidate signals. 32. A method (5000) for encoding input audio content containing one or more input audio signals, the method comprising: providing (5100) one or more encoded audio signals based on two or more input audio signals or based on two or more signals derived from them; the method comprising determining (5200) an overall directional loudness map based on the input audio signals and/or determining one or more separate directional loudness maps associated with the individual audio input signals; And wherein the method comprises encoding (5300) a common directional loudness map and/or one or more individual directional loudness maps as side information. 33. A computer-readable medium that stores a computer program having a program code for performing the method according to claim 30, or 31, or 32 when it is executed on a computer.

Images