RU2722391C2 - System and method of tracking movement of head for obtaining parametric binaural output signal - Google Patents

Abstract

FIELD: means for encoding and decoding of audio signal.

SUBSTANCE: initial rendering of input audio signal based on channel or object is performed to initial output representation. Estimating dominant audio component from input audio signal based on channel or object, and determining a sequence of weight components of the dominant audio component for displaying the initial output representation in the dominant audio component. An estimate of the direction or position of the dominant audio component is determined. Encoding initial output representation, weighting coefficients of dominant audio component, direction or position of the dominant audio component as a coded signal for reproduction.

EFFECT: technical result consists in improved efficiency of coding.

22 cl, 5 dwg

Description

FIELD OF THE INVENTION

[0001] The present invention provides systems and methods for producing an improved shape parametric binaural output signal, further utilizing head movement tracking.

Literature

[0002] Gundry, K., ʺ A New Matrix Decoder for Surround Sound, ʺ AES 19th International Conf., Schloss Elmau, Germany, 2001.

[0003] Vinton, M., McGrath, D., Robinson, C., Brown, P., “Next generation surround decoding and up-mixing for consumer and professional applications”, AES 57th International Conf, Hollywood, CA, USA, 2015.

[0004] Wightman, F. L., and Kistler, D. J. (1989). ʺHeadphone simulation of free-field listening. I. Stimulus synthesis, ʺ J. Acoust. Soc. Am. 85, 858-867.

[0005] ISO / IEC 14496-3: 2009 - Information technology - Coding of audio-visual objects - Part 3: Audio, 2009.

[0006] Mania, Katerina, et al. "Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity." Proceedings of the 1st Symposium on Applied perception in graphics and visualization. ACM, 2004.

[0007] Allison, R. S., Harris, L. R., Jenkin, M., Jasiobedzka, U., & Zacher, J. E. (2001, March). Tolerance of temporal delay in virtual environments. In Virtual Reality, 2001. Proceedings. IEEE (pp. 247-254). IEEE

[0008] Van de Par, Steven, and Armin Kohlrausch. "Sensitivity to auditory-visual asynchrony and to jitter in auditory-visual timing." Electronic Imaging. International Society for Optics and Photonics, 2000.

BACKGROUND OF THE INVENTION

[0009] Any discussion of the prior art of the invention throughout the description should in no way be construed as an admission that such a prior art is widely known or is part of ordinary general knowledge in the art.

[0010] Content creation, coding, distribution and reproduction of audio content is traditionally channel based. That is, one specific target reproduction system is contemplated for content flowing throughout the content ecosystem. Examples of such target playback systems are mono, stereo, 5.1, 7.1, 7.1.4, etc.

[0011] If the content is not to be played back on the system for which it is intended, downmix or upmix can be applied. For example, 5.1 content can be reproduced through a stereo reproduction system using certain known downmix equations. Another example is the playback of stereo content on a 7.1 speaker setup, which may include a so-called up-mix process, which may or may not be controlled by the information present in the stereo signal, such as that used by so-called matrix encoders such as Dolby Pro Logic. To control the up-mix process, information about the initial state of the signals before down-mix can be implicitly reported by introducing special phase relations into the down-mix equations, or, in other words, by applying the down-mix equations with complex values. A well-known example of such a downmix method using downmix coefficients with complex values for content with speakers in two dimensions is LtRt (Vinton et al., 2015).

[0012] The resulting (stereo) downmix signal can be played back through a stereo speaker system or can be mixed up for settings with surround speakers and / or front height speakers. The target location of the signal can be obtained by up-mixing from inter-channel phase relationships. For example, in the stereo representation of LtRt, a signal that does not match the phase (for example, having a normalized cross-correlation coefficient close to -1 for the shape of the inter-channel signal) should ideally be reproduced by one or more loudspeakers with an ambient sound effect, while positive a correlation coefficient (close to +1) indicates that the signal should be reproduced by the front speakers located in front of the listener.

[0013] A variety of upmix algorithms and strategies have been developed that differ in their strategies for reproducing a multi-channel signal from a stereo down mix. As for relatively simple boosters, the normalized cross-correlation coefficient of stereo signals is monitored as a function of time, while the signal (s) to the front or rear speakers is controlled depending on the value of the normalized cross-correlation coefficient. This approach works well for relatively simple content in which only one listening object is present at the same time. More advanced boost mixers are based on statistical information that is obtained from specific frequency domains to control the signal flow from the stereo input to the multi-channel outputs (Gundry 2001, Vinton et al., 2015). Specifically, a signal model based on an adjustable or dominant component and a residual (diffuse) stereo signal can be used in individual time / frequency split elements. In addition to evaluating the dominant component and the residual signals, the direction angle is also estimated (in azimuth, possibly increasing with elevation angle) and then the signal of the dominant component is adjusted for one or more loudspeakers in order to reconstruct (estimated) position during playback.

[0014] The use of matrix encoders and decoders / boosters is not limited to channel based content. Recent developments in the audio industry are based on audio objects rather than channels where one or more objects consist of an audio signal and associated metadata indicating, among other things, its target location as a function of time. As noted by Vinton et al., 2015, matrix encoders can also be used for such object-based audio content. In such a system, signals from objects are downmixed to a stereo representation using downmix coefficients depending on the positional metadata of the object.

[0015] Upmixing and reproduction of matrix encoded content are not necessarily limited to reproduction through speakers. Representation of an adjustable or dominant component, consisting of a dominant component signal and (target) location, enables playback through headphones through convolution with impulse responses associated with the head (HRIR) (Wightman et al., 1989). A simplified diagram of a system 1 implementing this method is shown in FIG. 1. Input signal 2 in the encoded matrix format 3 is first analyzed to determine the direction and magnitude of the dominant component. The dominant component signal is convolved 4, 5 by means of the HRIR pair obtained from the reference information 6 based on the direction of the dominant component to calculate the output signal for playback 7 through the headphones so that the reproduced signal is perceived as coming from the direction that was determined in step 3 of the dominant analysis Components. This scheme can be applied to wideband signals, as well as to individual subbands, and can be improved in various ways using specialized processing of residual (or diffuse) signals.

[0016] The use of matrix encoders is highly suitable for distribution and playback on AV receivers, but can be problematic for mobile applications requiring low data rates and low power consumption.

[0017] Regardless of whether channel-based or object-based content is used, matrix encoders and decoders rely on fairly accurate inter-channel phase relationships of signals that propagate from the matrix encoder to the decoder. In other words, the distribution format should largely preserve the waveform. This dependence on the preservation of the waveform can create problems in a limited bit rate environment where audio codecs use parametric methods rather than waveform coding tools to get the best sound quality. Examples of such parametric instruments that are generally known as waveformless are often referred to as spectral range replication, parametric stereo coding, spatial audio coding, etc., as they are used in MPEG 4 audio codecs (14496-3: 2009 ISO / IEC).

[0018] As briefly described in the previous section, up-mix consists of analyzing and regulating (or convolving HRIR) the signals. For network powered devices such as AV receivers, this is usually not a problem, but for battery powered devices such as mobile phones and tablets, the computational complexity and associated memory requirements associated with these processes are often undesirable due to their negative impact on battery life.

[0019] The above analysis usually also introduces additional audio delay. Such audio delay is undesirable because (1) it requires video delay in order to maintain synchronization of lip movement with a phonogram, which requires a significant amount of memory and processing power, and (2) such a delay can cause asynchrony / delay between head movements and rendering audio in case of tracking behind the movement of the head.

[0020] The matrix-coded down-mix may also not sound optimally on stereo speakers or headphones due to the potential presence of strongly mismatched signal components.

SUMMARY OF THE INVENTION

[0021] An object of the invention is to provide an improved form of a parametric binaural output signal.

[0022] In accordance with a first aspect of the present invention, there is provided a method of encoding an input audio signal based on a channel or an object for playback, said method including the steps of: (a) initially rendering the input audio signal based on a channel or an object in an initial output presentation (for example, an initial output presentation); (b) determining an estimate of the dominant audio component from an input audio signal based on a channel or an object, and determining a sequence of weights of the dominant audio component to map the initial output representation to the dominant audio component; (c) determine an estimate of the direction and position of the dominant audio component; and (d) encode the initial output representation, the weights of the dominant audio component, the direction or position of the dominant audio component as an encoded signal for playback. By providing a sequence of weights of the dominant audio component to map the initial output representation to the dominant audio component, it is possible to use the weights of the dominant audio component and the initial output representation to determine the estimate of the dominant component.

[0023] In some embodiments, the method further includes determining an estimate of the residual mix, which is the initial output representation minus the rendering of the dominant audio component or evaluation thereof. The method may also include generating an anechoic binaural mix of the input audio signal based on the channel or object, and determining an estimate of the residual mix, the estimate of the residual mix may be an anechoic binaural mix minus the rendering of the dominant audio component or its estimation. Additionally, the method may include determining a sequence of residual matrix coefficients for mapping the initial output representation into an estimate of the residual mix.

[0024] The initial output presentation may comprise a presentation via a speaker or headphones. An input audio signal based on a channel or an object can be divided into time and frequency breakdown elements, and the coding step can be repeated for a sequence of time steps and a sequence of frequency ranges. The initial output representation may comprise a stereo loudspeaker mix.

[0025] In accordance with an additional aspect of the present invention, there is provided a method for decoding an encoded audio signal, wherein the encoded audio signal includes: a first (eg, initial) output presentation (eg, first / initial output presentation); direction of the dominant audio component and weights of the dominant audio component; wherein the method comprises the steps of: (a) using weights of the dominant audio component and an initial output representation to determine the estimated dominant component; (b) rendering the estimated dominant component using binauralization at a spatial location relative to the target listener in accordance with the direction of the dominant audio component to form a rendered binauralized estimated dominant component; (c) reconstructing the estimate of the residual component from the first (e.g., initial) output representation; and (d) combining a rendered binauralized estimated dominant component and a residual component estimate to form an output spatially oriented encoded audio signal.

[0026] The encoded audio signal may further include a sequence of residual matrix coefficients representing the residual audio signal, and step (c) may further comprise step (c1) in which residual matrix coefficients are applied to the first (eg, initial) output representation to reconstruct assessment of the residual component.

[0027] In some embodiments, the residual component estimate may be reconstructed by subtracting the rendered binauralized estimated dominant component from the first (eg, initial) output representation. Step (b) may include an initial rotation of the estimated dominant component in accordance with an input signal tracking the head movement indicating the orientation of the head of the target listener.

[0028] In accordance with a further aspect of the present invention, there is provided a method for decoding and reproducing an audio stream for a listener using headphones, said method comprising the steps of: (a) receiving a data stream comprising a first audio presentation and additional audio conversion data; (b) receiving head orientation data representing the orientation of the listener; (c) creating one or more auxiliary signals based on the first audio presentation and received transform data; (d) create a second audio presentation, consisting of combining the first audio presentation and an auxiliary signal (s), in which one or more auxiliary signals were modified in response to head orientation data; and (e) outputting a second audio presentation as an output audio stream.

[0029] Some embodiments may further include modifying auxiliary signals, which consists of modeling the acoustic path from the position of the sound source to the ears of the listener. The transform data may consist of matrix coefficients and at least one of the position of the sound source and the direction of the sound source. The conversion process can be applied as a function of time or frequency. Auxiliary signals may represent at least one dominant component. The position or direction of the sound source can be taken as part of the conversion data and can be rotated in response to the head orientation data. In some embodiments, the maximum amount of rotation is limited to less than 360 degrees in azimuth or elevation. Secondary presentation can be obtained from the first presentation by matrixing in the transformation area or a set of filters. The transform data may further comprise additional matrixing coefficients, and step (d) may further comprise modifying the first audio presentation in response to additional matrixing coefficients before combining the first audio presentation and the auxiliary audio signal (s).

Brief Description of the Drawings

[0030] Now, only by way of example, embodiments of the invention will be described with reference to the accompanying drawings, in which:

[0031] FIG. 1 schematically illustrates a headphone decoder for matrix encoded content;

[0032] FIG. 2 schematically illustrates an encoder according to an embodiment;

[0033] FIG. 3 is a block diagram of a decoder;

[0034] FIG. 4 is a detailed visualization of an encoder; and

[0035] FIG. 5 illustrates in more detail one form of decoder.

The implementation of the invention

[0036] Embodiments show a system and method for presenting audio content based on a channel or an object that is (1) compatible with stereo playback, (2) allows binaural playback, including tracking head movement, (3) has little decoder complexity, and (4) not based, but nonetheless compatible with matrix coding.

[0037] This is achieved by combining the encoder-side analysis of one or more dominant components (or a dominant object or a combination thereof), including weights for predicting these dominant components from the down mix, in combination with additional parameters that minimize the error between binaural rendering based on only regulated or dominant components and the desired binaural representation of the full content.

[0038] In an embodiment, analysis of a dominant component (or multiple dominant components) is provided at the encoder, and not at the decoder / renderer. The audio stream then builds up with metadata indicating the direction of the dominant component, and information on how the dominant component (s) can be obtained from the accompanying down mix signal.

[0039] FIG. 2 shows one form of encoder 20 of a preferred embodiment. Content 21 based on an object or channel undergoes analysis 23 to determine the dominant component (s). This analysis can take place as a function of time and frequency (it is assumed that the audio content is divided into time elements and frequency sub-elements). The result of this process is a dominant component signal 26 (or multiple dominant component signals) and associated information 25 about position (s) or direction (s). Next, a score of 24 is made and 27 weights are derived to allow reconstruction of the dominant component signal (s) from the transmitted down mix. This downmix generator 22 does not have to adhere strictly to the LtRt downmix rules, but may be a standard ITU downmix (LoRo) using non-negative, valid downmix coefficients. Finally, the output of the down mix 29, weight 27, and positional data 25 are packaged by audio encoder 28 and prepared for distribution.

[0040] FIG. 3 shows a corresponding decoder 30 of a preferred embodiment. The audio decoder reconstructs the down mix signal. The signal is inputted 31 and decompressed by an audio decoder 32 into a down mix signal, the weight and direction of the dominant components. Further, the weights of the estimated dominant components are used to reconstruct 34 adjustable components, which are rendered 36 using positional data or directional data. Positional data, as an option, may be modified 33 depending on the rotation of the head or information 38 conversion. Additionally, reconstructed dominant component (s) can be subtracted 35 from the down mix. Alternatively, there is a subtraction of the dominant component (s) within the down mix path, but alternatively, the subtraction can also occur in the encoder, as described below.

[0041] In order to improve the removal or cancellation of the reconstructed dominant component in the subtractor 35, the output signal of the dominant component can first be rendered using transmitted positional data or directional data before subtraction. This optional rendering step 39 is shown in FIG. 3.

, когда мы опускаем соответствующие временные и частотные индексы; соответствующее положение в пределах текущего кадра задается единичным вектором

, и индекс i относится к номеру объекта, а индекс n относится ко времени (например, индекс выборки поддиапазона). Входные сигналы

объекта являются примером входного аудиосигнала, основывающегося на канале или объекте.[0042] Returning now to first describe the encoder in more detail, in FIG. 4 illustrates one form of encoder 40 for processing object-based audio content (e.g., Dolby Atmos system). Audio objects are initially stored as 41 Atmos objects and are initially divided into time and frequency elements using a set of 42 hybrid complex-valued quadrature mirror filters (HCQMF). The input signals of objects can be denoted as

when we omit the corresponding time and frequency indices; the corresponding position within the current frame is set by a unit vector

, and index i refers to the object number, and index n refers to time (for example, a subband sample index). Input signals

objects are an example of an input audio signal based on a channel or an object.

) создают 43, используя скаляры с комплексными значениями

(например, однополюсные HRTF 48), которые представляют презентацию поддиапазона для HRIR, соответствующих положению

:[0043] Anechoic, subband, binaural mix Y (

) create 43 using scalars with complex values

(e.g., single pole HRTF 48) that represent a subband presentation for HRIRs corresponding to the position

:

) может быть создан посредством свертки, используя связанные с головой импульсные реакции (HRIR). Дополнительно, стерео даун-микс

(как пример, реализующий начальное выходное представление) создают 44, используя коэффициенты

усиления амплитудного панорамирования:[0044] Alternatively, binaural mix Y (

) can be created by convolution using head impulse responses (HRIR). Optional stereo down mix

(as an example that implements the initial output representation) create 44 using the coefficients

amplification of amplitude panning:

(в качестве примера реализующий направление или положение доминантного аудиокомпонента) может оцениваться путем вычисления доминантного компонента 45, первоначально вычисляя взвешенную сумму единичных векторов направления для каждого объекта:[0045] The direction vector of the dominant component

(as an example, realizing the direction or position of the dominant audio component) can be estimated by calculating the dominant component 45, initially calculating the weighted sum of unit direction vectors for each object:

- энергия сигнала

:Where

- signal energy

:

and (.) * is the complex convolution operator.

[0046] The dominant / control signal d [ n ] (which implements the dominant audio component as an example) is further defined as follows:

- функция, выполняющая усиление, которое уменьшается с увеличением расстояния между единичными векторами

. Например, чтобы создать виртуальный микрофон с помощью модели направленности, основанной на сферических гармониках высшего порядка, одна из реализаций должна соответствовать следующему:[0047] where

- a function that performs amplification, which decreases with increasing distance between unit vectors

. For example, to create a virtual microphone using a directivity model based on higher-order spherical harmonics, one of the implementations should correspond to the following:

- единичный вектор направления в двух- или трехмерной системе координат,(.) - оператор скалярного произведения двух векторов, и a, b, c - примерные параметры (например a=b=0,5; c=1).Where

is the unit direction vector in a two- or three-dimensional coordinate system, (.) is the scalar product operator of two vectors, and a, b, c are approximate parameters (for example, a = b = 0.5; c = 1).

:[0048] The weights or prediction coefficients w _{l , d} , w _{r , d} are calculated 46 and used to calculate 47 the estimated adjustable signal

:

, заданными сигналами

даун-микса. Веса w _{l , d} , w _{r , d} являются примером весовых коэффициентов доминантного аудиокомпонента для отображения начального выходного представления (например,

) в доминантный аудиокомпонент (например,

). Известный способ получения этих весов заключается в применении устройства прогнозирования минимальной среднеквадратичной ошибки (MMSE):where the weights w _{l , d} , w _{r , d} minimize the mean square error between

set by signals

down mix. The weights w _{l , d} , w _{r , d} are an example of the weights of the dominant audio component to display the initial output representation (for example,

) into the dominant audio component (e.g.

) A known method of obtaining these weights is the use of a minimum mean square error prediction (MMSE) device:

where R _ab is the covariance matrix between signals for signals a and signals b , and ∈ is the regularization parameter.

из безэхового бинаурального микса

, чтобы создать остаточный бинауральный микс

, используя HRTF (HRIR)

50, связанный с направлением/положением

доминантного компонентного сигнала

:[0049] We can then subtract 49 rendered estimates of the dominant component signal

from anechoic binaural mix

to create a residual binaural mix

using HRTF (HRIR)

50 associated with direction / position

dominant component signal

:

из стереомикса

,используя оценочные минимальные среднеквадратичные ошибки:[0050] Finally, 51 different sets of prediction coefficients or weights w _{i , j} that allow reconstruction of the residual binaural mix are evaluated.

from stereo mix

using estimated minimum standard errors:

) в оценочный остаточный бинауральный микс

. Приведенное выше выражение может быть подвергнуто дополнительным ограничениям уровня, чтобы преодолеть любые потери прогнозирования. Кодер выводит следующую информацию:where R _ab is the covariance matrix between the signals for presentation a and presentation b , and ∈ is the regularization parameter. Prediction coefficients or weights w _{i , j} are an example of residual matrix coefficients for displaying an initial output representation (e.g.,

) into the estimated residual binaural mix

. The above expression may be subject to additional level restrictions to overcome any prediction loss. The encoder displays the following information:

(в качестве примера реализации начального выходного представления);[0051] Stereomix

(as an example of the implementation of the initial output representation);

[0052] Odds for evaluating the dominant componentw _{l , d} , w _{r , d} (as an example of implementing the weighting coefficients of the dominant audio component) are;

;[0053] the position or direction of the dominant component

;

[0054] and, optionally, the residual weights w _{i , j} (as an example of the implementation of the residual matrix coefficients).

[0055] Although the above description relates to rendering based on one single dominant component, in some embodiments, the encoder may be configured to detect multiple dominant components, determine weights and directions for each of the many dominant components, render and subtract each of the many the dominant components from the anechoic binaural mix Y, and then determine the residual weights after each of the many dominant components has been subtracted from the anechoic binaural mix Y.

Decoder / Renderer

для вывода слушателю 71 из распакованной входной информации z _l , z _r ; w _{l , d} , w _{r , d} ;

, w _{i , j} . Здесь стерео микс z _l , z _r является примером первой аудиопрезентации и коэффициенты или веса предсказания w _{i , j} и/или направление/положение

доминантного компонентного сигнала

являются примерами дополнительных данных аудиопреобразования.[0056] FIG. 5 shows in more detail one form of decoder / renderer 60. Decoder / renderer 60 employs a process aimed at reconstructing a binaural mix

to output to the listener 71 from the unpacked input informationz _l z _r ; w _{l , d} , w _{r , d} ;

, w _{i , j} . Here is a stereo mixz _l z _r is an example of a first audio presentation and prediction coefficients or weightsw _{i , j} and / or direction / position

dominant component signal

are examples of additional audio conversion data.

вычисляется 63, используя веса w _{l , d} , w _{r , d} коэффициентов предсказания:[0057] Initially, a stereo down mix is split into time / frequency elements using an appropriate filter set or transform 61, such as the HCQMF analytic group 61. Other transformations, such as a discrete Fourier transform, a (modified) cosine or sine transform, a set of filters in time domain or wavelet transform can also be equally applicable. Subsequently, the estimated dominant component signal

63 is calculated using the weights w _{l , d} , w _{r , d of} the prediction coefficients:

=w _{l , d} z _l +w _{r , d} z _r

= w _{l , d} z _l + w _{r , d} z _r

является примером вспомогательного сигнала. Следовательно, можно сказать, что этот этап соответствует созданию одного или более вспомогательных сигналов, основанных на упомянутой первой аудиопрезентации и принятых данных преобразования.Estimated Dominant Component Signal

is an example of an auxiliary signal. Therefore, it can be said that this step corresponds to the creation of one or more auxiliary signals based on said first audio presentation and received conversion data.

, возможно, модифицированных (повернутых) на основе информации, полученной из устройства 62 слежения за головой. Наконец, общий приглушенный бинауральный выходной сигнал состоит из отрендеренного доминантного компонентного сигнала, суммированного 66 с реконструированными остатками

, основанными на весах w _ij коэффициентов предсказания:[0058] This dominant component signal is further rendered 65 and modified 68 by HRTF 69 based on the transmitted position / direction data

possibly modified (rotated) based on information received from head tracking device 62. Finally, the overall muted binaural output signal consists of a rendered dominant component signal summed 66 with reconstructed residues

based on the weights w _ij prediction coefficients:

A full muted binaural output is an example of a second audio presentation. Therefore, this stage can be said to correspond to the creation of a second audio presentation consisting of a combination of said first audio presentation and said auxiliary signal (s), in which one or more of these auxiliary signals were modified in response to said head orientation data.

[0059] Additionally, it should be noted that if information about more than one dominant signal is received, each dominant signal can be rendered and added to the reconstructed residual signal.

должны быть очень близки (с точки зрения среднеквадратичной ошибки) к опорным бинауральным сигналам

, пока[0060] As long as no rotation or movement of the head is applied, the output signals

should be very close (in terms of standard error) to reference binaural signals

, till

Basic properties

, w _{i , j} и поворота и/или перемещения устройства слежения за головой. Это указывает, что сложность процесса является относительно низкой, поскольку анализ доминантных компонент применяется в кодере вместо декодера.[0061] As can be seen from the above equations, an effective operation for creating a muted binaural representation from a stereo representation consists of a 2 × 2 70 matrix in which the matrix coefficients depend on the transmitted information w _{l , d} , w _{r , d} ;

, w _{i , j} and turning and / or moving the head tracking device. This indicates that the complexity of the process is relatively low, since the analysis of dominant components is used in the encoder instead of the decoder.

[0062] If no dominant component is evaluated (for example, w _{l , d} , w _{r , d} = 0), the described solution is equivalent to the parametric binaural method.

[0063] In cases where there is a desire to exclude certain objects from tracking the rotation / movement of the head, these objects can be excluded from (1) analysis of the direction of dominant components, and (2) prediction of dominant component signals. As a result, these objects will be transformed from stereo to binaural by means of coefficientsw _{i , j} and therefore they are not affected by any rotation or movement of the head.

[0064] With such a train of thought, objects can be set in a “pass through” mode, which means that in the binaural representation they will be subjected to amplitude panning rather than HRIR convolution. This can be obtained simply by using the amplitude pan gain for the H _i coefficients instead of a single pole HRTF or any other appropriate binaural process.

Extensions

[0065] Embodiments are not limited to the use of down mixes, since samples of other channels may also be used.

[0066] The decoder 60 described with reference to FIG. 5 has an output signal consisting of the rendered direction of the dominant component plus an input signal matriced by matrix coefficients w _{i , j} . The latest coefficients can be obtained in various ways, for example:

. Другими словами, в этой реализации, коэффициенты w _{i , j} направлены на точную реконструкцию бинауральных сигналов

, которые могли бы быть получены при рендеринге первоначальных входных объектов/каналов бинауральным образом; другими словами, коэффициенты w _{i , j} управляются контентом.[0067] 1. The coefficients w _{i , j} can be determined in the encoder by parametric reconstruction of the signals

. In other words, in this implementation, the coefficients w _{i , j are} aimed at the exact reconstruction of binaural signals

that could be obtained by rendering the original input features / channels in a binaural way; in other words, the coefficients w _{i , j are} controlled by content.

[0068] 2. The coefficients w _{i , j} can be transmitted from the encoder to the decoder to represent HRTF for certain spatial positions, for example, with angles of +/- 45 degrees in azimuth. In other words, the residual signal is processed to simulate reproduction through two virtual speakers at specific locations. Because these HRTF coefficients are transmitted from the encoder to the decoder, the locations of the virtual speakers can vary in time and frequency. If this approach is used using static virtual speakers to represent the residual signal, the coefficients w _{i , j} do not require transmission from the encoder to the decoder, and can instead be connected by wires in the decoder. A variant of this approach may consist of a limited set of static locations that are available in the decoder, with their respective coefficients w _{i , j} , and their selection, in which a static location is used to process the residual signal, is reported from the encoder to the decoder.

могут подвергаться так называемому повышающему микшированию, реконструируя более 2 сигналов посредством статистического анализа этих сигналов в декодере с последующим бинауральным рендерингом результирующих сигналов повышающего микширования.[0069] Signals

can undergo the so-called up-mix, reconstructing more than 2 signals by statistical analysis of these signals in the decoder, followed by binaural rendering of the resulting up-mix signals.

[0070] The described methods may also be applicable in a system in which the transmitted signal Z is a binaural signal. In this particular case, the decoder 60 shown in FIG. 5 remains as it is, while the block labeled “Generate stereo (LoRo) mix” 44 and shown in FIG. 4 should be replaced with “Generate anechoic binaural mix” (generate a muted binaural mix) 43 (Fig. 4), which is the same as the block that creates the signal pair Y. Additionally, in accordance with the requirements, other forms of mixes can be generated.

[0071] This approach can be expanded using methods for reconstructing one or more input FDN signals from a transmitted stereo mix, which consists of a specific subset of objects or channels.

[0072] The approach can be extended using multiple dominant components predicted from the transmitted stereo mix and rendered on the side of the decoder. There is no fundamental restriction on predicting only one dominant component for each time / frequency partition element. In particular, the number of dominant components may vary in each time / frequency partition element.

Interpretation

[0073] Throughout this specification, a reference to “one embodiment”, “some embodiments” or “an embodiment” means that a particular feature, structure, or characteristic described in conjunction with an embodiment is entered in at least one embodiment the implementation of the present invention. Thus, the appearance of the expressions “in one embodiment”, “in some embodiments” or “in an embodiment” is not necessarily universally available in various places in the present description, since they all refer to the same embodiment. In addition, specific features, structures, or characteristics may be combined in any suitable manner, as should be apparent to one skilled in the art, based on this disclosure, in one or more embodiments.

[0074] Using the ordinal numbers "first", "second", "third", etc. to describe a common object, as used here, unless otherwise specified, simply indicates that reference is made to different instances of similar objects, and it is not intended to imply that the objects described in this way should follow in the given sequence in time or space, in order, or in any other way.

[0075] In the following claims and in the description herein, any of the terms “comprising”, “comprising” or “which contains” is an open term that means including at least elements / features corresponding to the term, but does not exclude others. Thus, the term “comprising”, when used in the claims, should not be construed as limiting to the means, elements, or steps listed hereinafter. For example, the expression “device containing A and B” should not be limited to devices consisting of elements A and B. Any of the terms “including” or “which includes” or “which include”, as these terms used here are also open terms, which mean the inclusion of at least elements / features corresponding to the term, but do not exclude others. Thus, “including” is synonymous with “including”.

[0076] The term "exemplary," as used here, is used in the sense of providing examples, and not as an indication of quality. That is, an “exemplary embodiment” is an embodiment presented as an example, and is not necessarily an embodiment exemplary in quality.

[0077] It should be understood that in the above description of exemplary embodiments of the invention, various features of the invention, in order to optimize the disclosure and assist in understanding one or more different inventive approaches, are sometimes grouped together into a single embodiment, drawing or description thereof. This disclosure method, however, should not be interpreted as a reflection of the intention that the claimed invention requires more features than is explicitly provided in each claim. Rather, as the following claims reflect, aspects of the invention are less than all features of a single prior disclosed embodiment. Thus, the claims following the section “Implementation of the invention” are hereby expressly included in this “Implementation of the invention” with each independent claim as a separate embodiment of this invention.

[0078] Additionally, although some of the embodiments described herein contain some, but not other features included in other embodiments, combinations of features in various embodiments mean that they are within the scope of the invention and form various embodiments, like this should understand the experts in this field of technology. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0079] Further, some of the embodiments are described herein as a method or combination of method elements that may be implemented by a computer system processor or other means of performing a function. Thus, a processor with the necessary instructions to execute such a method or method element forms a means of executing a method or method element. Additionally, an element of an embodiment of a device described herein is an example of a means of performing a function executed by an element for the purpose of carrying out the invention.

[0080] Numerous specific details are set forth herein. However, it should be understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and technologies have not been shown in detail so as not to interfere with the understanding of the present description.

[0081] Similarly, it should be noted that the term “coupled”, when used in the claims, should not be interpreted as limiting to direct links only. The terms “coupled” and “connected”, as well as their derivatives, may be used. It should be understood that these terms are not implied by each other's synonyms. Thus, the context of the expression “Device A associated with device B” should not be limited to devices or systems in which the output of device A is directly connected to the input of device B. This means that there is a path between the output of device A and the input of device B, which may be by passage containing other devices or means. “Connected” may mean that two or more elements are in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but are still working together or interacting with each other.

[0082] Thus, although embodiments of the invention have been described herein, those skilled in the art should recognize that other and additional modifications may be made therein without departing from the spirit of the invention, and it is understood that all such changes and modifications are claimed to be falling within the scope of the invention. For example, any of the formulas above are simply representative of the procedures that can be used. Functionality may be added or removed from flowcharts, and operations may alternate between function blocks. In the methods described within the scope of the present invention, the steps may be added or removed.

[0083] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE):

EEE 1. An encoding method for reproducing an input audio signal based on a channel or an object, said method comprising the steps of:

(a) initially rendering an input audio signal based on a channel or an object into an initial output representation;

(b) determining an estimate of the dominant audio component from an input audio signal based on a channel or an object, and determining a sequence of weights of the dominant audio component to map the initial output representation to the dominant audio component;

(c) determine an estimate of the direction or position of the dominant audio component; and

(d) encode the initial output representation, the weights of the dominant audio component, the direction or position of the dominant audio component as an encoded signal for playback.

EEE 2. The method of claim EEE 1, further comprising determining an estimate of the residual mix, which is the initial output representation, reduced by rendering the dominant audio component or evaluating it.

EEE 3. The method of claim EEE 1, further comprising generating an anechoic binaural mix of the input audio signal based on a channel or an object, and determining an estimate of the residual mix, the estimation of the residual mix being an anechoic binaural mix minus the rendering of the dominant audio component or its estimation.

EEE 4. The method of claim EEE 2 or 3, further comprising determining a sequence of residual matrix coefficients for mapping the initial output representation into an estimate of the residual mix.

EEE 5. The method according to any of the preceding EEE, wherein said initial output presentation comprises headphones or a speaker.

EEE 6. The method according to any of the preceding EEEs, wherein said channel or object based audio signal is broken down into elements by time and frequency, and said coding step is repeated with respect to a sequence of time steps and sets of frequency bands.

EEE 7. A method according to any of the preceding EEE, wherein said initial output representation comprises a stereo speaker mix.

EEE 8. A method for decoding an encoded audio signal, wherein the encoded audio signal comprises:

- first output presentation;

- weighting coefficients of the dominant audio component and the direction of the dominant audio component;

a method comprising the steps of:

(a) using weights of the dominant audio component and the initial output representation to determine the estimated dominant component;

(b) rendering the estimated dominant component with binauralization at a spatial location relative to the target listener in accordance with the direction of the dominant audio component to form a rendered binauralized estimated dominant component;

(c) reconstructing an estimate of the residual component from the first output representation; and

(d) combine the rendered binauralized estimated dominant component and the residual component estimate to form an output spatial encoded audio signal.

EEE 9. The method of claim EEE 8, wherein said encoded audio signal further comprises a sequence of residual matrix coefficients representing the residual audio signal, and said step (c) is further a step in which:

(c1) applying said residual matrix coefficients to the first output representation to reconstruct the residual component estimate.

EEE 10. The method of EEE 8, in which the estimate of the residual component is reconstructed by subtracting the rendered binauralized estimated dominant component from the first output representation.

EEE 11. The method according to EEE 8, wherein said step (b) includes initial rotation of the estimated dominant component in accordance with an input signal tracking the head movement indicating the orientation of the head of the target listener.

EEE 12. A method for decoding and reproducing an audio stream for a listener using headphones, said method comprising the steps of:

(a) receiving a data stream comprising a first audio presentation and additional audio transform data;

(b) receiving head orientation data representing the orientation of the listener;

(c) creating one or more auxiliary signals based on said first audio presentation and received conversion data;

(d) creating a second audio presentation consisting of a combination of the first audio presentation and said auxiliary signal (s), in which one or more of the auxiliary signals have been modified in response to said head orientation data; and

(e) outputting a second audio presentation as an output audio stream.

EEE 13. The method according to p. EEE 12, in which the modification of auxiliary signals consists of modeling the acoustic path from the position of the sound source to the ears of the listener.

EEE 14. The method of claim EEE 12 or 13, wherein said transform data consists of matrix coefficients and at least one of the following: position of the sound source or direction of the sound source.

EEE 15. The method according to any one of paragraphs. EEE 12-14, in which the conversion process is applied as a function of time or frequency.

EEE 16. The method according to any one of EEE 12-15, wherein the auxiliary signals are at least one dominant component.

EEE 17. The method according to any one of paragraphs. EEE 12-16, in which the position or direction of the sound source, adopted as part of the conversion data, rotates the response to the head orientation data.

EEE 18. The method of claim EEE 17, wherein the maximum amount of rotation is limited to less than 360 degrees in azimuth or elevation.

EEE 19. The method according to any one of paragraphs. EEE 12-18, in which a secondary presentation is obtained from the first presentation by matrixing in a transform field or filter block.

EEE 20. The method according to any one of paragraphs. EEE 12-19, wherein the transform data further comprises additional matrixing coefficients and step (d) further comprises modifying the first audio presentation in response to the additional matrixing coefficients before combining the first audio presentation and the auxiliary audio signal (s).

EEE 21. A device containing one or more other devices, configured to implement any of the methods according to claims. EEE 1-20.

EEE 22. A computer-readable medium containing a program consisting of instructions that, when executed by one or more processors, cause one or more devices to execute the method according to any one of claims. EEE 1-20.

Claims (39)

1. A method of encoding an input audio signal based on a channel or an object for reproduction, which method comprises the steps of (a) perform initial rendering of an input audio signal based on a channel or an object into an initial output representation; (b) determining an estimate of the dominant audio component from an input audio signal based on a channel or an object, and determining the sequence of weight components of the dominant audio component to map the initial output representation to the dominant audio component so as to enable the use of the weights of the dominant audio component and the initial output representation to determine the estimate dominant component; (c) determine an estimate of the direction or position of the dominant audio component; and (d) encode the initial output representation, the weights of the dominant audio component, the direction or position of the dominant audio component as an encoded signal for playback. 2. The method of claim 1, further comprising determining an estimate of the residual mix, which is the initial output representation minus the rendering of either the dominant audio component or its estimation. 3. The method of claim 1, further comprising generating an anechoic binaural mix of the input audio signal based on the channel or object, and determining an estimate of the residual mix, the estimate of the residual mix being the anechoic binaural mix minus either rendering the dominant audio component, or its assessment. 4. The method of claim 2 or 3, further comprising determining a sequence of residual matrix coefficients for mapping the initial output representation into an estimate of the residual mix. 5. The method according to any one of the preceding paragraphs, in which the initial output presentation comprises a presentation via headphones or a speaker. 6. The method according to any one of the preceding paragraphs, in which the input audio signal based on a channel or an object is divided into elements of the division of time and frequency, and said coding step is repeated with respect to a sequence of time steps and a sequence of frequency bands. 7. The method according to any one of the preceding paragraphs, in which the initial output representation comprises a mix of stereo speakers. 8. A method for decoding an encoded audio signal, wherein the encoded audio signal includes: initial output presentation; weighting factors of the dominant audio component and the direction of the dominant audio component; wherein the method comprises the steps in which: (a) using weights of the dominant audio component and the initial output representation to determine the estimated dominant component; (b) rendering the estimated dominant component with binauralization at a spatial location relative to the target listener in accordance with the direction of the dominant audio component to form a rendered binauralized estimated dominant component; (c) reconstructing an estimate of the residual component from the initial output representation; and (d) combine the rendered binauralized estimated dominant component and the residual component estimate to form an output spatially oriented encoded audio signal. 9. The method of claim 8, wherein the encoded audio signal further includes a sequence of residual matrix coefficients representing the residual audio signal, and step (c) further comprises the step of (c1) apply said residual matrix coefficients to the initial output representation to reconstruct the estimate of the residual component. 10. The method of claim 8, wherein the residual component estimate is reconstructed by subtracting the rendered binauralized estimated dominant component from the initial output representation. 11. The method according to any one of paragraphs. 8-10, in which step (b) includes an initial rotation of the estimated dominant component in accordance with an input signal tracking the head movement indicating the orientation of the head of the target listener. 12. A method of decoding and reproducing an audio stream for a listener using headphones, which method comprises the steps of (a) receiving a data stream comprising a first audio presentation and additional audio conversion data; (b) receiving head orientation data representing the orientation of the listener; (c) creating one or more auxiliary signals based on the first audio presentation and received conversion data; (d) creating a second audio presentation consisting of a combination of the first audio presentation and an auxiliary signal (s), where one or more auxiliary signals are modified in response to the head orientation data; and (e) outputting a second audio presentation as an output audio stream. 13. The method according to p. 12, in which said modification of the auxiliary signals consists of modeling the acoustic path from the position of the sound source to the ears of the listener. 14. The method of claim 12 or 13, wherein said transform data consists of matrix coefficients and at least one of a position of the sound source and a direction of the sound source. 15. The method according to any one of paragraphs. 12-14, in which the conversion process is applied as a function of time or frequency. 16. The method according to any one of paragraphs. 12-15, wherein said auxiliary signals represent at least one dominant component. 17. The method according to any one of paragraphs. 12-16, in which the position or direction of the sound source, adopted as part of the conversion data, is rotated in response to head orientation data. 18. The method according to p. 17, in which the maximum amount of rotation is limited to less than 360 degrees in azimuth or elevation. 19. The method according to any one of paragraphs. 12-17, in which the secondary presentation is obtained from the first presentation by matrixing in the transform area or filter block. 20. The method according to any one of paragraphs. 12-19, wherein the transform data further comprises additional matrix coefficients, and step (d) further comprises modifying the first audio presentation in response to these additional matrix coefficients, before combining the first audio presentation and the auxiliary audio signal (s). 21. Equipment containing one or more devices configured to implement the method according to any one of paragraphs. 1-20. 22. Machine-readable medium containing a program consisting of instructions that, when executed by one or more processors, instruct one or more devices to perform the method according to any one of claims. 1-20.

Images