RU2818687C2 - Head tracking system and method for obtaining parametric binaural output signal - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to computer engineering for processing audio data. Technical result is achieved by rendering an input audio signal based on a channel or an object into an initial output representation; determining an estimate of a dominant audio component from an input audio signal based on a channel or an object, which includes: determining a sequence of weight components of a dominant audio component for mapping an initial output representation to a dominant audio component and determining an estimate of the dominant audio component based on the weight coefficients of the dominant audio component and the initial output representation; determining the direction or position of the dominant audio component; and encoding the initial output representation, weight coefficients of the dominant audio component and at least one of the direction and position of the dominant audio component as the encoded signal for reproduction.

EFFECT: elimination of asynchrony/delay between head movements and audio rendering in case of head tracking.

8 cl, 5 dwg

Description

Field of technology to which the invention relates

[0001] The present invention provides systems and methods for producing an improved parametric binaural output signal, further utilizing head tracking.

Literature

[0002] Gundry, K., “A New Matrix Decoder for Surround Sound,” AES 19th International Conf., Schloss Elmau, Germany, 2001.

[0003] Vinton, M., McGrath, D., Robinson, C., Brown, P., “Next generation surround decoding and up-mixing for consumer and professional applications,” AES 57th International Conf, Hollywood, CA, USA, 2015.

[0004] Wightman, F. L., and Kistler, D. J. (1989). ʺHeadphone simulation of free-field listening. I. Stimulus synthesis, J. Acoust. Soc. Am. 85, 858-867.

[0005] ISO/IEC 14496-3:2009 - Information technology -- Coding of audio-visual objects -- Part 3: Audio, 2009.

[0006] Mania, Katerina, et al. "Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity." Proceedings of the 1st Symposium on Applied perception in graphics and visualization. ACM, 2004.

[0007] Allison, R. S., Harris, L. R., Jenkin, M., Jasiobedzka, U., & Zacher, J. E. (2001, March). Tolerance of temporal delay in virtual environments. In Virtual Reality, 2001. Proceedings. IEEE (pp. 247-254). IEEE.

[0008] Van de Par, Steven, and Armin Kohlrausch. "Sensitivity to auditory-visual asynchrony and to jitter in auditory-visual timing." Electronic Imaging. International Society for Optics and Photonics, 2000.

BACKGROUND OF THE INVENTION

[0009] Any discussion of the prior art of the invention throughout the specification should in no way be construed as an admission that such prior art is widely known or forms part of the usual general knowledge in the art.

[0010] Content creation, encoding, distribution and playback of audio content has traditionally been channel-based. That is, one specific target playout system is envisioned for content flowing throughout the content ecosystem. Examples of such target playback systems are mono, stereo, 5.1, 7.1, 7.1.4, etc. systems.

[0011] If content is to be played on a system other than the one for which it was intended, downmixing or upmixing may be applied. For example, 5.1 content can be played through a stereo playback system using certain known downmix equations. Another example is the playback of stereo content on a 7.1 speaker setup, which may contain a so-called upmixing process that may or may not be controlled by information present in the stereo signal, such as is used by so-called matrix encoders such as Dolby Pro Logic. To control the upmixing process, information about the initial state of the signals before downmixing can be communicated implicitly by introducing special phase relationships into the downmixing equations, or, put another way, using complex-valued downmixing equations. A well-known example of such a downmix technique that uses complex-valued downmix coefficients for content with speakers located in two dimensions is LtRt (Vinton et al., 2015).

[0012] The resulting downmixed (stereo) signal may be played through a stereo speaker system or may be upmixed for installations with surround speakers and/or front height speakers. The target signal location can be obtained by upmixing from the inter-channel phase relationships. For example, in the LtRt stereo representation, a signal that is out of phase (e.g., having a normalized cross-correlation coefficient close to -1 for the inter-channel waveform) should ideally be reproduced by one or more surround speakers, whereas a positive the correlation coefficient (close to +1) indicates that the signal should be reproduced by the front speakers located in front of the listener.

[0013] A variety of upmixing algorithms and strategies have been developed that differ in their strategies for recreating a multi-channel signal from a stereo downmix. For relatively simple upmixers, the normalized cross-correlation coefficient of stereo signals is tracked as a function of time, while the signal(s) to the front or rear speakers is adjusted depending on the value of the normalized cross-correlation coefficient. This approach works well for relatively simple content that only has one listening target present at a time. More advanced upmixers rely on statistical information derived from specific frequency regions to control the signal flow from the stereo input to the multichannel outputs (Gundry 2001, Vinton et al. 2015). Specifically, a signal model based on the adjustable or dominant component and the residual (diffuse) stereo signal may be used in the individual time/frequency bins. In addition to estimating the dominant component and residual signals, the directional angle (in azimuth, possibly increasing with elevation) is also estimated and the dominant component signal is subsequently adjusted for one or more loudspeakers to reconstruct the (estimated) position during playback.

[0014] The use of matrix encoders and decoders/upmixers is not limited to channel-based content. Recent developments in the audio industry are based on audio objects rather than channels, where one or more objects consist of an audio signal and associated metadata indicating, among other things, its target location as a function of time. As noted in Vinton et al., 2015, matrix encoders can also be used for such object-based audio content. In such a system, object signals are downmixed into a stereo signal representation using downmix coefficients dependent on the object's positional metadata.

[0015] Upmixing and playback of matrix encoded content is not necessarily limited to playback through loudspeakers. The representation of a regulated or dominant component, consisting of a dominant component signal and a (target) location, enables headphone recall through convolution with head-related impulse responses (HRIR) ( Wightman et al., 1989 ). A simplified diagram of a system 1 implementing this method is shown in FIG. 1. The input signal 2 in coded matrix format is first analyzed 3 to determine the direction and magnitude of the dominant component. The dominant component signal is convolved 4, 5 by the HRIR pair obtained from reference information 6 based on the direction of the dominant component to calculate an output signal for playback 7 through headphones, such that the playback signal is perceived as coming from the direction that was determined in step 3 of the dominant component analysis Components. This circuit can be applied to wideband signals as well as individual subbands, and can be enhanced in various ways by specialized residual (or diffuse) signal processing.

[0016] The use of matrix encoders is highly suitable for distribution and playback on AV receivers, but can be problematic for mobile applications requiring low data rates and low power consumption.

[0017] Regardless of whether channel-based or object-based content is used, matrix encoders and decoders rely on fairly accurate inter-channel phase relationships of signals that are propagated from the matrix encoder to the decoder. In other words, the distribution format must largely preserve the waveform. This dependence on waveform preservation can create problems in limited bitrate environments where audio codecs use parametric methods rather than waveform encoding tools to obtain better audio quality. Examples of such parametric tools, which are generally known to be non-waveform preserving, are often referred to as spectral band replication, parametric stereo coding, spatial audio coding, etc., as used in MPEG 4 (14496-3:2009 ISO/IEC) audio codecs.

[0018] As briefly described in the previous section, upmixing consists of analyzing and adjusting (or HRIR deconvolution) signals. For mains-powered devices such as AV receivers this is usually not a problem, but for battery-powered devices such as mobile phones and tablets, the computational complexity and associated memory requirements associated with these processes are often undesirable due to their negative impact on battery life.

[0019] The above analysis typically also introduces additional audio delay. This audio delay is undesirable because (1) it requires a video delay to maintain lip synchronization with the soundtrack, which requires a significant amount of memory and processing power, and (2) such a delay can cause asynchrony/latency between head movements and audio rendering in case of tracking behind the movement of the head.

[0020] A matrix encoded downmix may also not sound optimal on stereo speakers or headphones due to the potential presence of highly out-of-phase signal components.

The essence of the invention

[0021] An object of the invention is to provide an improved form of parametric binaural output signal.

[0022] According to a first aspect of the present invention, there is provided a method for encoding a channel-based or object-based input audio signal for reproduction, the method including the steps of: (a) initially rendering the channel-based or object-based input audio signal, or object, into an initial output presentation (eg, an initial output presentation); (b) determining a dominant audio component estimate from the input audio signal based on the channel or object, and determining a sequence of dominant audio component weights for mapping the initial output representation to the dominant audio component; (c) determine an estimate of the direction and position of the dominant audio component; and (d) encoding the initial output representation, the weights of the dominant audio component, the direction or position of the dominant audio component as an encoded signal for reproduction. By providing a sequence of dominant audio component weights to map the initial output representation to the dominant audio component, it is possible to allow the dominant audio component weights and the initial output representation to be used to determine a score of the dominant component.

[0023] In some embodiments, the method further includes determining a residual mix estimate, which is the initial output representation minus rendering of the dominant audio component or an estimate thereof. The method may also include generating an anechoic binaural mix of the input audio signal based on the channel or object, and determining an estimate of the residual mix, wherein the estimate of the residual mix may be the anechoic binaural mix minus rendering of a dominant audio component or an estimate thereof. Additionally, the method may include determining a sequence of residual matrix coefficients for mapping the initial output representation to a residual mix estimate.

[0024] The initial output presentation may comprise a speaker or headphone presentation. The input audio signal, based on channel or object, can be divided into time and frequency bins, and the encoding step can be repeated for a sequence of time steps and a sequence of frequency bands. The initial output view may contain a mix of stereo speakers.

[0025] In accordance with a further aspect of the present invention, there is provided a method for decoding an encoded audio signal, the encoded audio signal including: a first (eg, initial) output presentation (eg, a first/initial output presentation); the direction of the dominant audio component and the weighting coefficients of the dominant audio component; the method comprising the steps of: (a) using dominant audio component weights and an initial output representation to determine an estimated dominant component; (b) rendering the estimated dominant component by binauralization at a spatial location relative to the target listener in accordance with the direction of the dominant audio component to generate a rendered binauralized estimated dominant component; (c) reconstructing the residual component estimate from the first (eg, initial) output representation; and (d) combining the rendered binauralized dominant component estimate and the residual component estimate to generate an output spatially oriented encoded audio signal.

[0026] The encoded audio signal may further include a sequence of residual matrix coefficients representing the residual audio signal, and step (c) may further comprise step (c1) of applying the residual matrix coefficients to the first (e.g., initial) output representation to reconstruct assessment of the residual component.

[0027] In some embodiments, the residual component estimate may be reconstructed by subtracting the rendered binauralized dominant component estimate from the first (eg, initial) output representation. Step (b) may include an initial rotation of the estimated dominant component in accordance with a head tracking input indicating the head orientation of the target listener.

[0028] In accordance with a further aspect of the present invention, there is provided a method for decoding and reproducing an audio stream for a listener using headphones, the method comprising the steps of: (a) receiving a data stream containing a first audio presentation and additional audio transform data; (b) receiving head orientation data representing the orientation of the listener; (c) creating one or more auxiliary signals based on the first audio presentation and the received transform data; (d) creating a second audio presentation consisting of a combination of the first audio presentation and the auxiliary signal(s), in which one or more auxiliary signals have been modified in response to the head orientation data; and (e) outputting the second audio presentation as an output audio stream.

[0029] Some embodiments may further include modification of the auxiliary signals, which consists of modeling the acoustic path from the position of the sound source to the listener's ears. The transformation data may consist of matrixing coefficients and at least one of a sound source position and a sound source direction. The conversion process can be applied as a function of time or frequency. The auxiliary signals may represent at least one dominant component. The position or direction of the sound source may be received as part of the transformation data and may be rotated in response to the head orientation data. In some embodiments, the maximum amount of rotation is limited to less than 360 degrees in azimuth or elevation. The secondary presentation can be obtained from the first presentation by matrixing into a transform domain or a set of filters. The transform data may further comprise additional matrixing coefficients, and step (d) may further comprise modifying the first audio representation in response to the additional matrixing coefficients before combining the first audio representation and the auxiliary audio signal(s).

Brief description of drawings

[0030] By way of example only, embodiments of the invention will now be described with reference to the accompanying drawings, in which:

[0031] FIG. 1 schematically illustrates a headphone decoder for matrix encoded content;

[0032] FIG. 2 schematically illustrates an encoder according to an embodiment;

[0033] FIG. 3 is a block diagram of a decoder;

[0034] FIG. 4 is a detailed visualization of the encoder; And

[0035] FIG. Figure 5 illustrates one form of decoder in more detail.

Carrying out the invention

[0036] Embodiments show a system and method for presenting channel- or object-based audio content that (1) is compatible with stereo playback, (2) allows binaural playback including head tracking, (3) has low decoder complexity, and (4) does not rely on, but is nevertheless compatible with, matrix encoding.

[0037] This is achieved by combining an encoder-side analysis of one or more dominant components (or a dominant object or a combination thereof), including weights for predicting those dominant components from the down-mix, in combination with additional parameters that minimize the error between binaural rendering based on the regulated or dominant components alone, and the desired binaural representation of the complete content.

[0038] In an embodiment, analysis of the dominant component (or multiple dominant components) is provided in the encoder rather than in the decoder/renderer. The audio stream is then augmented with metadata indicating the direction of the dominant component and information about how the dominant component(s) can be derived from the accompanying down-mix signal.

[0039] In FIG. 2 shows one form of encoder 20 of a preferred embodiment. The object- or channel-based content 21 is analyzed 23 to determine the dominant component(s). This analysis can take place as a function of time and frequency (assuming that audio content is broken down into time elements and frequency sub-elements). The result of this process is a dominant component signal 26 (or multiple dominant component signals) and associated position(s) or direction(s) information 25. Next, an estimate 24 is made and weights 27 are derived to allow reconstruction of the dominant component signal(s) from the transmitted downmix. This downmix generator 22 does not need to strictly follow the LtRt downmix rules, but can be a standard ITU downmix (LoRo) using non-negative, real-valued downmix coefficients. Finally, the downmix output 29, weights 27, and position data 25 are packaged by audio encoder 28 and prepared for distribution.

[0040] In FIG. 3 shows a corresponding decoder 30 of a preferred embodiment. The audio decoder reconstructs the downmix signal. The signal is input 31 and decompressed by audio decoder 32 into a downmix signal, the weights and direction of the dominant components. Next, the weights of the estimated dominant components are used to reconstruct the 34 adjustable components, which are rendered 36 using positional or directional data. The positional data may optionally be modified 33 depending on head rotation or transformation information 38 . Additionally, the reconstructed dominant component(s) can be subtracted 35 from the down mix. Alternatively, subtraction of the dominant component(s) within the down-mix path takes place, but alternatively, subtraction may also occur at the encoder, as described below.

[0041] To improve the removal or cancellation of the reconstructed dominant component in the subtractor 35, the output signal of the dominant component may first be rendered using the transmitted position data or direction data before subtraction. This optional rendering step 39 is shown in FIG. 3.

[0042] Returning now to first describe the encoder in more detail, FIG. 4 illustrates one form of encoder 40 for processing object-based audio content (eg, Dolby Atmos). Audio objects are initially stored as Atmos objects 41 and are initially divided into time and frequency elements using a set of 42 hybrid complex-valued quadrature mirror filters (HCQMF). Object input signals can be designated as , when we omit the corresponding time and frequency indices; the corresponding position within the current frame is specified by a unit vector , and the index i refers to the object number and the index n refers to the time (for example, the sub-range sampling index). Input signals objects are an example of an audio input signal based on a channel or object.

[0043] Anechoic, sub-band, binaural mix Y ( ) create 43 using complex-valued scalars (e.g. single-pole HRTF 48) which provide a subband presentation for the HRIRs corresponding to the position :

[0044] Alternatively, binaural mix Y ( ) can be created through convolution using head-related impulse responses (HRIR). Additionally, stereo down mix (as an example implementing the initial output representation) create 44 using the coefficients amplitude panning gain:

[0045] Direction vector of the dominant component (by way of example, implementing the direction or position of a dominant audio component) can be estimated by calculating the dominant component 45 by initially computing a weighted sum of unit direction vectors for each object:

Where - signal energy :

and (.)* is the complex convolution operator.

[0046] The dominant/control signal d [ n ] (by way of example, implementing the dominant audio component) is further specified as follows:

[0047] where - a function that performs a gain that decreases with increasing distance between unit vectors . For example, to create a virtual microphone using a directivity model based on higher order spherical harmonics, one of the implementations must conform to the following:

Where is a unit direction vector in a two- or three-dimensional coordinate system, (.) is the scalar product operator of two vectors, and a, b, c are approximate parameters (for example a=b=0.5; c=1).

[0048] The weights or prediction coefficients w _{l , d} , w _{r , d} are calculated 46 and used to calculate 47 the estimated controlled signal :

where the weights w _{l , d} , w _{r , d} minimize the root mean square error between , given signals down mix. The weights w _{l , d} , w _{r , d} are an example of dominant audio component weights for mapping the initial output representation (e.g. ) into the dominant audio component (e.g. ). A known way to obtain these weights is to use a minimum mean square error (MMSE) predictor:

where R _ab is the covariance matrix between signals for signals a and signals b , and ∈ is the regularization parameter.

[0049] We can then subtract 49 the rendered estimate of the dominant component signal from an anechoic binaural mix to create a residual binaural mix using HRTF (HRIR) 50 related to direction/position dominant component signal :

[0050] Finally, 51 different sets of prediction coefficients or weights w _{i , j} are evaluated that allow reconstruction of the residual binaural mix from stereo mix ,using the estimated minimum root mean square errors:

_where Rab is the covariance matrix between the signals for presentation a and presentation b , and ∈ is the regularization parameter. The prediction coefficients or weights w _{i , j} are an example of residual matrix coefficients for mapping the initial output representation (e.g. ) into the estimated residual binaural mix . The above expression can be subjected to additional level constraints to overcome any prediction loss. The encoder outputs the following information:

[0051] Stereomix (as an example of the implementation of the initial output representation);

[0052] Coefficients for estimating the dominant componentw _{l , d} ,w _{r , d} (as an example of implementing the weighting coefficients of the dominant audio component) are;

[0053] position or direction of the dominant component ;

[0054] and, additionally, residual weights w _{i , j} (as an example of the implementation of residual matrix coefficients).

[0055] Although the above description relates to rendering based on one single dominant component, in some embodiments the encoder may be configured to detect multiple dominant components, determine weights and directions for each of the multiple dominant components, render and subtract each of the multiple dominant components from the anechoic binaural mix Y, and then determining the residual weights after each of the multiple dominant components has been subtracted from the anechoic binaural mix Y.

Decoder/renderer

[0056] In FIG. 5 shows one form of decoder/renderer 60 in more detail. Decoder/renderer 60 employs a process aimed at reconstructing a binaural mix. to output to listener 71 from unpacked input informationz _l , z _r ; w _{l , d} ,w _{r , d} ; ,w _{i , j} . Here's a stereo mixz _l , z _r is an example of the first audio presentation and the coefficients or weights of the predictionw _{i , j} and/or direction/position dominant component signal are examples of additional audio conversion data.

[0057] Initially, the stereo down mix is split into time/frequency elements using an appropriate filter bank or transform 61, such as the HCQMF analysis group 61. Other transforms, such as the discrete Fourier transform, (modified) cosine or sine transform, filter bank in time domain or wavelet transform may also be equally applicable. In the following, the estimated dominant component signal 63 is calculated using the weights w _{l , d} , w _{r , d} prediction coefficients:

= w _{l , d} z _l +w _{r , d} z _r

Estimated dominant component signal is an example of an auxiliary signal. Therefore, it can be said that this step corresponds to the creation of one or more auxiliary signals based on said first audio presentation and the received transform data.

[0058] This dominant component signal is further rendered 65 and modified 68 by HRTF 69 based on the transmitted position/direction data , possibly modified (rotated) based on information received from the head tracking device 62 . Finally, the overall muted binaural output consists of the rendered dominant component signal summed 66 with the reconstructed residuals , based on the weights w _ij of the prediction coefficients:

The full muted binaural output is an example of a second audio presentation. Therefore, this step may be said to correspond to the creation of a second audio presentation consisting of a combination of said first audio presentation and said auxiliary signal(s), in which one or more of said auxiliary signals have been modified in response to said head orientation data.

[0059] Additionally, it should be noted that if information about more than one dominant signal is received, each dominant signal may be rendered and added to the reconstructed residual signal.

[0060] As long as no head rotation or movement is applied, the output signals should be very close (in terms of mean square error) to the reference binaural signals , Bye

Basic properties

[0061] As can be seen from the above equations, an efficient operation to create a muted binaural representation from a stereo representation consists of a 2×2 matrix 70 in which the matrix coefficients depend on the transmitted information w _{l , d} , w _{r , d} ; , w _{i , j} and rotation and/or movement of the head tracking device. This indicates that the process complexity is relatively low since dominant component analysis is applied in the encoder instead of the decoder.

[0062] If no dominant component is estimated (eg, w _{l , d} , w _{r , d} =0), the described solution is equivalent to the parametric binaural method.

[0063] In cases where there is a desire to exclude certain objects from head rotation/translation tracking, these objects can be excluded from (1) dominant component direction analysis, and (2) dominant component signal prediction. As a result, these objects will be converted from stereo to binaural using coefficientsw _{i , j} and are therefore not affected by any rotation or movement of the head.

[0064] With this line of thinking, objects could be set to "pass through" mode, which means they would be amplitude panned rather than HRIR deconvoluted in the binaural presentation. This can be obtained by simply using amplitude panning gains for the H _i coefficients instead of single-pole HRTFs or any other appropriate binaural process.

Extensions

[0065] Embodiments are not limited to the use of down mixes, as samples of other channels may also be used.

[0066] Decoder 60, described with reference to FIG. 5 has an output signal consisting of the rendered direction of the dominant component plus an input signal matrixed by the matrix coefficients w _{i , j} . The latter coefficients can be obtained in various ways, for example:

[0067] 1. The coefficients w _{i , j} can be determined in the encoder by parametric signal reconstruction . In other words, in this implementation, the coefficients w _{i , j} are aimed at accurately reconstructing binaural signals , which could be obtained by rendering the original input objects/channels in a binaural manner; in other words, the coefficients w _{i , j} are controlled by the content.

[0068] 2. The coefficients w _{i , j} may be passed from the encoder to the decoder to represent the HRTF for certain spatial positions, for example, at angles of +/-45 degrees in azimuth. In other words, the residual signal is processed to simulate playback through two virtual speakers at specific locations. Because these coefficients representing the HRTF are passed from the encoder to the decoder, the locations of the virtual speakers may vary in time and frequency. If this approach is applied using static virtual speakers to represent the residual signal, the coefficients w _{i , j} do not require transmission from the encoder to the decoder, and can instead be wired at the decoder. A variant of this approach may consist of a limited set of static locations that are available at the decoder, with their corresponding coefficients w _{i , j} , and their selection, which uses the static location to process the residual signal, is communicated from the encoder to the decoder.

[0069] Signals can undergo what is called upmixing, reconstructing more than 2 signals through statistical analysis of those signals in the decoder, followed by binaural rendering of the resulting upmix signals.

[0070] The described methods may also be applicable in a system in which the transmitted signal Z is a binaural signal. In this particular case, the decoder 60 shown in FIG. 5 remains as is, while the block labeled "Generate stereo (LoRo) mix" 44 and shown in FIG. 4 should be replaced with "Generate anechoic binaural mix" (generate muted binaural mix) 43 (Fig. 4), which is the same as the block that creates the Y signal pair. Additionally, other forms of mixes can be generated according to requirements.

[0071] This approach can be extended with methods for reconstructing one or more FDN input signals from a transmitted stereo mix that consists of a specific subset of objects or channels.

[0072] The approach can be extended with multiple dominant components predicted from the transmitted stereo mix and rendered at the decoder side. There is no fundamental limitation to predicting only one dominant component for each time/frequency bin. In particular, the number of dominant components may differ in each time/frequency partition element.

Interpretation

[0073] Throughout this specification, reference to “one embodiment,” “certain embodiments,” or “an embodiment” means that the particular feature, structure, or characteristic described in combination with the embodiment is introduced into at least one embodiment implementation of the present invention. Thus, the expressions “in one embodiment,” “in some embodiments,” or “in an embodiment” are not required to appear everywhere throughout the specification as long as they all refer to the same embodiment. Moreover, specific features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one skilled in the art from this disclosure, in one or more embodiments.

[0074] The use of the ordinal numbers "first", "second", "third", etc. to describe a common object as used herein, unless otherwise specified, simply indicates that reference is being made to different instances of similar objects, and it is not intended to imply that the objects so described must appear in the given sequence in time or space, in order, or in any other way.

[0075] In the following claims and description herein, any of the terms “comprising,” “contained,” or “which contains” is an open-ended term that means to include at least the elements/features corresponding to the term, but does not exclude others. Thus, the term “comprising,” when used in the claims, should not be construed as limiting the means, elements, or steps listed hereinafter. For example, the scope of the expression "device comprising A and B" should not be limited to devices consisting only of elements A and B. Any of the terms "including" or "which includes" or "which include" as these terms used here are also open-ended terms which mean to include at least the elements/features corresponding to the term, but do not exclude others. Thus, "including" is a synonym and means "containing".

[0076] The term "exemplary" as used herein is used in the sense of providing examples and not as an indication of quality. That is, “exemplary embodiment” is an embodiment presented as an example, and is not necessarily an embodiment exemplary in quality.

[0077] It should be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, drawing, or description thereof for the purpose of streamlining the disclosure and assisting in understanding one or more different inventive approaches. This manner of disclosure, however, should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly provided in each claim. Rather, as reflected in the following claims, aspects of the invention are embodied in less than all of the features of the single preceding disclosed embodiment. Thus, the claims following the "Details of the Invention" section are hereby expressly included in this "Details of the Invention" with each claim standing on its own as a separate embodiment of that invention.

[0078] Additionally, although some embodiments described herein contain some but not other features included in other embodiments, combinations of features in different embodiments mean that they are within the scope of the invention and form different embodiments, as this should be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0079] Additionally, some of the embodiments are described herein as a method or combination of elements of a method that may be implemented by a computer system processor or other means of performing a function. Thus, a processor with the necessary instructions for executing such method or method element constitutes means for executing the method or method element. Additionally, the element of the apparatus embodiment described herein is an example of a means of performing a function performed by the element for the purpose of carrying out the invention.

[0080] Numerous specific details are set forth in the description provided herein. However, it should be understood that embodiments of the invention may be practiced without these specific details. In other cases, known methods, structures and technologies have not been shown in detail so as not to interfere with the understanding of the present description.

[0081] Likewise, it should be noted that the term “related” when used in the claims should not be interpreted as limiting only direct connections. The terms "linked" and "connected", as well as their derivatives, may be used. It should be understood that these terms are not intended to be synonymous with each other. Thus, the context of the expression "Device A coupled to device B" should not be limited to devices or systems in which the output of device A is directly connected to the input of device B. This means that there is a path between the output of device A and the input of device B that may be a path of passage containing other devices or means. "Coupled" can mean that two or more elements are in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but still act together or interact with each other.

[0082] Thus, while embodiments of the invention have been described herein, those skilled in the art will recognize that other and additional modifications may be made thereto without departing from the spirit of the invention, and all such changes and modifications are intended to be within within the scope of the invention. For example, any formulas given above are merely representative of procedures that may be used. Functionality can be added or removed from block diagrams, and operations can be interleaved between function blocks. In the methods described within the scope of the present invention, steps may be added or removed.

[0083] Various aspects of the present invention can be understood from the following enumerated example embodiment (EEE):

EEE 1. An encoding method for reproducing an input audio signal based on a channel or an object, said method comprising the steps of:

(a) initially rendering the input audio signal, based on the channel or object, into an initial output representation;

(b) determining a dominant audio component estimate from the input audio signal based on the channel or object, and determining a sequence of dominant audio component weights for mapping the initial output representation to the dominant audio component;

(c) determine an estimate of the direction or position of the dominant audio component; And

(d) encoding the initial output representation, the weights of the dominant audio component, the direction or position of the dominant audio component as an encoded signal for reproduction.

EEE 2. The method of claim EEE 1, further comprising determining an estimate of the residual mix, which is the initial output representation reduced by the rendering of the dominant audio component or its estimate.

EEE 3. The method of claim EEE 1, further comprising generating an anechoic binaural mix of the input audio signal based on the channel or object, and determining a residual mix estimate, wherein the residual mix estimate is the anechoic binaural mix minus the rendering of the dominant audio component or the estimate thereof.

EEE 4. The method of claim EEE 2 or 3, further comprising determining a sequence of residual matrix coefficients for mapping the initial output representation to a residual mix estimate.

EEE 5. A method according to any one of the preceding EEEs, wherein said initial output representation comprises headphones or a loudspeaker.

EEE 6. A method according to any one of the preceding EEEs, wherein said input audio signal based on a channel or object is decomposed into time and frequency elements and said encoding step is repeated over a sequence of time steps and sets of frequency bands.

EEE 7. The method of any one of the preceding EEEs, wherein said initial output representation comprises a mix of stereo speakers.

EEE 8. A method for decoding an encoded audio signal, wherein the encoded audio signal comprises:

- first output performance;

- weighting coefficients of the dominant audio component and the direction of the dominant audio components;

a method comprising the steps of:

(a) using the dominant audio component weights and the initial output representation to determine the estimated dominant component;

(b) rendering the binauralized estimated dominant component at a spatial location relative to the target listener in accordance with the direction of the dominant audio component to generate a rendered binauralized estimated dominant component;

(c) reconstructing the residual component estimate from the first output representation; And

(d) combining the rendered binauralized dominant component estimate and the residual component estimate to generate an output spatial encoded audio signal.

EEE 9. The method of claim EEE 8, wherein said encoded audio signal further comprises a sequence of residual matrix coefficients representing the residual audio signal, and said step (c) further is a step of:

(c1) apply said residual matrix coefficients to the first output representation to reconstruct an estimate of the residual component.

EEE 10. An EEE 8 method in which the residual component estimate is reconstructed by subtracting the rendered binauralized dominant component estimate from the first output representation.

EEE 11. The method of EEE 8, wherein said step (b) includes an initial rotation of the estimated dominant component in accordance with a head tracking input indicating the head orientation of the target listener.

EEE 12. A method of decoding and reproducing an audio stream for a listener using headphones, the method comprising the steps of:

(a) receiving a data stream containing the first audio presentation and additional transform audio data;

(b) receiving head orientation data representing the orientation of the listener;

(c) creating one or more auxiliary signals based on said first audio presentation and the received transform data;

(d) creating a second audio presentation consisting of a combination of the first audio presentation and said auxiliary signal(s), wherein one or more of said auxiliary signals have been modified in response to said head orientation data; And

(e) outputting the second audio presentation as an output audio stream.

EEE 13. The method of claim EEE 12, wherein the modification of the auxiliary signals consists of modeling the acoustic path from the position of the sound source to the listener's ears.

EEE 14. The method of claim EEE 12 or 13, wherein said transformation data consists of matrixing coefficients and at least one of a sound source position or a sound source direction.

EEE 15. Method according to any one of paragraphs. EEE 12-14, in which the conversion process is applied as a function of time or frequency.

EEE 16. The method according to any one of EEE 12-15, wherein the auxiliary signals are at least one dominant component.

EEE 17. Method according to any one of paragraphs. EEE 12-16, in which the position or direction of the sound source, received as part of the transformation data, is rotated in response to the head orientation data.

EEE 18. The method according to clause EEE 17, in which the maximum amount of rotation is limited to less than 360 degrees in azimuth or elevation.

EEE 19. Method according to any one of paragraphs. EEE 12-18, in which the secondary presentation is obtained from the first presentation by matrixing in a transform domain or filter bank.

EEE 20. Method according to any one of paragraphs. EEE 12-19, wherein the transform data further comprises additional matrixing coefficients and step (d) further comprises modifying the first audio representation in response to the additional matrixing coefficients prior to combining the first audio representation and the auxiliary audio signal(s).

EEE 21. A device containing one or more other devices, configured to implement any of the methods according to claims. EEE 1-20.

EEE 22. A computer-readable medium containing a program consisting of instructions that, when executed by one or more processors, cause one or more devices to perform the method of any one of claims. EEE 1-20.

Claims (22)

1. A system configured to encode an input audio signal based on a channel or object for playback, the system comprising: one or more processors; And A computer-readable medium on which instructions are stored that, when executed by one or more processors, direct a system to perform operations comprising: rendering the input audio signal, based on the channel or object, into an initial output representation; determining an estimate of the dominant audio component from an input audio signal based on a channel or object, which determination includes: determining a sequence of weight components of the dominant audio component to map the initial output representation to the dominant audio component; and determining a dominant audio component score based on the dominant audio component weights and the initial output representation; determining an estimate of the direction or position of the dominant audio component; And encoding the initial output representation, dominant audio component weights, and at least one of a direction and position of the dominant audio component as an encoded signal for reproduction. 2. The system of claim 1, wherein the operations further comprise determining a residual mix estimate, which is the initial output representation minus the rendering of either the dominant audio component or an estimate thereof. 3. The system of claim 1, wherein the operations further comprise generating an anechoic binaural mix of the input audio signal based on the channel or object, and determining a residual mix estimate, wherein the residual mix estimate is the anechoic binaural mix minus either rendering of the dominant audio component or an estimate thereof. . 4. The system of claim 2, wherein the operations further comprise determining a sequence of residual matrix coefficients for mapping the initial output representation to a residual mix estimate. 5. The system of claim 1, wherein the initial output presentation comprises a headphone presentation or a speaker presentation. 6. The system of claim 1, wherein the channel- or object-based input audio signal is decomposed into time and frequency bins, and said encoding operation is repeated over a sequence of time steps and a sequence of frequency bands. 7. The system of claim 1, wherein the initial output representation comprises a mix of stereo speakers. 8. A non-transitory computer-readable medium on which instructions are stored that, when executed by one or more processors, instruct one or more devices to perform operations containing: rendering the input audio signal, based on the channel or object, into an initial output representation; determining an estimate of the dominant audio component from an input audio signal based on a channel or object, which determination includes: determining a sequence of weight components of the dominant audio component to map the initial output representation to the dominant audio component; and determining a dominant audio component score based on the dominant audio component weights and the initial output representation; determining an estimate of the direction or position of the dominant audio component; And encoding the initial output representation, dominant audio component weights, and at least one of a direction and position of the dominant audio component as an encoded signal for reproduction.

Images