RU2809609C2 - Representation of spatial sound as sound signal and metadata associated with it - Google Patents

Abstract

FIELD: acoustics.

SUBSTANCE: method for representing spatial audio, wherein the spatial audio is a combination of directional audio and diffuse audio, is provided, the method includes: creating a single-channel or multi-channel downmix audio signal by downmixing input audio signals from a plurality of microphones (m1, m2, m3) in an audio capture unit capturing spatial audio, wherein downmixing to create a single-channel or multi-channel audio signal x downmixing is described as follows: x = D·m, where D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and m is a matrix representing the input audio signals from the plurality of microphones; determining first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; and combining the generated downmix audio signal and the first metadata parameters into a spatial audio representation.

EFFECT: improvement of presentation of spatial sound.

35 cl, 9 dwg

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to US Provisional Patent Application No. 62/760262, filed November 13, 2018; US Provisional Patent Application No. 62/795248, filed January 22, 2019; US Provisional Patent Application No. 62/828038, filed April 2, 2019; and US Provisional Patent Application No. 62/926,719, filed October 28, 2019, the contents of which are incorporated herein by reference.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0002] The present disclosure generally relates to encoding an audio scene containing audio objects. In particular, it relates to methods, systems, computer program products and data formats for presenting spatial audio and associated encoder, decoder and renderer for encoding, decoding and presenting spatial audio.

BACKGROUND OF THE ART

[0003] The introduction of high-speed wireless access to 4G/5G telecommunications networks, coupled with the availability of increasingly powerful hardware platforms, has provided the basis for faster and easier deployment of advanced communications and multimedia services than ever before.

[0004] The 3rd Generation Partnership Project (3GPP) Enhanced Voice Services (EVS) codec has provided very significant improvements in user experience by introducing ultra-wideband (SWB) and full-bandwidth (FB) speech and audio coding along with improved packet loss tolerance. However, increased audio bandwidth is only one of the parameters needed for a truly immersive experience. Support beyond the mono and multi-mono that EVS currently offers is ideally needed to immerse the user in a compelling virtual world in a resource-efficient manner.

[0005] Additionally, currently defined audio codecs in 3GPP provide suitable quality and compression for stereo content, but do not have the conversational features (eg, low enough latency) needed for spoken language and teleconferencing. These encoders also lack the multi-channel functionality required for immersive services such as streaming, virtual reality (VR), and immersive teleconferencing.

[0006] An extension to the EVS codec has been proposed for immersive voice and audio services (IVAS) to fill this technology gap and meet the growing demand for feature-rich multimedia services. Additionally, 4G/5G teleconferencing applications will benefit from using the IVAS codec as an enhanced conversational encoder that supports multi-stream encoding (e.g., channel-, object-, and scene-based audio). Use cases for this next-generation codec include, but are not limited to, voice-over-conversation, multi-stream teleconferencing, conversational virtual reality, and user-generated real-time and non-real-time content streaming.

[0007] Although the goal is to develop a single codec with attractive features and performance (e.g., excellent audio quality, low latency, support for spatial audio encoding, suitable range of bit rates, high-quality error tolerance, practical implementation complexity), there is currently no final agreement on the audio input format of the IVAS codec. Metadata-Aware Spatial Audio (MASA) has been proposed as one possible audio input format. However, conventional MASA parameters make certain idealistic assumptions, such as that the audio is captured at a single point. However, in a real-life scenario where a mobile phone or tablet is used as an audio capture device, this assumption of capturing audio at a single point may not hold. Rather, depending on the form factor of a particular device, the various microphones of the device may be located some distance apart, and the different captured signals of the microphones may not be completely synchronized in time. This is especially true when one also considers how the sound source may move through space.

[0008] Another assumption underlying the MASA format is that all microphone channels are provided at the same level and that there are no differences in frequency and phase response between them. Again, in a real-life scenario, microphone channels may have different frequency and phase characteristics depending on direction, which may also depend on time. It may be assumed, for example, that the audio capture device is temporarily held in such a way that one of the microphones is covered, or that there is some object in the vicinity of the telephone that causes reflections or diffractions of the incoming sound waves. As such, there are many additional factors that need to be considered when determining which audio format is suitable for use with a codec such as the IVAS codec.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Exemplary embodiments will now be described with reference to the accompanying drawings, in which:

[0010] FIG. 1 is a flowchart of a method for presenting spatial audio according to exemplary embodiments;

[0011] FIG. 2 is a schematic illustration of an audio capture apparatus and directional and diffuse audio sources, respectively, according to exemplary embodiments;

[0012] In FIG. 3A shows a table (Table 1A) describing how the value of the channel bit parameter indicates how many channels are used for the MASA format, according to exemplary embodiments.

[0013] In FIG. 3B shows a table (Table 1B) of a metadata structure that can be used to represent a planar FOA and capture FOA downmixed into two MASA channels according to exemplary embodiments;

[0014] In FIG. 4 shows a table (Table 2) of delay compensation values for each microphone and for each TF cell according to exemplary embodiments;

[0015] In FIG. 5 shows a table (Table 3) of a metadata structure that can be used to indicate which set of compensation values is applied to which TF cell, according to exemplary embodiments;

[0016] In FIG. 6 shows a table (Table 4) of a metadata structure that can be used to represent the gain setting for each microphone according to exemplary embodiments;

[0017] In FIG. 7 shows a system that includes an audio capture device, an encoder, a decoder, and a renderer, according to exemplary embodiments.

[0018] In FIG. 8 shows an audio capture device according to exemplary embodiments.

[0019] In FIG. 9 shows a decoder and renderer according to exemplary embodiments.

[0020] All figures are schematic and generally show only those parts necessary to illustrate the disclosure, while other parts may be omitted or merely implied. Unless otherwise indicated, like numerals refer to like parts in different figures.

DETAILED DESCRIPTION

[0021] In view of the above, it is an object to provide methods, systems, and computer program products and data formats for improved spatial audio presentation. An encoder, decoder, and renderer for spatial audio are also provided.

I. Overview - spatial representation of sound

[0022] According to a first aspect, a method, system, computer program product, and data format for representing spatial audio are provided.

[0023] According to exemplary embodiments, a method for representing spatial sound, wherein the spatial sound is a combination of directional sound and diffuse sound, is provided, including:

- creating a single-channel or multi-channel downmix audio signal by downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio;

- defining first metadata parameters associated with the downmix audio signal, the first metadata parameters indicating one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And

- combining the generated downmix audio signal and the first metadata parameters into a spatial audio representation.

[0024] Using the above scheme, improved spatial audio representation can be achieved by taking into account the different properties and/or spatial positions of multiple microphones. Moreover, the use of metadata in subsequent stages of encoding, decoding or rendering processing can facilitate the faithful representation and reconstruction of captured audio by presenting the audio in a data rate efficient encoded form.

[0025] According to exemplary embodiments, combining the generated downmix audio signal and first metadata parameters into a spatial audio representation may further include including second metadata parameters into the spatial audio representation, wherein the second metadata parameters indicate a downmix configuration for the input audio signals .

[0026] The advantage of this is that it allows the input audio signals to be reconstructed (eg, through an upmix operation) in the decoder. Additionally, by providing the second metadata, additional downmixing may be performed in a separate block before encoding the spatial audio representation into the bitstream.

[0027] According to exemplary embodiments, first metadata parameters may be defined for one or more frequency bands of microphone audio input signals.

[0028] The advantage of this is that it allows delay, gain and/or phase adjustment parameters to be individually adjusted, for example to accommodate different frequency characteristics for different frequency ranges of microphone signals.

[0029] According to exemplary embodiments, downmixing to create a single-channel or multi-channel downmix audio signal x can be described as follows:

Where:

D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and

M is a matrix representing the input audio signals from multiple microphones.

[0030] According to exemplary embodiments, downmix factors can be selected to select the microphone audio input signal currently having the best signal-to-noise ratio relative to the directional audio, and to discard audio input signals from any other microphones.

[0031] The advantage of this is that it allows achieving good quality spatial audio representation with reduced computational complexity in the audio capture unit. In this embodiment, only one input audio signal is selected to represent spatial audio at a particular audio frame and/or time-frequency bin. Consequently, the computational complexity of the downmixing operation is reduced.

[0032] According to exemplary embodiments, the selection may be determined based on time-frequency (TF) cells.

[0033] The advantage of this is that it allows the downmix operation to be improved, for example by taking into account different frequency characteristics for different frequency bands of microphone signals.

[0034] According to exemplary embodiments, a selection may be made for a specific audio frame.

[0035] The advantage of this is that it allows for tuning with respect to time-varying microphone acquisition signals and, in turn, with respect to improving audio quality.

[0036] According to exemplary embodiments, downmix factors can be selected to maximize the signal-to-noise ratio with respect to directional audio when combining audio input signals from different microphones.

[0037] The advantage of this is that it improves the quality of the downmix by attenuating unwanted signal components that do not originate from directional sources.

[0038] According to exemplary embodiments, maximization may be performed for a specific frequency band.

[0039] According to exemplary embodiments, maximization may be performed for a specific audio frame.

[0040] According to exemplary embodiments, determining the first metadata parameters may include analyzing one or more of the following: delay, gain, and phase characteristics of input audio signals from a plurality of microphones.

[0041] According to exemplary embodiments, first metadata parameters may be determined based on a time-frequency (TF) cell.

[0042] According to exemplary embodiments, at least a portion of the downmixing may occur in an audio capture unit.

[0043] According to exemplary embodiments, at least a portion of the downmixing may occur in the encoder.

[0044] According to exemplary embodiments, when more than one directional sound source is detected, first metadata may be determined for each source.

[0045] According to exemplary embodiments, the spatial audio representation may include at least one of the following parameters: a directional index, a ratio of direct to total energy; propagation coherence; arrival time, gain and phase for each microphone; ratio of dissipated energy to total energy; volumetric coherence; ratio of remainder to total energy; and distance.

[0046] In exemplary embodiments, the metadata parameter of the second or first metadata parameters may indicate whether the generated downmix audio signal is generated from: left and right stereo signals, first order planar ambiophonic (FOA) signals, or FOA component signals.

[0047] According to exemplary embodiments, the spatial audio representation may comprise metadata parameters organized in a definition field and a selector field, wherein the definition field is defined as at least one set of delay compensation parameters associated with a plurality of microphones, and the selector field is defined as a selection. set of delay compensation parameters.

[0048] According to exemplary embodiments, the selector field may indicate which set of delay compensation parameters are applied to any given time-frequency cell.

[0049] According to exemplary embodiments, the relative time delay value may be in the approximate range of [-2.0 ms, 2.0 ms].

[0050] According to exemplary embodiments, the metadata parameters in the spatial audio view may further include a field specifying the gain setting to be applied and a field specifying the phase setting.

[0051] According to exemplary embodiments, the gain setting may be in the approximate range of [+10 dB, -30 dB].

[0052] According to exemplary embodiments, at least portions of the first and/or second metadata elements are determined in the audio capture device using stored lookup tables.

[0053] According to exemplary embodiments, at least portions of the first and/or second metadata elements are determined in a remote device connected to the audio capture device.

II. Review - system

[0054] According to a second aspect, a system for presenting spatial audio is provided.

[0055] According to exemplary embodiments, a system for presenting spatial audio is provided, comprising:

a receiving component configured to receive input audio signals from a plurality of microphones in the audio capturing unit capturing spatial audio;

a downmix component configured to create a single-channel or multi-channel downmix audio signal by downmixing the received audio signals;

a metadata determining component configured to determine first metadata parameters associated with the downmix audio signal, the first metadata parameters indicating one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And

a combining component configured to combine the generated downmix audio signal and the first metadata parameters into a spatial audio representation.

III. Overview - Data Format

[0056] According to a third aspect, a data format for representing spatial audio is provided. The data format can be used advantageously in combination with physical components related to spatial audio, such as audio capture devices, encoders, decoders, renderers, etc., as well as with various types of computer software products and other equipment that are used to transmitting spatial audio between devices and/or locations.

[0057] According to exemplary embodiments, the data format comprises:

a downmix audio signal obtained by downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio; And

first metadata parameters indicating one or more of: a downmix configuration for the input audio signals, a relative time delay value, a gain value, and a phase value associated with each input audio signal.

[0058] According to one example, the data format is stored in non-volatile memory.

IV. Review - Encoder

[0059] According to a fourth aspect, an encoder for encoding a spatial audio representation is provided.

[0060] According to exemplary embodiments, an encoder is provided configured to:

receiving a representation of spatial sound, wherein the representation contains:

a single-channel or multi-channel downmix audio signal created by downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio, and

first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And

encoding the single-channel or multi-channel downmix audio signal into a bitstream using the first metadata, or

encoding the single-channel or multi-channel downmix audio signal and the first metadata into a bitstream.

V. Review - decoder

[0061] According to a fifth aspect, a decoder for decoding a spatial audio representation is provided.

[0062] According to exemplary embodiments, a decoder is provided configured to:

receiving a bitstream indicating an encoded spatial audio representation, the representation comprising:

a single-channel or multi-channel downmix audio signal created by downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio, and

first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And

decoding the bitstream into a spatial audio approximation by using the first metadata parameters.

VI. Review - Renderer

[0063] According to a sixth aspect, a renderer is provided for rendering a spatial audio representation.

[0064] According to exemplary embodiments, a renderer configured to:

receiving a representation of spatial sound, wherein the representation contains:

a single-channel or multi-channel downmix audio signal created by downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio, and

first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And

rendering spatial audio using first metadata.

VII. Review - general description

[0065] Aspects two through six may have generally the same characteristics and advantages as the first aspect.

[0066] Other objects, characteristics and advantages of the present invention will become apparent from the following detailed description, the appended dependent claims, and the drawings.

[0067] The steps of any method disclosed herein do not necessarily have to be performed in the exact order disclosed unless explicitly stated otherwise.

VIII. Example Embodiments

[0068] As described above, capturing and presenting spatial audio has a distinct set of challenges with respect to ensuring that the captured audio can be accurately reproduced at the receiving end. Various embodiments of the present invention described in the present disclosure solve various aspects of these problems by including various metadata parameters along with the downmix audio signal when transmitting the downmix audio signal.

[0069] The invention will be described by way of example and with reference to the MASA audio format. However, it is important to understand that the general principles of the invention apply to a wide range of formats that can be used to represent audio, and the description given in this disclosure is not limited to MASA.

[0070] In addition, it should be understood that the metadata parameters that are described below are not a complete list of metadata parameters, but there may be additional metadata parameters (or a smaller subset of metadata parameters) that can be used to convey data about the downmix audio signal to various devices used in audio encoding, decoding and rendering.

[0071] In addition, although the examples in the present disclosure will be described in the context of an IVAS encoder, it should be noted that this is only one type of encoder to which the general principles of the invention may be applied, and that there may be many other types of encoders, decoders and means rendering, which can be used in conjunction with various embodiments described in this disclosure.

[0072] Finally, it should be noted that although the terms “upmixing” and “downmixing” are used in the present disclosure, they may not necessarily mean increasing and decreasing, respectively, the number of channels. While this may often be the case, it should be understood that either term can refer to either a reduction or an increase in the number of channels. Thus, both terms fall under the more general concept of "mixing". Likewise, the term "downmix audio signal" will be used throughout the description, but it should be understood that other terms such as "MASA channel", "transport channel" or "downmix channel" may sometimes be used, all of which have essentially the same meaning as "downmix audio signal".

[0073] Referring now to FIG. 1; a method 100 for presenting spatial audio in accordance with one embodiment is described. As can be seen in FIG. 1, the method begins with capturing spatial audio using an audio capture device, step 102. Referring to FIG. 2 shows a schematic view of an audio environment 200 in which an audio capture device 202, such as a cell phone or tablet computer, for example, captures audio from a diffuse ambient source 204 and a directional source 206, such as a speaker. In the illustrated embodiment, audio capture device 202 has three microphones m1, m2 and m3, respectively.

[0074] Directional sound comes from the direction of arrival (DOA), represented by azimuth and elevation angles. The diffuse ambient sound is assumed to be omnidirectional, that is, spatially invariant or spatially homogeneous. The following discussion also addresses the possibility of a second directional sound source, which is not shown in FIG. 2.

[0075] The signals from the microphones are then downmixed to create a single-channel or multi-channel downmixed audio signal, step 104. There are many reasons for distributing only a monaural downmixed audio signal. For example, there may be data rate limitations or an intention to make a high-quality, down-mixed monaural audio signal available after making certain proprietary enhancements such as beam shaping and noise correction or suppression. In other embodiments, the downmix produces a multi-channel downmix audio signal. Typically, the number of channels in a downmix audio signal is less than the number of input audio signals, however in some cases the number of channels in a downmix audio signal may be equal to the number of input audio signals, and the purpose of the downmix is rather to achieve an increased ratio signal to noise or reduce the amount of data in the resulting downmix audio signal compared to the input audio signals. This is discussed in more detail below.

[0076] Propagating the appropriate parameters used during the downmix to the IVAS codec as part of the MASA metadata may enable the stereo and/or spatial audio signal of the downmix to be reconstructed with the best possible accuracy.

[0077] In this scenario, a single MASA channel is obtained using the following downmix operation:

, Where

And

[0078] The m and x signals may, during various processing steps, not necessarily be represented as full-bandwidth time signals, but possibly also as component signals of various subbands in the time or frequency domain (TF cells). In this case, they will end up being re-arranged and potentially converted to the time domain before being propagated to the IVAS codec.

[0079] Audio encoding/decoding systems typically partition time-frequency space into frequency/time bins, for example, by applying suitable filter banks to the input audio signals. A frequency/time cell usually refers to a part of the time-frequency space corresponding to a frequency band and a time interval. The time interval may typically correspond to the duration of a time frame used in the audio encoding/decoding system. A frequency band is a portion of the entire frequency range of the audio signal/object that is being encoded or decoded. The frequency band may typically correspond to one or more contiguous frequency bands defined by a filter bank used in the encoding/decoding system. In case the frequency band corresponds to several contiguous frequency bands defined by the filter bank, this allows for non-uniform frequency bands in the process of decoding the downmix audio signal, for example, wider frequency bands for higher frequencies of the downmix audio signal.

[0080] In an implementation using a single MASA channel, there are at least two options for how the downmix matrix can be determined. One option is to select a microphone signal that has the best signal-to-noise ratio (SNR) for directional sound. In the configuration shown in FIG. 2, it is likely that microphone m1 captures the best signal since it is aimed at a directional sound source. Then signals from other microphones could be discarded. In this case, the downmix matrix could be as follows:

D = (1 0 0)

[0081] While the audio source is moving relative to the audio capture device, another more suitable microphone can be selected so that either the m ₂ signal or the m ₃ signal is used as the resulting MASA channel.

[0082] When switching microphone signals, it is important to ensure that the MASA channel signal does not suffer from any potential discontinuities. Discontinuities can occur due to different arrival times of a directional sound source at different microphones or due to different gain or phase characteristics of the acoustic path from the source to the microphones. Therefore, the individual delay, gain, and phase characteristics of the various microphone inputs must be analyzed and compensated for. Thus, the actual microphone signals may undergo some delay adjustment and filtering before being downmixed by MASA.

[0083] In another embodiment, the coefficients of the downmix matrix are set such that the SNR of the MASA channel with respect to the directional source is maximized. This can be achieved, for example, by adding different microphone signals with correctly adjusted weights κ _1.1 , κ _1.2 , κ _1.3 . In order to do this job efficiently, the individual delay, gain and phase characteristics of the various microphone inputs must again be analyzed and compensated for, which can also be understood as shaping the acoustic beam in the direction of a directional source.

[0084] Gain/phase adjustments can be understood as a frequency selective filtering operation. As such, the corresponding settings can also be optimized to achieve reduction in acoustic noise or enhancement of directional audio signals, for example according to Wiener's approach.

[0085] As an additional variation, an example with three MASA channels can be given. In this case, the downmix matrix can be defined by the following 3 by 3 matrix:

[0086] Therefore, there are now three signals x ₁ , x ₂ , x ₃ (instead of one in the first example) that can be encoded using the IVAS codec.

[0087] The first MASA channel may be generated as described in the first example. A second MASA channel can be used to carry a second directional audio if present. The downmix matrix coefficients can then be selected according to principles similar to those for the first MASA channel, however, such that the SNR of the second directional audio is maximized. The downmix matrix coefficients κ _3.1 , κ _3.2 , κ _3.3 for the third MASA channel can be adjusted to extract the diffuse audio component while minimizing directional sounds.

[0088] Typically, stereo capture of dominant directional sources in the presence of some background ambient sound may be performed, as shown in FIG. 2 and described above. This can often happen in certain applications, such as telephony. In accordance with various embodiments described in the present disclosure, metadata parameters are also determined in connection with the downmix, step 104, and will subsequently be added and distributed along with the single mono audio downmix signal.

[0089] In one embodiment, three basic metadata parameters are associated with each captured audio signal: a relative time delay value, a gain value, and a phase value. In accordance with the general approach, the MASA channel is obtained as a result of the following operations:

setting the delay of each microphone signal m _i ( i =1.2) to the value τ _i = Δτ _{i +} τ _ref .

adjusting the gain and phase of each time-frequency (TF) component/cell of each microphone signal with adjustable delay using the gain and phase control parameter, α and φ, respectively.

[0090] The delay setting element τ _i in the above expression can be interpreted as the arrival time of a plane sound wave from the directional source, and thus it is also convenient to express it as the arrival time relative to the arrival time of the sound wave at the reference point τ _ref , for example, at the geometric center of audio capture device 202, although any reference point may be used. For example, when using two microphones, the delay setting can be formulated as the difference between τ ₁ and τ ₂ , which is equivalent to moving the control point to the position of the second microphone. In one embodiment, the arrival time parameter allows the relative arrival time to be modeled in the interval [-2.0 ms, 2.0 ms], which corresponds to a maximum microphone offset from the origin of about 68 cm.

[0091] Regarding the gain and phase adjustments, in one embodiment these are parameterized for each TF cell such that gain changes can be modeled in the range [+10 dB, -30 dB] and phase changes can be represented in the range [- Pi, +Pi].

[0092] In a base case with only one dominant directional source, such as source 206 shown in FIG. 2, the delay setting is usually constant across the entire frequency spectrum. Since the position of the directional source 206 may change, the two delay settings (one for each microphone) will change over time. Thus, the delay settings depend on the signal.

[0093] In a more complex case where multiple directional sound sources 206 may be present, one source from a first direction may be dominant in a certain frequency band, while another source from a different direction may be dominant in another frequency band. In such a scenario, delay adjustment is preferably performed on a per-band basis instead.

[0094] In one embodiment, this can be done by compensating for the delay of the microphone signals in a given time-frequency (TF) cell relative to the sound direction that happens to be dominant. If no dominant sound direction is detected in the TF cell, no delay compensation is performed.

[0095] In another embodiment, microphone signals in a given TF cell may be delay compensated to maximize the signal-to-noise ratio (SNR) with respect to directional audio captured by all microphones.

[0096] In one embodiment, a suitable limit on the number of different sources for which delay compensation can be performed is three. This makes it possible to perform delay compensation in the TF cell either in relation to one of the three dominant sources, or not to perform it at all. The corresponding set of delay compensation values (the set applies to all microphone signals) can thus only be signaled by two bits per TF cell. This covers the most practical capture scenarios and has the advantage that the volume of metadata or its transfer rate remains low.

[0097] Another possible scenario is to capture and mix, for example, first order ambiophonic (FOA) signals rather than stereo signals into a single MASA channel. The concept of FOA is well known to those skilled in the art, but can be briefly described as a method for recording, mixing and playing back 3D audio in 360 degrees. The basic approach of ambiophony is to treat the soundstage as a complete 360-degree sphere of sound coming from different directions around a central point where the microphone is placed during recording or where the listener's "sweet spot" is located during playback.

[0098] Planar FOA capture and MASA downmix FOA are relatively simple extensions of the stereo capture case described above. The planar FOA case is characterized by three microphones, as shown in FIG. 2 performing capture before downmixing. In the latter case, the FOA is captured by four microphones whose placement or directivity extends across all three spatial dimensions.

[0099] Delay compensation, amplitude and phase adjustment parameters can be used to recover three or four original capture signals, respectively, and provide more accurate spatial rendering using MASA metadata than would be possible based on a mono downmix signal alone. Alternatively, delay compensation parameters, amplitude and phase adjustments can be used to create a more accurate (planar) FOA representation that approaches that which would be obtained using a regular microphone array.

[00100] In yet another scenario, a planar FOA or FOA may be captured and downmixed into two or more MASA channels. This case is an extension of the previous case with the difference that the captured three or four microphone signals are downmixed to two rather than just one MASA channel. The same principles apply when the purpose of providing delay compensation, amplitude and phase settings is to ensure the best possible restoration of the original signals before downmixing.

[00101] As one skilled in the art will appreciate, in order to accommodate all of these use cases, the spatial audio representation must include metadata not only about delay, gain and phase, but also about parameters that indicate the downmix configuration for the downmix audio signal .

[00102] Returning now to FIG. 1; certain metadata parameters are combined with the downmix audio signal into a spatial audio representation, step 108, which completes process 100. The following is a description of how these metadata parameters may be represented in accordance with one embodiment of the invention.

[00103] To support the downmix applications described above into one or multiple MASA channels, two metadata elements are used. One of the metadata elements is signal-independent configuration metadata that indicates downmixing. This metadata element is described below in connection with FIG. 3A-3B. Another metadata element is related to downmixing. This metadata element is described below in connection with FIG. 4-6 and may be determined as described above in connection with FIGS. 1. This element is required when transmitting a downmix signal.

[00104] In Table 1A shown in FIG. 3A, a metadata structure is presented that can be used to indicate the number of MASA channels, from one (mono) MASA channel, and two (stereo) MASA channels, to a maximum of four MASA channels, represented by channel bit values 00, 01, 10 and 11, respectively.

[00105] Table 1B shown in FIG. 3B contains the channel bit values from Table 1A (in this particular case, only channel values "00" and "01" are shown for illustrative purposes) and shows how a microphone capture configuration may be represented. For example, as can be seen in Table 1B for a single (mono) MASA channel, the signal may carry information about whether the acquisition configurations are mono, stereo, planar FOA, or FOA. As can also be seen in Table 1B, the microphone capture configuration is encoded as a 2-bit field (in the column called "bit value"). Table 1B also includes additional description of the metadata. An additional signal-independent configuration could, for example, represent that the sound is coming from the microphone array of a smartphone or similar device.

[00106] In the case where the downmix metadata is signal dependent, some additional details are necessary, as will be described below. As indicated in Table 1B for the specific case where the transport signal is a mono signal obtained by downmixing signals from multiple microphones, these details are provided in the signal-specific metadata field. The information provided in this metadata field describes the applied delay adjustment (for the possible purpose of shaping the acoustic beam towards directional sources) and filtering of the microphone signals (for the possible purpose of equalization/noise reduction) before downmixing. The result offers additional information that can help with encoding, decoding, and/or rendering.

[00107] In one embodiment, the downmix metadata includes four fields, a definition field, and selectors for signaling the applied delay compensation, followed by two fields signaling the applied gain and phase settings, respectively.

[00108] The number of microphone signals n that are downmixed is indicated in the "bit value" field in Table 1B, i.e. n=2 for stereo downmix ('bit value=01'), n=3 for in-plane FOA downmix ( 'bit value=10') and n=4 for FOA downmix ('bit value=11').

[00109] Up to three different sets of delay compensation values for up to n microphone signals can be defined and signaled for each TF cell. Each set corresponds to the direction of the directional source. Defining delay compensation value sets and signaling which set applies to which TF cell is done using two separate fields (definition and selector).

[00110] In one embodiment, the definition field is an nx3 matrix with 8-bit elements B _{i, j} encoding the applied delay compensation. These parameters correspond to the set to which they belong, that is, they correspond to the direction of the directed source ( j =1...3). In addition, the elements B _{i, j} correspond to the capture microphone (or the corresponding capture signal) ( i =1… n , n ≤4). This is illustrated schematically in Table 2 shown in FIG. 4.

[00111] FIG. 4 in combination with FIG. 3 thus shows an embodiment in which the spatial audio representation contains metadata parameters that are organized into a definition field and a selector field. The definition field specifies at least one set of delay compensation parameters associated with the plurality of microphones, and the selector field specifies a selection of a set of delay compensation parameters. The advantage of this representation of the relative time delay value between microphones is that it is compact and thus requires a lower data rate when transmitted to the next encoder or the like.

[00112] The delay compensation parameter represents the relative time of arrival of the assumed plane sound wave from the source side compared to the arrival of the wave at the (arbitrary) geometric center point of the audio capture device 202 . The encoding of this parameter using an 8-bit integer codeword B is done according to the following equation:

. Equation No. (1)

[00113] The result is a linear quantification of the relative delay in the interval [-2.0 ms, 2.0 ms], which corresponds to a maximum microphone offset from the reference point of about 68 cm. This is, of course, just one example, and other characteristics quantization and resolution may also be considered.

[00114] Signaling which set of delay compensation values is applied to which TF cell is done using a selector field representing 4*24 TF cells in a 20 ms frame, which assumes 4 subframes in a 20 ms frame and 24 bandwidths. Each field element contains a 2-bit set of encodings of 1...3 delay compensation values with the corresponding codes “01”, “10” and “11”. The "00" entry is used if no delay compensation is applied to the TF cell. This is illustrated schematically in Table 3 shown in FIG. 5.

[00115] The gain setting is signaled in 2-4 metadata fields, one for each microphone. Each field is a matrix of 8-bit gain adjustment B _α codes corresponding to 4*24 TF cells in a 20 ms frame. Encoding the gain settings using an integer codeword is done according to the following equation:

. Equation No. (2)

[00116] 2-4 metadata fields for each microphone are organized as shown in Table 4 shown in FIG. 6.

[00117] The phase setting is conveyed similarly to the gain setting in 2-4 metadata fields, one for each microphone. Each field is a matrix of 8-bit phase adjustment codes B _φ corresponding to 4*24 TF cells in a 20 ms frame. The encoding of phase adjustment parameters using an integer codeword B _φ is performed according to the following equation:

. Equation No. (3)

[00118] The 2-4 metadata fields for each microphone are organized as shown in Table 4, with the only difference being that the field elements are B _φ phase adjustment codewords.

[00119] This representation of MASA signals, which include associated metadata, can then be used by encoders, decoders, renderers, and other types of audio equipment that will be used to transmit, receive, and faithfully reconstruct the recorded spatial audio environment. Techniques for doing this are well known to those skilled in the art and can be easily adapted to suit the spatial audio presentation described in the present disclosure. Therefore, no further discussion of these specific devices is necessary in this context.

[00120] As one skilled in the art will appreciate, the metadata elements described above may be persistently stored or determined in various ways. For example, metadata may be defined locally on a device (such as an audio capture device, encoder, etc.), may otherwise be obtained from other data (such as from the cloud or other remote service), or may be stored in a predefined table values. For example, based on the delay setting between microphones, the delay compensation value (FIG. 4) for the microphone may be determined using a lookup table stored in the audio capture device, or received from a remote device based on a delay setting calculation made in the audio capture device, or received from such a remote device based on a delay setting calculation performed at the remote device (ie, based on the input signals).

[00121] In FIG. 7 illustrates a system 700 in accordance with an exemplary embodiment in which the features of the invention described above can be implemented. System 700 includes an audio capture device 202, an encoder 704, a decoder 706, and a renderer 708. The various components of system 700 may communicate with each other through a wired or wireless connection, or any combination thereof, and data is typically sent between units in the form of a bit stream. The audio capture device 202 has been described above and with reference to FIG. 2, and is configured to capture spatial sound, which is a combination of directional sound and diffuse sound. Audio capture device 202 creates a single-channel or multi-channel downmix audio signal by downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio. Audio capture device 202 then determines first metadata parameters associated with the downmix audio signal. This will be further illustrated below in connection with FIG. 8. The first metadata parameters indicate a relative time delay value, a gain value, and/or a phase value associated with each input audio signal. The audio capture device 202 finally combines the downmix audio signal and the first metadata parameters into a spatial audio representation. It should be noted that while in the current embodiment all audio capture and merging is performed in audio capture device 202, there may also be alternative embodiments in which certain portions of the creation, determination, and merging operations are performed in encoder 704.

[00122] Encoder 704 receives the spatial audio representation from audio capture device 202. That is, encoder 704 receives a data format comprising a single-channel or multi-channel downmix audio signal resulting from downmixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio, and first metadata parameters indicating a downmix configuration for the input audio signals. signals, a relative time delay value, a gain value, and/or a phase value associated with each input audio signal. It should be noted that the data format may be stored in non-volatile memory before/after it is received by the encoder. Encoder 704 then encodes the single-channel or multi-channel downmix audio signal into a bitstream using the first metadata. In some embodiments, encoder 704 may be an IVAS encoder as described above, but as one skilled in the art will appreciate, other types of encoders 704 may have similar capabilities and may also be used.

[00123] The encoded bitstream, which indicates the encoded representation of spatial audio, is then received by decoder 706. Decoder 706 decodes the bitstream into a spatial audio approximation using metadata parameters that are included in the bitstream from encoder 704. Finally, renderer 708 receives the decoded representation of the spatial audio and renders the spatial audio using the metadata to create an accurate reproduction of the spatial audio at the receiving end, for example, using one or more speakers.

[00124] In FIG. 8 shows an audio capture device 202 according to some embodiments. The audio capture device 202 may, in some embodiments, include a memory 802 with lookup tables stored therein for defining first and/or second metadata. The audio capture device 202 may, in some embodiments, be connected to a remote device 804 (which may be hosted in the cloud or be a physical device connected to the audio capture device 202), which may include a memory 806 with lookup tables stored therein for determining the first and /or second metadata. The audio capture device may, in some embodiments, perform the necessary calculations/processing (e.g., using processor 803), for example, to determine the relative time delay value, gain value, and phase value associated with each input audio signal, and transmit such parameters to the remote a device for receiving first and/or second metadata from the device. In other embodiments, audio capture device 202 transmits input signals to remote device 804, which performs the necessary calculations/processing (eg, using processor 805) and determines first and/or second metadata for transmission back to audio capture device 202. In yet another embodiment implementation, the remote device 804, which performs the necessary calculations/processing, passes the parameters back to the audio capture device 202, which determines the first and/or second metadata locally based on the received parameters (eg, using memory 806 with lookup tables stored therein).

[00125] In FIG. 9 shows a decoder 706 and a renderer 708 (each including a processor 910, 912 for performing various processing, such as decoding, rendering, etc.) according to embodiments. The decoder and renderer can be separate devices or the same device. Processor(s) 910, 912 may be shared between the decoder and renderer, or may be separate processors. As described with reference to FIG. 8, interpretation of the first and/or second metadata may be performed using a lookup table stored in either memory 902 in decoder 706, memory 904 in renderer 708, or memory 906 on remote device 905 (containing processor 908) connected either to the decoder or to the renderer.

Equivalents, extensions, alternatives and more

[00126] Additional embodiments of the present disclosure will become apparent to one skilled in the art upon examination of the above description. Although the present description and drawings disclose embodiments and examples, the disclosure is not limited to these specific examples. Numerous modifications and changes may be made without departing from the scope of the present disclosure as defined by the appended claims. Any numerical designations appearing in the claims should not be understood as limiting their scope.

[00127] Moreover, modifications to the disclosed embodiments can be understood and implemented by one skilled in the art of the disclosure based on examination of the drawings, the disclosure, and the accompanying claims. In the claims, the word “comprising” does not exclude other elements or steps, and the singular number does not exclude the plurality. The fact that certain measures are set forth in mutually different dependent claims does not mean that a combination of these measures cannot be used to advantage.

[00128] The systems and methods disclosed above may be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks among the functional blocks mentioned in the above description does not necessarily correspond to the division into physical blocks; in contrast, a single physical component can have multiple functions, and a single task can be performed jointly by multiple physical components. Some or all of the components may be implemented as software, executed by a digital signal processor or microprocessor, or implemented as hardware or an application specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or nonvolatile media) and communications media (or volatile media). As is well known to those skilled in the art, the term "computer storage media" includes both volatile and non-volatile, removable and non-removable media implemented by any methods or technologies for storing information, such as machine-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or others magnetic storage devices, or any other medium that can be used to store desired information and that can be accessed by a computer. In addition, one of ordinary skill in the art is well aware that communication media typically embody machine-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.

All figures are schematic and generally show only those parts necessary to illustrate the disclosure, while other parts may be omitted or merely implied. Unless otherwise indicated, like numerals refer to like parts in different figures.

Claims (81)

1. A method of presenting spatial sound, wherein the spatial sound is a combination of directional sound and diffuse sound, wherein said method includes: creating a single-channel or multi-channel downmix audio signal by downmixing input audio signals from a plurality of microphones (m1, m2, m3) in an audio capture unit capturing spatial audio, wherein the downmixing to create a single-channel or multi-channel downmix audio signal x is described as follows : , where D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and m is a matrix representing input audio signals from a plurality of microphones; determining first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And combining the generated downmix audio signal and the first metadata parameters into a spatial audio representation. 2. The method of claim 1, wherein combining the generated downmix audio signal and the first metadata parameters into a spatial audio representation further comprises: including second metadata parameters in the spatial audio representation, wherein the second metadata parameters indicate a downmix configuration for the input audio signals. 3. The method of claim 1 or 2, wherein the first metadata parameters are determined for one or more frequency bands of the microphone audio input signals. 4. The method of claim 3, wherein the downmix coefficients are selected to select the microphone audio input signal currently having the best signal-to-noise ratio with respect to the directional audio, and to discard audio input signals from any other microphones. 5. The method of claim 4, wherein the selection is performed for each time-frequency (TF) cell. 6. The method of claim 4, wherein the selection is performed for all frequency bands of a particular audio frame. 7. The method of claim 3, wherein the downmix coefficients are selected to maximize the signal-to-noise ratio with respect to directional audio when combining input audio signals from different microphones. 8. The method of claim 7, wherein the maximization is performed for a specific frequency band. 9. The method of claim 7, wherein the maximization is performed for a specific audio frame. 10. Method according to any one of paragraphs. 1-9, wherein determining the first metadata parameters includes analyzing one or more of the following: delay, gain, and phase characteristics of input audio signals from a plurality of microphones. 11. Method according to any one of paragraphs. 1-10, in which the first metadata parameters are determined for each time-frequency (TF) cell. 12. Method according to any one of paragraphs. 1-11, wherein at least a portion of the downmixing occurs in an audio capture unit. 13. Method according to any one of paragraphs. 1-11, in which at least a portion of the downmixing occurs in the encoder. 14. Method according to any one of paragraphs. 1-13, further including: in response to detection of more than one directional sound source, determining first metadata for each source. 15. Method according to any one of paragraphs. 1-14, in which the spatial sound representation includes at least one of the following parameters: a direction index, a ratio of direct to total energy; propagation coherence; arrival time, gain and phase for each microphone; ratio of dissipated energy to total energy; volumetric coherence; ratio of remainder to total energy; and distance. 16. Method according to any one of paragraphs. 1-15, wherein the second or first metadata parameter metadata parameter indicates whether the generated downmix audio signal is generated from: left and right stereo signals, first order planar ambiophonic (FOA) signals, or first order component ambiophonic signals. 17. Method according to any one of paragraphs. 1-16, wherein the spatial audio representation comprises metadata parameters organized in a definition field and a selector field, wherein the definition field specifies at least one set of delay compensation parameters associated with a plurality of microphones, and the selector field specifies a selection of a set of delay compensation parameters. 18. The method of claim 17, wherein the selector field specifies which set of delay compensation parameters are applied to any given time-frequency cell. 19. Method according to any one of paragraphs. 1-18, in which the value of the relative time delay is approximately in the interval [-2.0 ms, 2.0 ms]. 20. The method of claim 17, wherein the metadata parameters in the surround sound view further comprise a field specifying an applied gain setting and a field specifying a phase setting. 21. The method of claim 20, wherein the gain setting is approximately in the range [+10 dB, -30 dB]. 22. Method according to any one of paragraphs. 1-21, wherein at least portions of the first and/or second metadata elements are determined in the audio capture device using lookup tables stored in memory. 23. Method according to any one of paragraphs. 1-21, wherein at least portions of the first and/or second metadata elements are determined on a remote device connected to the audio capture device. 24. A system for presenting spatial sound, comprising: a receiving component configured to receive input audio signals from a plurality of microphones (m1, m2, m3) in the audio capturing unit capturing spatial audio; a downmix component configured to create a single-channel or multi-channel downmix audio signal by downmixing received audio signals, wherein the downmixing to create a single-channel or multi-channel downmix audio signal x is described as follows: , where D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and m is a matrix representing input audio signals from a plurality of microphones; a metadata determining component configured to determine first metadata parameters associated with the downmix audio signal, the first metadata parameters indicating one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And a combining component configured to combine the generated downmix audio signal and the first metadata parameters into a spatial audio representation. 25. The system of claim 24, wherein the combining component is further configured to include second metadata parameters in the spatial audio representation, wherein the second metadata parameters indicate a downmix configuration for the input audio signals. 26. A computer-readable medium on which instructions are stored that cause the processor to execute the method of any one of claims. 1-23. 27. Spatial audio encoder configured for: receiving a spatial sound representation containing: a single-channel or multi-channel down-mix audio signal created by down-mixing input audio signals from a plurality of microphones (m1, m2, m3) in an audio capture unit capturing spatial audio, wherein the down-mixing to create a single-channel or multi-channel down-mix audio signal is described as follows way: , where D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and m is a matrix representing the input audio signals from a plurality of microphones, and first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And doing one of: encoding the single-channel or multi-channel downmix audio signal into a bitstream using the first metadata, and encoding the single-channel or multi-channel downmix audio signal and the first metadata into a bit stream. 28. Encoder according to clause 27, in which: the spatial audio representation further includes second metadata parameters indicating a downmix configuration for the input audio signals; And the encoder is configured to encode the single-channel or multi-channel downmix audio signal into a bitstream using the first and second metadata parameters. 29. The encoder of claim 27, wherein the downmixing portion occurs in the audio capture unit, and the downmixing portion occurs in the encoder. 30. Spatial audio decoder configured for: receiving a bitstream indicating an encoded spatial audio representation, said representation including: a single-channel or multi-channel down-mix audio signal created by down-mixing input audio signals from a plurality of microphones (m1, m2, m3) in an audio capture unit (202) capturing spatial audio, down-mixing to create a single-channel or multi-channel down-mix audio signal mixing is described as follows: , where D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and m is a matrix representing the input audio signals from a plurality of microphones, and first metadata parameters associated with the downmix audio signal, wherein the first metadata parameters indicate one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And decoding the bitstream into a spatial audio approximation using the first metadata parameters. 31. The decoder according to claim 30, in which: the spatial audio representation further includes second metadata parameters indicating a downmix configuration for the input audio signals; And the decoder is configured to decode the bitstream into a spatial audio approximation using the first and second metadata parameters. 32. The decoder according to claim 30 or 31, additionally including: using the first metadata parameter to recover inter-channel time differences or to adjust the magnitude or phase of the decoded audio output. 33. The decoder according to claim 31, additionally including: using a second metadata parameter to define an upmix matrix for directional source signal recovery or surround sound signal recovery. 34. Spatial audio renderer configured for: receiving a spatial sound representation, said representation including: a single-channel or multi-channel downmix audio signal created by downmixing input audio signals from a plurality of microphones (m1, m2, m3) in an audio capture unit capturing spatial audio, wherein the downmixing to create a single-channel or multi-channel downmix audio signal is described as follows way: , where D is a downmix matrix containing downmix coefficients defining weights for each input audio signal from the plurality of microphones, and m is a matrix representing the input audio signals from a plurality of microphones, and first metadata parameters associated with the downmix audio signal, the first metadata parameters indicating one or more of: a relative time delay value, a gain value, and a phase value associated with each input audio signal; And rendering spatial audio using first metadata. 35. The rendering tool according to claim 34, in which: the spatial audio representation further includes second metadata parameters indicating a downmix configuration for the input audio signals; And the renderer is configured to render spatial audio using the first and second metadata parameters.

Images