RU2427978C2 - Audio coding and decoding - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: audio signal coder comprises a facility to receive M-channel audio signal, where M>2, a facility of downmix to downmix M-channel audio signal into the first stereo signal and related parametric data, a facility of modification to modify the first stereo signal in order to generate the second stereo signal in response to related parametric data and data of spatial parameters, which specify transfer function of binaural perception, besides, the second stereo signal is a binaural signal, a facility for coding of the second stereo signal with the purpose to generate coded data and an output facility to generate out data flow, containing coded data and related parametric data.

EFFECT: increased efficiency of stereo coding of multichannel signals with reduction of coding complexity.

35 cl, 11 dwg

Description

The invention relates to encoding and / or decoding of audio, in particular, but not exclusively, to encoding and / or decoding of audio, including binaural virtual spatial signal.

Digital coding of various source signals has become increasingly important over the past decades, since digital representation and signal transmission have largely replaced analog representation and transmission. For example, the distribution of audiovisual content such as video and music is more based on the encoding of digital content.

Moreover, in the last decade, there has been a tendency towards multichannel audio, and especially towards spatial audio that goes beyond traditional stereo signals. For example, traditional stereo recordings contain only two channels, while modern advanced audio systems typically use five or six channels, as in the popular 5.1 surround sound systems. This provides a more involved listening experience where the user may be surrounded by sound sources.

Various technologies and standards have been developed to transmit such multi-channel signals. For example, six discrete channels representing a 5.1 surround sound system can be transmitted in accordance with standards such as Advanced Audio Coding (AAC) or Dolby Digital.

However, in order to ensure backward compatibility, it is known that it is necessary to down-mix more channels to fewer, and, more precisely, this is often used to down-mix 5.1 surround signal to a stereo signal, giving the stereo signal the ability to play back (stereo) decoders of the previous system, and the signal 5.1 surround sound decoders.

One example is the backward compatible encoding method of the MPEG2 standard. The multi-channel signal is down-mixed into a stereo signal. Additional signals are encoded in a portion of overhead data, giving the MPEG2 multi-channel decoder the ability to form a multi-channel signal representation. The MPEG1 decoder will ignore overhead and thus only decode the down-mix stereo signal. The main disadvantage of the encoding method used in MPEG2 is that the additional data rate required for additional signals is in the same order of magnitude as the data rate required for encoding a stereo signal. The additional data rate for expanding stereo to multi-channel audio, therefore, is significant.

Other existing methods for backward compatible multichannel transmission without additional multichannel information can typically be characterized by matrixed padding methods. Examples of matrix encoding for surround sound include methods such as Dolby Prologic II and Logic-7. The general principle of these methods is that they matrixly multiply multiple channels of the input signal by a suitable non-square matrix, thereby forming an output signal with fewer channels. More specifically, a matrix encoder typically applies phase shifts to surround channels before mixing them with the front and center channels.

Another reason for channel conversion is coding efficiency. It has been found that, for example, surround audio signals can be encoded as stereo channel audio signals combined with a parametric bitstream describing the spatial properties of the audio signal. The decoder can reproduce stereo audio signals with a very satisfactory degree of accuracy. Thus, significant savings in bit rate can be obtained.

There are several parameters that can be used to describe the spatial properties of audio signals. One such parameter is cross-channel cross-correlation, such as cross-correlation between the left channel and the right channel for stereo signals. Another parameter is the channel power ratio. In the so-called (parametric) spatial audio encoders, these and other parameters are extracted from the original audio signal in order to reproduce an audio signal having a reduced number of channels, for example, only one channel, plus a set of parameters describing the spatial properties of the original audio signal. In the so-called (parametric) spatial decoders, the audio spatial properties as described by the transmitted spatial parameters are restored.

Such spatial audio encoding preferably utilizes a cascading or tree-based hierarchical structure comprising standard units in an encoder and decoder. In the encoder, these building blocks can be downmixers combining channels into fewer channels, such as 2-in-1, 3-in-1, 3-in-2 downmixers, etc., along with the decoder, the corresponding building blocks can be boosters that split channels into more channels, such as boosters 1-in-2, 2-in-3.

An example of a system in which a multi-channel signal is down-mixed into a stereo signal, which is subsequently post-processed using spatial down-mix parameters, is presented in patent application publication WO 2005 / 098826A under the Patent Cooperation Treaty. An example of an audio encoding and decoding apparatus capable of transmitting an audio signal or an audio signal together with an audio signal processed by sound field effects is given in US Patent Application Publication US2005 / 0273322A1.

3-dimensional (3D) positioning of the sound source is currently gaining interest, especially in the mobile field. Playing music and sound effects in mobile games can add significant value to the consumer experience when positioned in 3 dimensions, effectively creating a 3-dimensional 'head-to-head' effect. More precisely, it is known that binaural audio signals that contain direction-specific information to which the human ear is sensitive should be recorded and reproduced. Binaural recordings are typically made using two microphones mounted on a human head simulator, so that the recorded sound corresponds to the sound picked up by the human ear and includes any influences due to the shape of the head or ears. Binaural recordings differ from stereo (i.e. stereo) recordings in that the binaural recording is usually intended for use with headphones or headphones, while stereo recording is usually done for playback with speakers. While binaural recording provides the ability to reproduce all spatial information using only two channels, stereo recording would not provide the same spatial perception. Conventional two-channel (stereo) or multi-channel (e.g. 5.1) recordings can be transformed into binaural recordings by convolution of each ordinary signal with a set of transfer perception functions. Such transfer functions of perception model the influence of the human head and, possibly, other objects on the signal. A widely known type of transfer function of spatial perception is the so-called Head-Related Transfer Function (HRTF). An alternative type of transfer function of spatial perception, which also takes into account the reflections caused by the walls, ceiling and floor of the room, is the binaural impulse response of the room (BRIR).

Typically, 3D positioning algorithms employ HRTFs, which describe the transfer from a certain location of a sound source to the eardrums via an impulse response. The 3D positioning of the sound source can be applied to multi-channel signals via HRTF, thereby enabling binaural signals to deliver spatial sound information to the user, for example, using a pair of headphones.

It is known that the perception of the elevation angle is predominantly facilitated by certain peaks and dips in the spectra coming in both ears. On the other hand, the (perceived) course angle of the sound source is captured in the 'binaural' control signals, such as level differences and differences in arrival times between the signals on the eardrum. The perception of distance is for the most part facilitated by the overall signal level and, in the case of a reverberant environment, the ratio of directed and reverberation energy. In most cases, it is allowed that, especially in the last final phase of the reverb, there are no tell-tale signals to reliably determine the location of the sound source.

Perception tell-tales for elevation, course angle and distance can be captured by (a pair) of impulse responses: one impulse response to describe transmission from a specific position of the sound source to the left ear; and one for the right ear. From here, control signals of perception for elevation, course angle and distance are determined by the corresponding properties (pairs) of HRTF impulse characteristics. In most cases, an HRTF pair is measured for a large set of sound source locations; typically with a spatial resolution of approximately 5 degrees both in elevation and course angle.

Traditional binaural 3-dimensional synthesis contains filtering (convolution) of the input signal with a pair of HRTF for the desired location of the sound source. However, since HRTFs are typically measured in anechoic conditions, perception of 'distance' or location of 'off-head' is often absent. Although convolution of the signal with anechoic HRTFs is not sufficient for 3D sound synthesis, the use of anechoic HRTFs is often preferred in terms of complexity and flexibility. The effect of an echo-containing environment (required to create a perception of distance) can be added at a later stage, leaving some flexibility for the end user to modify the acoustic properties of the room. Moreover, since it is often assumed that reverb is unidirectional (without directional control signals), this processing method is often more efficient than convolving each sound source with an echo-containing HRTF pair. Moreover, in addition to the arguments of complexity and flexibility for room acoustics, the use of anechoic HRTF also has the advantage of synthesizing signals (directional control signals) 'with flat sound'.

A recent study of 3D positioning showed that the frequency resolution, which is represented by the anechoic impulse response of HRTF, is in many cases higher than necessary. More precisely, it seems that for both phase and amplitude spectra, the nonlinear frequency resolution proposed by the ERB scale is sufficient for the synthesis of 3-dimensional sound sources with an accuracy that does not differ in perception from processing with full anechoic HRTFs. In other words, the spectra of anechoic HRTFs do not require a spectral resolution that is higher than the frequency resolution of the human auditory system.

The traditional binaural synthesis algorithm is outlined in FIG. The set of input channels is filtered by the HRTF set. Each input channel is split into two signals (left 'L' and right 'R' components); each of these signals is subsequently filtered by HRTF corresponding to the desired location of the sound source. All left ear signals are subsequently summed to form a left binaural output signal, and right ear signals are summed to form a right binaural output signal.

HRTF convolution can be performed in the time domain, but it is often preferable to perform filtering as a product in the frequency domain. In this case, summation can also be performed in the frequency domain.

Known decoder systems that can receive an encoded surround sound signal and provide the impression of surround sound from a binaural signal. For example, headphone systems are known that provide the surround signal with the ability to convert to a binaural surround signal to provide a surround sound experience to the headphone user.

2 illustrates a system in which an MPEG surround decoder receives a stereo signal with spatial parametric data. The input bitstream is demultiplexed, resulting in spatial parameters and a down-mix bitstream. The last bitstream is decoded using a traditional mono or stereo decoder. The decoded down-mix signal is decoded by a spatial decoder, which generates a multi-channel output signal based on the transmitted spatial parameters. In conclusion, the multi-channel output signal is then processed by a cascade of binaural synthesis (similar to that of FIG. 1), resulting in a binaural output signal giving the impression of surround sound to the user.

However, this approach has a number of disadvantages.

For example, cascading a spatial sound decoder and binaural synthesis involves computing a multi-channel signal representation as an intermediate step, followed by HRTF convolution and downmixing in the binaural synthesis step. This may result in increased complexity and reduced performance.

In addition, the system is very complex. For example, spatial decoders typically operate in the subband domain (QMF). HRTF convolution, on the other hand, can typically be implemented most efficiently in the field of FFT (FFT, Fast Fourier Transform). Therefore, it is necessary to cascade the multichannel comb of QMF synthesis filters, multichannel FFT conversion and stereo inverse FFT conversion, resulting in a system with high computing needs.

The quality of the user experience provided may be reduced. For example, coding artifacts generated by a spatial decoder to create multi-channel reconstruction will still be heard in the (stereo) binaural output.

Moreover, the approach requires dedicated decoders and complex signal processing, which must be performed by individual user devices. This can slow down the application in many situations. For example, devices of previous systems that are only capable of decoding stereo downmixes will not be able to provide a user experience of surround sound.

From here, enhanced audio coding / decoding would be useful.

Accordingly, the invention preferably seeks to mitigate, alleviate or eliminate one or more of the above disadvantages separately or in any combination.

According to a first aspect of the invention, there is provided an audio encoder comprising: means for receiving an M-channel audio signal, where M> 2; down-mix means for down-mixing the M-channel audio signal into a first stereo signal and associated parameter data; generating means for modifying the first stereo signal to generate a second stereo signal, in response to the associated parametric data and spatial parameter data for the binaural perception transfer function, the second stereo signal is a binaural signal; means for encoding a second stereo signal to generate encoded data; and output means for generating an output data stream comprising encoded data and associated parametric data.

The invention may provide an opportunity for improved audio coding. In particular, the invention can provide the ability to efficiently stereo-encode multi-channel signals, while providing stereo decoders of previous systems with the ability to provide an enhanced spatial impression. Moreover, the invention provides the ability to reverse the process of binaural virtual spatial synthesis in the decoder, thereby enabling high-quality multi-channel decoding. The invention may provide the ability of a low complexity encoder and, in particular, may enable the formation of a low complexity binaural signal. The invention may provide the possibility of facilitated implementation and reuse of functionality.

The invention, in particular, can provide a parameter-based definition of a binaural virtual spatial signal from a multi-channel signal.

The binaural signal, more precisely, can be a binaural virtual spatial signal, such as a virtual 3-dimensional binaural stereo signal. The M channel audio signal may be a surround signal, such as a 5.1 or 7.1 surround signal. A binaural virtual surround signal can simulate one location of a sound source for each channel of an M-channel audio signal. The spatial parameter data may contain data indicating a transfer function from the intended location of the sound source to the eardrum of the intended user.

The binaural perception transfer function, for example, can be a human sound perception modeling function (HRTF) or a room binaural impulse response characteristic (BRIR).

According to an additional feature of the invention, the generating means is configured to generate a second stereo signal by calculating subband data values for the second stereo signal in response to associated parametric data, spatial parameter data, and subband data values for the first stereo signal.

This may provide improved coding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources. The intervals of the frequency subbands of the first stereo signal, the second stereo signal, the associated parametric data and the spatial parameter data may be different, or some or all of the subbands may be substantially identical for some or all of them.

According to an additional feature of the invention, the generating means is configured to generate subband values for the first subband of the second stereo signal in response to multiplying the corresponding stereo values of the subband for the first stereo signal by the matrix of the first subband; the generating means further comprises parameter means for determining data values for the matrix of the first subband in response to the associated parametric data and spatial parameter data for the first subband.

This may provide improved coding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources. The invention, in particular, can provide a parameter-based definition of a binaural virtual spatial signal from a multi-channel signal by performing matrix operations on individual subbands. The values of the matrix of the first subband may reflect the combined result of the cascading inclusion of multi-channel decoding and HRTF / BRIR filtering of the resulting multiple channels. Subband matrix multiplication can be performed for all subbands of the second stereo signal.

According to an additional feature of the invention, the generating means further comprises means for converting the data value of at least one of the stereo signal, the associated parametric data and the spatial parameter data associated with a subband containing a frequency interval other than the interval of the first subband to a corresponding data value for the first subrange.

This may provide improved coding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources. More specifically, the invention can provide different processes and algorithms with the ability to rely on sub-band separations that are most suitable for an individual process.

According to an additional feature of the invention, the shaping means is configured to determine the stereo values of the subband L _B , R _B for the first subband of the second stereo signal, essentially as:

,

,

wherein L ₀ , R ₀ are the corresponding values of the subband of the first stereo signal, and the parameter tool is configured to determine the data values of the multiplication matrix essentially as:

h ₁₁ = m ₁₁ H _L ( L ) + m ₂₁ H _L ( R ) + m ₃₁ H _L ( C )

h ₁₂ = m ₁₂ H _L ( L ) + m ₂₂ H _L ( R ) + m ₃₂ H _L ( C )

h ₂₁ = m ₁₁ H _R ( L ) + m ₂₁ H _R ( R ) + m ₃₁ H _R ( C )

h ₂₂ = m ₁₂ H _R ( L ) + m ₂₂ H _R ( R ) + m ₃₂ H _R ( C ),

where m _{k, l} are parameters determined in response to the associated parametric data for downmixing by means of downmixing channels L, R and C into the first stereo signal; and H _J (X) is determined in response to the spatial parameter data for channel X with respect to the stereo output channel J of the second stereo signal.

This may provide improved coding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources.

According to a further feature of the invention, at least one of the channels L and R corresponds to down-mix of at least two down-mixed channels, and the parameter means is configured to determine H _J (X) in response to a weighted combination of spatial parameter data for at least two downmix channels.

This may provide improved coding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources.

According to an additional feature of the invention, the parameter means is configured to determine the weighting of the spatial parameter data for at least two downmix channels in response to a relative measure of energy for the at least two downmix channels.

This may provide improved coding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources.

According to a further feature of the invention, the spatial parameter data includes at least one parameter selected from the group consisting of: a mid-range parameter over a subband; parameter of average time of receipt; phase of at least one stereo channel; time reference parameter; group delay parameter; phases between stereo channels; and channel cross-correlation parameter.

These parameters can provide highly predominant coding and, in particular, can be particularly suitable for processing subbands.

According to an additional feature of the invention, the output means is configured to include location data of the sound source in the output stream.

This may provide the decoder with the ability to determine suitable spatial parameter data and / or may provide an efficient way of indicating spatial parameter data with low overhead. This can provide an efficient way to reverse the binaural virtual spatial synthesis process in the decoder, thereby enabling high-quality multi-channel decoding. The feature, moreover, may provide an opportunity for an improved user experience and may enable or facilitate the implementation of binaural virtual spatial signal with moving sound sources. The feature, alternatively or additionally, may provide the ability to configure spatial synthesis in the decoder, for example, by first reversing the synthesis performed in the encoder, followed by synthesis using a customized or individualized binaural perception transfer function.

According to an additional feature of the invention, the output means is configured to include at least some of the spatial parameter data in the output stream.

This can provide an efficient way to reverse the binaural virtual spatial synthesis process in the decoder, thereby enabling high-quality multi-channel decoding. The feature, moreover, may provide an opportunity for an improved user experience and may enable or facilitate the implementation of binaural virtual spatial signal with moving sound sources. The spatial parameter data can be directly or indirectly included in the output stream, for example, by including in the composition of the information, which provides the decoder with the ability to determine the spatial parameter data. The feature, alternatively or additionally, may provide the ability to configure spatial synthesis in the decoder, for example, by first reversing the synthesis performed in the encoder, followed by synthesis using a customized or individualized binaural perception transfer function.

According to a further feature of the invention, the encoder further comprises means for determining the spatial parameter data in response to the desired locations of the audio signal.

This may provide improved coding and / or lightweight implementation. The desired locations of the audio signal may correspond to the locations of the sound sources for individual channels of the M-channel signal.

According to another aspect of the invention, there is provided an audio decoder comprising: means for receiving input data comprising a first stereo signal and parametric data associated with a down-mixed stereo signal of an M-channel audio signal, where M> 2, the first stereo signal is a binaural signal corresponding to the M-channel audio signal; and generating means for modifying the first stereo signal to generate the down-mixed stereo signal, in response to the parametric data and the first spatial parameter data for the binaural perception transfer function, the first spatial parameter data is associated with the first stereo signal.

The invention may provide enhanced audio decoding. In particular, the invention can provide high-quality stereo decoding and, more precisely, can enable the encoder binaural virtual spatial synthesis process to be reversed in a decoder. The invention may provide the ability to perform a low complexity audio decoder. The invention may provide the possibility of facilitated implementation and reuse of functionality.

The binaural signal, more precisely, can be a binaural virtual spatial signal, such as a virtual 3-dimensional binaural stereo signal. The spatial parameter data may contain data indicating a transfer function from the intended location of the sound source to the intended user's ear. The binaural perception transfer function, for example, can be a human sound perception modeling function (HRTF) or a room binaural impulse response characteristic (BRIR).

According to a further feature of the invention, the audio signal decoder further comprises means for generating an M-channel audio signal in response to the down-mixed stereo signal and parametric data.

The invention may provide enhanced audio decoding. In particular, the invention can provide high-quality multi-channel decoding and, more precisely, can enable the encoder binaural virtual spatial synthesis process to be reversed in the decoder. The invention may provide a low complexity audio decoder. The invention may provide the possibility of facilitated implementation and reuse of functionality.

The M channel audio signal may be a surround signal, such as a 5.1 or 7.1 surround signal. The binaural signal may be a virtual spatial signal that simulates one location of a sound source for each channel of an M-channel audio signal.

According to a further feature of the invention, the generating means is configured to generate the down-mixed stereo signal by calculating the subband data values for the down-mixed stereo signal in response to the associated parametric data, spatial parameter data and sub-band data values for the first stereo signal.

This may provide improved decoding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources. The intervals of the frequency subbands of the first stereo signal subjected to down-mixing of the stereo signal, the associated parametric data and spatial parameter data may be different, or some or all of the subbands may be substantially identical for some or all of them.

According to a further feature of the invention, the generating means is configured to generate subband values for the first subband of the stereo down-mixed stereo signal in response to multiplying the corresponding stereo subband values for the first stereo signal by the matrix of the first subband;

the generating means further comprises parameter means for determining data values for the matrix of the first subband in response to the parametric data and spatial parameter data for the first subband.

This may provide improved decoding and / or lightweight implementation. More specifically, the feature may provide reduced complexity and / or reduced computational resources. The values of the matrix of the first subband may reflect the combined result of the cascading inclusion of multi-channel decoding and HRTF / BRIR filtering of the resulting multiple channels. Subband matrix multiplication can be performed for all subbands of the down-mixed stereo signal.

According to a further feature of the invention, the input data contains at least some spatial parameter data.

This can provide an efficient way to reverse the binaural virtual spatial synthesis process performed in an encoder, thereby enabling high-quality multi-channel decoding. The feature, moreover, may provide an opportunity for an improved user experience and may enable or facilitate the implementation of binaural virtual spatial signal with moving sound sources. The spatial parameter data can be directly or indirectly included in the input data, for example, they can be any information that allows the decoder to determine the spatial parameter data.

According to a further feature of the invention, the input data contains location data of the sound source, and the decoder comprises means for determining spatial parameter data in response to the location data of the sound source.

This may provide improved coding and / or lightweight implementation. The desired locations of the audio signal may correspond to the locations of the sound sources for individual channels of the M-channel signal.

The decoder, for example, may comprise a data store containing HRTF spatial parameter data associated with different locations of the sound source, and may determine spatial parameter data for use by retrieving the parameter data for the specified locations.

According to an additional feature of the invention, the audio signal decoder further comprises a spatial decoder unit for generating a pair of binaural output channels by modifying the first stereo signal in response to the associated parametric data and second spatial parameter data for the second binaural perception transfer function, the second spatial parameter data is different than the first spatial parameter data .

The feature may provide an opportunity for improved spatial synthesis and, in particular, may enable an individual or customized spatial synthesized binaural signal, which is especially suitable for a particular user. This can still be achieved, along with providing stereo decoders of previous systems with the ability to generate spatial binaural signals without requiring spatial synthesis in the decoder. From here, an improved audio system can be achieved. The second transfer function of binaural perception, more precisely, may be different than the transfer function of binaural perception of the first spatial data. The second binaural transfer function and the second spatial data, more precisely, can be adjusted for the individual user of the decoder.

According to an additional feature of the invention, the spatial decoder comprises: a parameter conversion unit for converting parametric data into binaural synthesis parameters using second spatial parameter data and a spatial synthesis unit for synthesizing a pair of binaural channels using binaural synthesis parameters and a first stereo signal.

This may provide improved performance, and / or lightweight implementation, and / or reduced complexity. The binaural parameters may be parameters that can be multiplied with the sub-band sound samples of the first stereo signal and / or the down-mixed stereo signal to form sub-band sound samples for binaural channels. Multiplication, for example, can be matrix multiplication.

According to an additional feature of the invention, binaural synthesis parameters contain matrix coefficients for a 2 by 2 matrix defining the ratio of stereo samples of the stereo-downmixed stereo signal to stereo samples of a pair of binaural output channels.

This may provide improved performance, and / or lightweight implementation, and / or reduced complexity. The stereo samples of the sound can be stereo samples of the sound of a subband, for example, the frequency subbands of the QMF or Fourier transform.

According to an additional feature of the invention, binaural synthesis parameters comprise matrix coefficients for a 2 by 2 matrix defining the ratio of stereo samples of a subband of the first stereo signal to stereo samples of a pair of binaural output channels.

This may provide improved performance, and / or lightweight implementation, and / or reduced complexity. The stereo samples of the sound can be stereo samples of the sound of a subband, for example, the frequency subbands of the QMF or Fourier transform.

According to another aspect of the invention, there is provided a method for encoding an audio signal, the method comprising: receiving an M-channel audio signal, where M> 2; down-mixing the M-channel audio signal into a first stereo signal and associated parametric data; modifying the first stereo signal to generate a second stereo signal, in response to the associated parametric data and spatial parameter data for the binaural perception transfer function, the second stereo signal is a binaural signal; encoding a second stereo signal to generate encoded data; and generating an output data stream containing encoded data and associated parametric data.

According to another aspect of the invention, there is provided a method for decoding an audio signal, the method comprising:

receiving input data containing the first stereo signal and parametric data associated with the down-mixed stereo signal of the M-channel audio signal, where M> 2, the first stereo signal is a binaural signal corresponding to the M-channel audio signal; and

modifying the first stereo signal to produce a down-mixed stereo signal, in response to parametric data and spatial parameter data for the binaural perception transfer function, the spatial parameter data is associated with the first stereo signal.

According to another aspect of the invention, there is provided a receiver for receiving an audio signal comprising: means for receiving input data comprising a first stereo signal and parametric data associated with a down-mixed stereo signal of an M-channel audio signal, where M> 2, the first stereo signal is a binaural signal corresponding to M -channel audio signal; and generating means for modifying the first stereo signal to generate the down-mixed stereo signal, in response to the parametric data and spatial parameter data for the binaural perception transfer function, the spatial parameter data is associated with the first stereo signal.

According to another aspect of the invention, there is provided a transmitter for transmitting an output data stream, the transmitter comprising: means for receiving an M-channel audio signal, where M> 2; down-mix means for down-mixing the M-channel audio signal into a first stereo signal and associated parameter data; generating means for modifying the first stereo signal to generate a second stereo signal, in response to the associated parametric data and spatial parameter data for the binaural perception transfer function, the second stereo signal is a binaural signal; means for encoding a second stereo signal to generate encoded data; output means for generating an output data stream containing encoded data and associated parametric data; and means for transmitting the output data stream.

According to yet another aspect of the invention, there is provided a transmission system for transmitting an audio signal, a transmission system comprising: a transmitter comprising: means for receiving an M-channel audio signal, where M> 2, down-mix means for down-mixing an M-channel audio signal into a first stereo signal and related parametric data , forming means for modifying the first stereo signal to generate a second stereo signal, in response to the associated parametric data and spatial parameter data for a binaural perception transfer function, the second stereo signal is a binaural signal, means for encoding a second stereo signal to generate encoded data, output means for generating an output audio data stream containing encoded data and associated parametric data, and means for transmitting an output audio data stream; and a receiver comprising: means for receiving an output stream of audio data and means for modifying a second stereo signal to generate a first stereo signal in response to parametric data and spatial parameter data.

According to another aspect of the invention, there is provided a method for receiving an audio signal, the method comprising: receiving input data comprising a first stereo signal and parametric data associated with a down-mixed stereo signal of an M-channel audio signal, where M> 2, the first stereo signal is a binaural signal corresponding to the M-channel audio signal; and modifying the first stereo signal to produce a down-mixed stereo signal in response to parametric data and spatial parameter data for the binaural perception transfer function, the spatial parameter data is associated with the first stereo signal.

According to another aspect of the invention, there is provided a method for transmitting an audio data output stream, the method comprising: receiving an M-channel audio signal, where M> 2; down-mixing the M-channel audio signal into a first stereo signal and associated parametric data; modifying the first stereo signal to generate a second stereo signal, in response to the associated parametric data and spatial parameter data for the binaural perception transfer function, the second stereo signal is a binaural signal; encoding a second stereo signal to generate encoded data; and generating an output audio data stream containing encoded data and associated parametric data; and transmitting the audio output data stream.

According to another aspect of the invention, there is provided a method for transmitting and receiving an audio signal, the method comprising: receiving an M-channel audio signal, where M> 2; down-mixing the M-channel audio signal into a first stereo signal and associated parametric data; modifying the first stereo signal to generate a second stereo signal, in response to the associated parametric data and spatial parameter data for the binaural perception transfer function, the second stereo signal is a binaural signal; encoding a second stereo signal to generate encoded data; and generating an output audio data stream containing encoded data and associated parametric data; transmission of the output audio data stream; receiving output audio data stream; and modifying the second stereo signal to generate a first stereo signal in response to parametric data and spatial parameter data.

According to another aspect of the invention, a computer program product for performing any of the above methods is provided.

According to another aspect of the invention, there is provided an audio recording apparatus comprising an encoder according to the encoder described above.

According to another aspect of the invention, there is provided an audio reproducing apparatus comprising a decoder according to the above-described decoder.

According to another aspect of the invention, there is provided an audio data stream for an audio signal comprising: a first stereo signal; and parametric data associated with the down-mixed stereo signal of the M-channel audio signal, where M> 2; wherein the first stereo signal is a binaural signal corresponding to an M-channel audio signal.

According to another aspect of the invention, there is provided a storage medium comprising a signal stored thereon, as described above.

These and other aspects, features, and advantages of the invention will be apparent from and elucidated with reference to the embodiment (s) described below.

Embodiments of the invention will be described by way of example only with reference to the drawings, of which

figure 1 - illustration of binaural synthesis in accordance with the prior art;

figure 2 - illustration of the cascade enable multichannel decoder and binaural synthesis;

3 illustrates a transmission system for transmitting an audio signal in accordance with some embodiments of the invention;

4 illustrates an encoder in accordance with some embodiments of the invention;

5 illustrates an encoder for parametric surround down-mix;

6 illustrates an example of the location of the sound source relative to the user;

7 illustrates a multi-channel decoder in accordance with some variants of the invention;

Fig. 8 illustrates a decoder in accordance with some embodiments of the invention;

FIG. 9 illustrates a decoder in accordance with some embodiments of the invention; FIG.

10 illustrates an audio encoding method in accordance with some embodiments of the invention; and

11 illustrates an audio decoding method in accordance with some embodiments of the invention.

FIG. 3 illustrates a transmission system 300 for transmitting an audio signal in accordance with some embodiments of the invention. The transmission system 300 includes a transmitter 301, which is connected to the receiver 303 via a network 305, which, more precisely, may be the Internet.

In a separate example, the transmitter 301 is a signal recorder, and the receiver is a signal player device 303, but it will be appreciated that in other embodiments, the transmitter and receiver may be used in other applications and for other purposes. For example, transmitter 301 and / or receiver 303 may be part of the transcoding functionality and, for example, may be coupled to other sources or signal destinations.

In a separate example where a signal recording function is supported, the transmitter 301 comprises a digitizer 307 that receives an analog signal that is converted to a digital PCM signal by sampling and analog-to-digital conversion. Digitizer 307 samples a plurality of signals, thereby forming a multi-channel signal.

A transmitter 301 is coupled to an encoder 309 of FIG. 1, which encodes a multi-channel signal in accordance with a coding algorithm. The encoder 309 is connected to a network transmitter 311, which receives the encoded signal and serves as a means of communication with the Internet network 305. The network transmitter may transmit the encoded signal to the receiver 303 via the Internet network 305.

The receiver 303 includes a network receiver 313, which serves as a means of communication with the Internet network 305 and which is configured to receive the encoded signal from the transmitter 301.

The network receiver 313 is connected to the decoder 315. The decoder 315 receives the encoded signal and decodes it in accordance with the decoding algorithm.

In a separate example where the signal reproduction function is supported, the receiver 303 further comprises a signal player 317 that receives the decoded audio signal from the decoder 315 and presents it to the user. More specifically, the signal player 317 may comprise a digital-to-analog converter, amplifiers and speakers, which are required to output a decoded audio signal.

In a separate example, encoder 309 receives a five-channel surround signal and downmixes it into a stereo signal. The stereo signal is then post-processed to form a binaural signal, which, more precisely, is a binaural virtual spatial signal in the form of a 3-dimensional binaural downmix signal. By using the 3-D post-processing cascade acting on the downmix signal after spatial coding, the 3-D processing can be inverted in the decoder 315. As a result, the multi-channel decoder for playing through the speakers will not show any significant degradation due to the modified stereo downmix, along with so that, at the same time, even traditional stereo decoders will synthesize a 3-dimensional compatible signal. Thus, the encoder 309 can generate a signal that enables high-quality multi-channel decoding and, at the same time, allows a pseudo-spatial impression of a traditional stereo output signal, for example, such as from a traditional decoder that feeds a pair of headphones.

4 illustrates an encoder 309 in more detail.

Encoder 309 comprises a multi-channel receiver 401 that receives a multi-channel audio signal. Although the principles described will apply to a multi-channel signal containing any number of channels, more than two, a separate example will focus on a five-channel signal corresponding to a standard surround signal (for clarity and brevity, the low-frequency channel, often used for surround signals, will be ignored. However, the specialist it will be clear in the art that a multi-channel signal can have an additional low-frequency channel, for example, this channel can be combined with neutral channel down-mix processor).

A multi-channel receiver 401 is connected to a downmix processor 403, which is configured to downmix a five-channel audio signal into a first stereo signal. In addition, the downmix processor 403 generates parametric data 405 associated with the first stereo signal and containing control audio signals and information defining a ratio of the first stereo signal to the original channels of the multi-channel signal.

The downmix processor 403, for example, may implement a multi-channel MPEG surround encoder. An example of this is illustrated in FIG. In the example, the multi-channel input signal consists of the channels Lf (left front), Ls (left surround), C (central), Rf (right front) and Rs (right surround). Channels Lf and Ls are fed into the first two-to-one TTO down-mixer 501, which generates a mono down-mix signal for the left (L) channel, as well as parameters setting the ratio of the two input channels Lf and Ls to the output channel L. Channels Rf and Rs fed into the second TTO down-mixer 503, which generates a down-mix mono signal for the right (R) channel, as well as parameters setting the ratio of the two input channels Rf and Rs to the output channel R. The R, L, and C channels are then fed to the TTT down-mixer 505 (three to two) which combines e and signals to generate a stereo down-mix and additional spatial parameters.

The parameters resulting from the TTT downmixer 505 typically consist of a pair of prediction coefficients for each parameter range, or a pair of level drops to describe the energy ratios of the three input signals. The parameters of the down-mixers 501, 503 TTO typically consist of level differences and coherence or cross-correlation between the input signals for each frequency band.

The generated first stereo signal is thus a standard conventional stereo signal comprising a number of downmixed channels. A multi-channel decoder can recreate the original multi-channel signal by upmixing and applying related parametric data. However, a standard stereo decoder will only output a stereo signal, thereby losing spatial information and creating a reduced user experience.

However, in the encoder 309, the down-mixed stereo signal is not encoded or transmitted directly. Rather, the first stereo signal is supplied to the spatial processor 407, which also receives the associated parameter data 405 from the downmix processor 403. Spatial processor 407 is further coupled to HRTF processor 409.

The HRTF processor 409 generates the parameter data of a human sound perception modeling function (HRTF) used by the spatial processor 407 to generate a 3D binaural signal. More precisely, HRTF describes a transfer function from a predetermined location of a sound source to the eardrum by means of an impulse response. HRTF processor 409, more precisely, generates HRTF parameter data corresponding to a value of a desired HRTF function in a frequency subband. An HRTF processor 409, for example, can calculate an HRTF for the location of the sound source of one of the channels of a multi-channel signal. This transfer function may be converted to a suitable region of the frequency subbands (such as the QMF or FFT subband region) and the corresponding HRTF parameter value in each subband may be determined.

It will be appreciated that although the description focuses on the application of human sound perception modeling functions, the described approach and principles apply equally well to other transfer functions of (spatial) binaural perception, such as the binaural impulse response characteristic of a room (BRIR). Another example of the binaural perception transfer function is a simple amplitude panning rule that describes the relative magnitude of the signal level from one input channel to another of binaural stereo output channels.

In some embodiments, HRTF parameters can be calculated dynamically, while in other embodiments, they can be predefined and stored in a suitable data warehouse. For example, HRTF parameters may be stored in a database as a function of heading angle, elevation angle, distance and frequency band. Appropriate HRTF parameters for a given frequency sub-band, in this case, can simply be retrieved by selecting values for the desired spatial position of the sound source.

The spatial processor 407 modifies the first stereo signal to generate a second stereo signal, in response to the associated parametric data and HRTF spatial parameter data. In contrast to the first stereo signal, the second stereo signal is a binaural virtual spatial signal, and more specifically, a 3-dimensional binaural signal, which, when presented through a traditional stereo system (for example, a pair of headphones), can give an expanded spatial impression that mimics the presence of more than two sources sound in different locations of sound sources.

The second stereo signal is supplied to an encoding processor 411, which is coupled to the spatial processor 407 and which encodes the second signal into a data stream suitable for transmission (for example, using appropriate quantization levels, etc.). An encoding processor 411 is coupled to an output processor 413 that generates an output stream by combining at least encoded second stereo signal data and associated parameter data 405 generated by the downmix processor 403.

Typically, HRTF synthesis requires vibrational signals for all individual sound sources (for example, speaker signals in the context of a surround sound signal). However, in encoder 309, HRTF pairs are parameterized for the frequency subbands, thereby, for example, allowing the virtual 5.1 speaker setup to be generated by post-processing with low complexity down-mix signal of the multi-channel input signal, using spatial parameters that were extracted during the encoding sequence (and down mix).

A spatial processor, more precisely, can operate in a subband domain, such as a QMF or FFT subband domain. Rather than decoding the down-mixed first stereo signal to generate an original multi-channel signal followed by HRTF synthesis using HRTF filtering, the spatial processor 407 generates parameter values for each subband corresponding to the combined result of decoding the down-mixed first stereo signal into a multi-channel signal, followed by re-encoding multi-channel signal as 3-dimensional binaura signal.

More precisely, the inventors realized that a 3-dimensional binaural signal can be generated by applying multiplication by a 2 × 2 matrix to the sub-band signal values of the first signal. The resulting signal values of the second signal closely correspond to the signal values that can be generated by cascading HRTF decoding and synthesis. Thus, the combined signal processing of multi-channel coding and HRTF synthesis can be combined into four parameter values (matrix coefficient), which can simply be applied to the values of the subband signals to form the required subband values of the second signal. Since the matrix parameter values reflect the combined process of decoding the multi-channel signal and HRTF synthesis, parameter values are determined in response to the associated parametric data from the downmix processor 403, as well as HRTF parameters.

In encoder 309, HRTF functions are parameterized for individual frequency bands. The purpose of HRTF parameterization is to capture the most important control signals to determine the location of the sound source from each HRTF pair. These options may include:

- (Medium) level per frequency sub-band for impulse response of the left ear;

- (Medium) level per frequency sub-band for impulse response of the right ear;

- (Average) arrival time or phase difference between the impulse characteristics of the left ear and the right ear;

- (Average) absolute phase or time (or group delay) per frequency sub-band for both left and right impulse responses (in this case, the time or phase difference, in most cases, becomes absolute);

- Inter-channel cross-correlation or coherence per frequency sub-band between respective impulse responses.

Level parameters per frequency subband can facilitate the synthesis of the elevation angle (thanks to special peaks and dips in the spectrum), as well as level differences for the heading angle (determined by the ratio of level parameters for each subband).

Absolute phase values or phase difference values can capture arrival time differences between both ears, which are also important tell-tales for the heading angle of the sound source. A coherence value could be added to simulate the fine structure drops between both ears, which cannot be introduced into level and / or phase differences, averaged over the range (parameters).

Subsequently, a separate example of processing by the spatial processor 407 is described. In the example, the location of the sound source is determined relative to the listener by the azimuthal angle α and the distance D , as shown in FIG. 6. The sound source located to the left of the listener corresponds to positive azimuthal angles. The transfer function from the location of the sound source to the left ear is designated H _L , the transfer function from the location of the sound source to the left ear is H _R.

The transfer functions H _L and H _{R are} dependent on the azimuthal angle α, distance D and elevation angle ε (not shown in FIG. 6). In a parametric representation, the transfer functions can be described as a set of three parameters per frequency sub-band b _h HRTF. This set of parameters includes the average level per frequency band for the left transfer function, P _l (α, ε, D , b _h ), the average level per frequency band for the right transfer function, P _r (α, ε, D , b _h ), the average phase difference per frequency band, ϕ (α, ε, D , b _h ). A possible extension of this set should include a measure of the coherence of the left and right transfer functions to the HRTF, ρ (α, ε, D , b _h ) frequency band. These parameters can be stored in the database as a function of heading angle, elevation angle, distance and frequency band and / or can be calculated using some analytical function. For example, the parameters P _l and P _r could be stored as a function of the heading angle or elevation angle, along with the fact that the influence of the distance is achieved by dividing these values by the distance itself (provided that the ratio 1 / D between the signal level and the distance). Subsequently, the notation P _l ( Lf ) denotes the spatial parameter P _l corresponding to the location of the sound source of the channel Lf .

It should be noted that the number of frequency subbands for parameterization (b _h) HRTF and the bandwidth of each subband is not necessarily equal to the frequency resolution of the comb (k) filters (QMF), used by the spatial processor 407, or the resolution of the spatial parameter processor 403 downmix and associated ranges ( b _p ) of parameters . For example, a QMF hybrid filter bank may have 71 channels, HRTF may be parameterized in 28 frequency bands, and spatial coding may be performed using 10 parameter ranges. In such cases, the mapping from spatial parameters and HRTF parameters to a hybrid QMF index can be applied, for example, using a look-up table, or an interpolation function, or averaging. The following parameter indices will be used in the description:

Index Description b _h Parameter Range Index for HRTF b _p Parameter Range Index for Multichannel Downmix k QMF Hybrid Range Index

In a separate example, the spatial processor 407 divides the first stereo signal into suitable frequency subbands by QMF filtering. For each subband, the values of L _B , R _{B of the} subband are determined as:

,

,

where L ₀ , R ₀ are the corresponding values of the subband of the first stereo signal, and the values of h _{j, k of the} matrix are the parameters that are determined from the HRTF parameters and associated with the down-mix of parametric data.

The matrix coefficients are aimed at reproducing the down-mix properties, as if all the individual channels were processed using HRTFs corresponding to the desired location of the sound source, and they include the combined result of decoding the multi-channel signal and performing HRTF synthesis on it.

More precisely, with reference to figure 5 and its description, the values of the matrix can be defined as:

h ₁₁ = m ₁₁ H _L ( L ) + m ₂₁ H _L ( R ) + m ₃₁ H _L ( C )

h ₁₂ = m ₁₂ H _L ( L ) + m ₂₂ H _L ( R ) + m ₃₂ H _L ( C )

h ₂₁ = m ₁₁ H _R ( L ) + m ₂₁ H _R ( R ) + m ₃₁ H _R ( C )

h ₂₂ = m ₁₂ H _R ( L ) + m ₂₂ H _R ( R ) + m ₃₂ H _R ( C ),

where m _{k, l} are parameters determined in response to the parametric data generated by the TT5 step-down mixer.

More specifically, the signals L, R and C are formed from the stereo signal L ₀ , R ₀ down-mix according to:

,

,

where m _{k, l are} dependent on two prediction coefficients c ₁ and c ₂ , which are part of the transmitted spatial parameters:

.

.

The values of H _J (X) are determined in response to the HRTF parameter data for channel X regarding the stereo output channel J of the second stereo signal, as well as the appropriate downmix parameters.

More specifically, the parameters H _J (X) relate to the left (L) and right (R) downmix signals generated by the two TTO downmixers 501, 503 and can be determined in response to the HRTF parameter data for the two downmixed channels. More precisely, a weighted combination of HRTF parameters can be used for two separate left (Lf and Ls) or right (Rf and Rs) channels. Individual parameters can be weighted by the relative energy of individual signals. As a separate example, the following values can be determined for the left signal (L):

,

,

,

,

where the weights w _{x are} given according to:

,

,

,

,

and CLD _l is the 'channel level difference' between the left front (Lf) and left surround (Ls) defined in decibels (which is part of the spatial parameter bit stream):

,

,

with σ ² _lf , power in the sub-band of channel parameters Lf, and σ ² _ls , power in the corresponding sub-band of channel Ls.

Similarly, the following values can be determined for the right signal (R):

,

,

,

,

,

,

and for the central (C) signal:

.

.

Thus, using the described approach, spatial processing with low complexity can provide the binaural virtual spatial signal with the opportunity to be formed on the basis of the down-mixed multi-channel signal.

As mentioned, an advantage of the described approach is that the frequency sub-band of the associated down-mix parameters, spatial processing by the spatial processor 407, and HRTF parameters do not need to be the same. For example, a mapping may be performed between the parameters of one subband into subbands of spatial processing. For example, if the spatial processing subband covers a frequency range corresponding to two HRTF parameter subbands, the spatial processor 407 can simply apply (individual) processing to the HRTF parameter subbands using the same spatial parameter for all HRTF parameter subbands that correspond to that spatial parameter.

In some embodiments, an encoder 309 may be configured to include location data of a sound source that enables a decoder to identify desired location data of one or more sound sources in an output stream. This allows the decoder to determine the HRTF parameters applied by the encoder 309, thereby enabling it to reverse the operation of the spatial processor 407. Additionally or alternatively, the encoder may be configured to include at least some of these HRTF parameters in the output stream.

Thus, optionally, HRTF parameter data and / or speaker locations can be included in the output stream. This, for example, can provide the ability to dynamically update the speaker location data as a function of time (in the case of transmitting the speaker location) or using individualized HRTF data (in the case of transmitting HRTF parameters).

In the case where the HRTF parameters are transmitted as part of the bit stream, at least the parameters P _l , P _r and ϕ can be transmitted for each frequency band and for each location of the sound source. The amplitude parameters P _l , P _r can be quantized using a quantizer or can be quantized in the logarithmic region. Phase angles ϕ can be quantized linearly. Quantizer indices can then be included in the bitstream.

Moreover, the phase angles ϕ can be assumed to be zero for frequencies, typically in the vicinity of 2.5 kHz, since information about the (inside) phase is inappropriate in terms of perception for high frequencies.

After quantization, various lossless compression schemes can be applied to the quantizer parameters of the HRTF parameters. For example, entropy coding may be used, possibly in combination with differential coding over frequency bands. Alternatively, HRTF parameters may be presented as a difference with respect to a set of general or average HRTF parameters. This is supported especially for amplitude parameters. Otherwise, the phase parameters can be approximated fairly accurately by simply coding the elevation angle and course angle. By calculating the difference in the arrival times [typically, the difference in the arrival times is practically frequency-dependent; it mainly depends on the heading angle and elevation angle], if the difference of the paths to both ears is given, the corresponding phase parameters can be displayed. In addition to the measurements, the differences can be differential encoded with respect to the predicted values based on the heading angle and elevation angle.

In addition, lossless compression schemes may be used, such as the principal componentization, followed by the transfer of several of the most important PCA weights.

7 illustrates an example of a multi-channel decoder in accordance with some embodiments of the invention. The decoder, more precisely, may be the decoder 315 of FIG. 3.

Decoder 315 comprises an input receiver 701 that receives an output stream from an encoder 309. An input receiver 701 demultiplexes a received data stream and provides relevant data to the appropriate functional elements.

The input receiver 701 is connected to a decoding processor 703, into which encoded data of the second stereo signal is supplied. A decoding processor 703 decodes this data to generate a binaural virtual spatial signal generated by the spatial processor 407.

The decoding processor 703 is coupled to the reversing processor 705, which is configured to reverse the operation performed by the spatial processor 407. Thus, the reversing processor 705 generates the downmixed stereo signal generated by the downmix processor 403.

More specifically, the reversal processor 705 generates a downmix stereo signal by applying matrix multiplication to the subband values of the received binaural virtual spatial signal. Matrix multiplication occurs through a matrix corresponding to the inverse of the matrix used by the spatial processor 407, thereby reversing this operation:

.

.

This matrix multiplication can also be described as:

.

.

The coefficients q _{k, l of the} matrix are determined from the parametric data associated with the down-mix signal (and received in the data stream from the encoder 309), as well as HRTF parameter data. More specifically, the approach described with reference to encoder 309 can also be used by HRTF processor 409 to generate matrix coefficients h _xy . The coefficients q _{xy of the} matrix, in this case, can be found by standard matrix inversion.

A reverse processor 705 is coupled to a parameter processor 707, which determines the HRTF parameter data to be used. HRTF parameters, in some embodiments, may be included in the received data stream and may simply be retrieved from it. In other embodiments, different HRTF parameters, for example, can be stored in a database for different locations of sound sources, and parameter processor 707 can determine HRTF parameters by retrieving values corresponding to the desired location of the signal source. In some embodiments, the desired location (s) of the signal source may be included in the data stream from encoder 309. Parameter processor 707 may extract this information and use it to determine HRTF parameters. For example, it can retrieve HRTF parameters stored to indicate the location (s) of a sound source.

In some embodiments, the stereo signal generated by the reversal processor may be directly output. However, in other embodiments, it can be supplied to a multi-channel decoder 709, which can generate an M-channel signal from a stereo down-mix signal and received parametric data.

In an example, 3-D binaural synthesis reversal is performed in a subband region, such as in the QMF or Fourier frequency subbands. Thus, the decoding processor 703 may comprise a comb of QMF filters or a fast Fourier transform (FFT) to form subband sound samples supplied to the reversing processor 705. Similarly, the reverse processor 705 or multi-channel decoder 709 may comprise a comb of inverse FFT or QMF filters to convert the signals back to the time domain.

The generation of a 3-dimensional binaural signal on the encoder side provides that the impression of spatial listening is provided for the headphone user by a traditional stereo encoder. Thus, the described approach has the advantage that the inherited stereo devices can reproduce a 3-dimensional binaural signal. Essentially, in order to reproduce 3-dimensional binaural signals, no additional post-processing is necessary, resulting in a solution of low complexity.

However, this approach typically uses generalized HRTF, which, in some cases, can lead to suboptimal spatial formation compared to the generation of a 3D binaural signal in a decoder using specialized HRTF data optimized for a particular user.

More precisely, limited perception of distance and possible errors in determining the location of a sound source can sometimes arise from the use of non-individualized HRTFs (such as impulse responses measured for a head simulator or other person). In principle, HRTFs differ from person to person due to differences in the anatomical geometry of the human body. Optimal results in terms of correct positioning of the sound source, therefore, can be better achieved with individualized HRTF data.

In some embodiments, the decoder 315 further comprises functionality, first for reversing the spatial processing of the encoder 309, accompanied by generation of a 3D binaural signal using local HRTF data, more specifically, using individual HRTF data optimized for a particular user. Thus, in this embodiment, the decoder 315 generates a pair of binaural output channels by modifying the down-mixed stereo signal using the associated parametric data and HRTF parameter data, which are other than the HRTF data used in encoder 309. From this approach, a combination is provided 3D synthesis on the encoder side, accessing on the decoder side, followed by another stage of 3D synthesis on the decoder side.

The advantage of this approach is that the inherited stereo devices will have 3-dimensional binaural signals as output signals providing basic 3-dimensional quality, while advanced decoders have the ability to use personalized HRTFs, enabling improved 3-dimensional quality. Thus, legacy 3-D synthesis, as well as high-quality specialized 3-D synthesis, are used in the same audio system.

A simple example of such a system is illustrated in FIG. 8, which shows how an additional spatial processor 801 can be added to the decoder of FIG. 7 to provide a custom 3-dimensional binaural output signal. In some embodiments, the spatial processor 801 may provide only a simple direct 3-dimensional binaural synthesis using individual HRTF functions for each of the audio channels. Thus, the decoder can recreate the original multi-channel signal and convert it into a 3-dimensional binaural signal using HRTF filtering configured to customer requirements.

In other embodiments, the implementation of the inverse synthesis of the encoder and synthesis of the decoder can be combined to provide operations of low complexity. More specifically, the individualized HRTFs used to synthesize the decoder can be parameterized and combined with (inverted) parameters used by the 3D encoder synthesis.

More precisely, as described previously, the synthesis of the encoder involves multiplying stereo samples of the sound of the subband down-mixed signals by a 2 × 2 matrix:

,

,

where L ₀ , R ₀ are the corresponding subband values of the stereo down-mixed stereo signal, and the matrix values h _{j, k} are the parameters that are determined from the HRTF parameters and the parametric data associated with the down-mixing, as described previously.

The reversal performed by the reverse processor 705 may then be specified by:

,

,

where L _B , R _B are the corresponding sub-band values of the down-mixed stereo decoder.

In order to guarantee the proper sequence of accesses on the decoder side, the HRTF parameters used in the encoder to generate the 3D binaural signal and the HRTF parameters used for reversing the 3D binaural processing are identical or sufficiently similar. Since a single bitstream typically serves multiple decoders, the personification of 3D binaural downmix is difficult to obtain through encoder synthesis.

However, since the 3-D binaural synthesis process is reversible, the reverse processor 705 regenerates the down-mixed stereo signal, which is then used to generate the 3-dimensional binaural signal based on individualized HRTFs.

More precisely, by analogy with the operation in encoder 309, a 3-dimensional binaural synthesis in decoder 315 can be formed by a simple 2 × 2 matrix operation associated with subbands on a downmix signal L ₀ , R ₀ to generate a 3-dimensional binaural signal L _{B '} , R _{B '} :

,

,

where the parameters p _{x, y are} determined based on individualized HRTFs in the same way that h _{x, y} are generated by the encoder 309 based on the overall HRTF. More specifically, in encoder 309, the parameters h _{x, y are} determined from multichannel parametric data and common HRTFs. Since multichannel parametric data is transmitted to decoder 315, the same approach can be used as such to calculate p _{x, y} based on individual HRTFs.

Combining this with the operation of the reverse processor 705

.

.

In this equation _, matrix elements h _{x, y} are obtained using the common non-individualized HRTF set used in the encoder, while p _{x, y are} obtained using another and preferably personalized HRTF set. From here, the 3-dimensional binaural input signal L _B , R _B generated using non-individualized HRTF data is converted to an alternative 3-dimensional binaural output signal L _{B '} , R _B' using other personalized HRTF data.

Moreover, as illustrated, the combined approach of inverting encoder synthesis and decoder synthesis can be achieved with a simple 2 × 2 matrix operation. Hence, the computational complexity of the combined sequence of operations is, in fact, the same as for simple 3-dimensional binaural handling.

9 illustrates an example of a decoder 315 operating in accordance with the above principles. More specifically, the stereo samples of the subband sound of the 3-D binaural stereo down-mix stereo signal from the encoder 309 are supplied to a reversing processor 705, which generates the original stereo samples of the down-mix sound through a 2 × 2 matrix operation.

.

.

The resulting subband sound samples are supplied to spatial synthesis unit 901, which generates an individualized 3-dimensional binaural signal by multiplying these sound samples by a 2 × 2 matrix.

.

.

Matrix coefficients are generated by a parameter conversion unit (903), which generates parameters based on an individualized HRTF and multi-channel extension data received from encoder 309.

Samples of the synthesis subband sound L _{B ′} , R _{B ′} are fed to a transform 905 from the subband region to the time domain, which generates 3-dimensional binaural signals in the time domain that can be provided to the user.

Although FIG. 9 illustrates the steps of 3-D inversion based on non-individualized HRTFs and 3-D synthesis based on individualized HRTFs as sequential operations through different function blocks, it will be appreciated that in many embodiments these operations are applied simultaneously using a single matrix. The 2 × 2 matrix is more accurately calculated

,

,

and output sound samples are calculated as

.

.

It will be appreciated that the described system provides a number of advantages, including:

- no or insignificant deterioration in the quality (perception) of multichannel reconstruction, since spatial stereo processing can be reversed in multichannel decoders;

- (3-dimensional) binaural stereo impression can be provided even by traditional stereo decoders;

- reduced complexity compared to existing methods of spatial positioning. Difficulty is reduced in a number of ways:

- efficient storage of HRTF parameters. Instead of storing the impulse characteristics of HRTF, only a limited number of parameters are used to characterize HRTF;

- effective 3-dimensional processing. Since HRTFs are characterized as parameters at a limited frequency resolution, and HRTF parameters are applied in the region of (strongly discretized with decreasing frequency) parameters, the spatial synthesis stage is more efficient than traditional synthesis methods based on full HRTF convolution;

- the required processing can be performed, for example, in the QMF domain, resulting in less computational and memory related load than FFT based methods;

- effective reuse of existing surround sound building blocks (such as standard MPEG surround sound encoding / decoding functionality), providing the possibility of minimal implementation complexity;

- the possibility of personification by modifying the data (parameterized) HRTF transmitted by the encoder;

- locations of sound sources can change on the fly according to the transmitted location information.

10 illustrates an audio coding method in accordance with some embodiments of the invention.

The method begins at step 1001, which receives the M-channel audio signal (M> 2).

Step 1001 is followed by step 1003, in which the M-channel audio signal is down-mixed into the first stereo signal and the associated parametric data.

Step 1003 is followed by step 1005, in which the first stereo signal is modified to form a second stereo signal, in response to the associated parametric data and spatial parameter data of the human sound perception modeling function (HRTF). The second stereo signal is a binaural virtual spatial signal.

Step 1005 is followed by step 1007, in which a second stereo signal is encoded to generate encoded data.

Step 1007 is followed by step 1009, which generates an output data stream containing encoded data and associated parametric data.

11 illustrates an audio decoding method in accordance with some embodiments of the invention.

The method begins at step 1101, in which the decoder receives input data containing the first stereo signal and parametric data associated with the down-mixed stereo signal of the M-channel audio signal, where M> 2. The first stereo signal is a binaural virtual spatial signal.

Step 1101 is followed by step 1103 in which the first stereo signal is modified to form a down-mixed stereo signal in response to parametric data and spatial parameter data of a human sound perception modeling function (HRTF) associated with the first stereo signal.

Step 1103 is followed by optional step 1105, in which an M-channel audio signal is generated in response to the down-mixed stereo signal and parametric data.

It will be appreciated that the foregoing description, for clarity, described embodiments of the invention with reference to different function blocks and processors. However, it will be apparent that any suitable distribution of functionality between different functional units or processors can be used without detracting from the invention. For example, the functionality illustrated by executing separate processors or controllers may be performed by the same processor or controllers. Hence, references to certain functional blocks should be considered only as references to a suitable means for providing the described functionality, rather than indicating a strict logical or physical structure or organization.

The invention may be implemented in any suitable form, including hardware, software, hardware-implemented software, or a combination thereof. The invention, optionally, may be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. The elements and components of an embodiment of the invention may be implemented physically, functionally, and logically in any suitable manner. Indeed, the functionality may be implemented in a single block, multiple blocks, or as part of other functional blocks. Essentially, the invention may be implemented in a single unit, or may be physically or functionally distributed between different units and processors.

Although the present invention has been described in connection with certain embodiments, it is not intended to be limited to the individual forms set forth in the materials of this application. Rather, the scope of the present invention is limited only by the attached claims. Additionally, although a feature may be described as described in connection with specific embodiments, one skilled in the art will recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term “comprises” does not exclude the presence of other elements or steps.

Moreover, although they are listed separately, many means, elements or steps of the method can be implemented, for example, by a single unit or processor. Additionally, although individual features may be included in different claims, they can be combined advantageously as soon as possible, and inclusion in different claims does not imply that a combination of features is not feasible and / or useful. In addition, the inclusion of a feature in one category of the claims does not imply a restriction to this category, but rather indicates that the feature is equally applicable to the categories of the other claims. Moreover, the sequence of features in the claims does not imply any particular order in which the features should be processed, and, in particular, the sequence of individual steps in a claim of a method does not imply that steps should be performed in that order. Rather, the steps can be performed in any suitable order. In addition, singular references do not exclude plurality. Singular expressions, 'first', 'second', etc. do not eliminate pluralities. The reference symbols in the claims are provided only as clarifying examples, which should not be construed as limiting the scope of the claims in any way.

Claims (35)

1. An audio encoder comprising:
- means (401) for receiving an M-channel audio signal, where M>2;
- down-mix means (403) for down-mixing the M-channel audio signal into a first stereo signal and associated parametric data;
- forming means (407) for modifying the first stereo signal in order to generate a second stereo signal in response to associated parametric data and spatial parameter data indicating a binaural perception transfer function, the second stereo signal being a binaural signal;
- means (411) for encoding the second stereo signal to generate encoded data and
- output means (413) for generating an output data stream containing encoded data and associated parametric data. 2. The encoder according to claim 1, wherein the generating means (407) is configured to generate a second stereo signal by calculating subband data values for the second stereo signal in response to associated parametric data, spatial parameter data, and subband data values for the first stereo signal. 3. The encoder according to claim 2, wherein the generating means (407) is configured to generate subband values for the first subband of the second stereo signal in response to multiplying the corresponding stereo values of the subband for the first stereo signal by the matrix of the first subband; wherein the generating means (407) further comprises parameter means for determining matrix data values of the first subband in response to the associated parametric data and spatial parameter data for the first subband. 4. The encoder according to claim 3, wherein the generating means (407) further comprises means for converting the data value of at least one of the first stereo signal, related parametric data and spatial parameter data associated with a subband containing a frequency interval other than the interval of the first subband into the corresponding data value for the first subband. 5. The encoder according to claim 3, in which the means (407) of the formation is configured to determine the stereo values L _B , R _{B of the} subband for the first subband of the second stereo signal essentially as:

,
wherein L ₀ , R ₀ are the corresponding values of the subband of the first stereo signal, and the parameter tool is configured to determine the data values of the multiplication matrix essentially as:
h ₁₁ = m ₁₁ H _L (L) + m ₂₁ H _L (R) + m ₃₁ H _L (C);
h ₁₂ = m ₁₂ H _L (L) + m ₂₂ H _L (R) + m ₃₂ H _L (C);
h ₂₁ = m ₁₁ H _R (L) + m ₂₁ H _R (R) + m ₃₁ H _R (C);
h ₂₂ = m ₁₂ H _R (L) + m ₂₂ H _R (R) + m ₃₂ H _R (C),
where m _{k, l} are parameters determined in response to the associated parametric data for downmixing by means of downmixing channels L, R and C into the first stereo signal; a H _j (X) is determined in response to the spatial parameter data for channel X with respect to the output channel J of the second stereo signal. 6. The encoder according to claim 5, in which at least one of the channels L and R corresponds to the down-mix of at least two down-mix channels, and the parameter means is configured to determine H _j (X) in response to a weighted combination of spatial parameter data for at least two downmixed channels. 7. The encoder according to claim 6, in which the parameter means is configured to determine the weighting of the spatial parameters for at least two downmix channels in response to a relative measure of energy for the at least two downmix channels. 8. The encoder according to claim 1, in which the data of the spatial parameters include at least one parameter selected from the group consisting of:
- mid-range parameter for the sub-range;
- parameter of the average time of receipt;
- phases of at least one stereo channel;
- time reference parameter;
- group delay parameter;
- phases between stereo channels and
- parameter cross-correlation of channels. 9. The encoder according to claim 1, wherein the output means (413) is configured to include location data of the sound source in the output stream. 10. The encoder according to claim 1, wherein the output means (413) is configured to include at least some spatial parameter data in the output stream. 11. The encoder according to claim 1, further comprising means (409) for determining the spatial parameter data in response to the desired locations of the audio signal. 12. An audio decoder comprising:
- means (701, 703) for receiving input data containing the first stereo signal and parametric data associated with the down-mixed stereo signal of the M-channel audio signal, where M> 2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal;
- forming means (705) for modifying the first stereo signal in order to generate the down-mixed stereo signal in response to the parametric data and the first spatial parameter data indicating a transfer function of binaural perception, wherein the first spatial parameter data is associated with the first stereo signal. 13. The decoder according to claim 12, further comprising means (709) for generating the M-channel audio signal in response to the down-mixed stereo signal and parametric data. 14. The decoder of claim 12, wherein the generating means (705) is configured to generate the down-mixed stereo signal by calculating the sub-band data values for the down-mixed stereo signal in response to the associated parametric data, the first spatial parameter data and the sub-band data values for the first stereo signal. 15. The decoder of claim 14, wherein the generating means (705) is configured to generate subband values for the first subband of the stereo down-mixed stereo signal in response to multiplying the corresponding stereo subband values for the first stereo signal by the first subband matrix; moreover, the means (705) formation further comprises a means of parameters for determining the data values of the matrix of the first subband in response to parametric data and parameter data of the binaural perception transfer function for the first subband. 16. The decoder of claim 12, wherein the input data comprises at least some of the first spatial parameter data. 17. The decoder according to claim 12, in which the input data contains the location data of the sound source and the decoder comprises means (707) for determining the first spatial parameter data in response to the location data of the sound source. 18. The decoder of claim 12, further comprising:
- a spatial decoder unit (709, 801) for generating a pair of binaural output channels by modifying the first stereo signal in response to the associated parametric data and second spatial parameter data indicating a second transfer function of binaural perception, the second spatial parameter data being different from the first spatial parameter data. 19. The decoder of claim 18, wherein the spatial decoder unit (709, 801) comprises:
- block (903) converting parameters for converting parametric data into binaural synthesis parameters using the second spatial parameter data and
- spatial synthesis unit (901) for synthesizing a pair of binaural channels using binaural synthesis parameters and a first stereo signal. 20. The decoder according to claim 19, in which the binaural synthesis parameters contain matrix coefficients for a 2 by 2 matrix that determines the ratio of stereo samples of the stereo signal down-mixed to stereo samples of a pair of binaural output channels. 21. The decoder according to claim 19, in which the binaural synthesis parameters comprise matrix coefficients for a 2 by 2 matrix defining the ratio of stereo samples of the subband of the first stereo signal to stereo samples of a pair of binaural output channels. 22. A method of encoding an audio signal, wherein the method is that:
- receive (1001) M-channel audio signal, where M>2;
- carry out down-mixing (1003) of the M-channel audio signal to the first stereo signal and the associated parametric data;
- modify (1005) the first stereo signal to generate a second stereo signal in response to the associated parametric data and spatial parameter data indicating a transfer function of binaural perception, the second stereo signal being a binaural signal;
- encode (1007) the second stereo signal to generate encoded data and
- form (1009) the output data stream containing encoded data and associated parametric data. 23. A method for decoding an audio signal, wherein the method is that:
- receive (1101) input data containing the first stereo signal and parametric data related to the down-mixed stereo signal of the M-channel audio signal, where M> 2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal; and
- modify (1103) the first stereo signal to generate subjected to down-mixing stereo signal in response to parametric data and spatial parameter data indicating the transfer function of binaural perception, and the spatial parameter data is associated with the first stereo signal. 24. A receiver for receiving an audio signal comprising:
- means (701, 703) for receiving input data containing the first stereo signal and parametric data associated with the down-mixed stereo signal of the M-channel audio signal, where M> 2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal; and
- forming means (705) for modifying the first stereo signal to form a down-mixed stereo signal in response to parametric data and spatial parameter data indicating a binaural perception transfer function, wherein the spatial parameter data is associated with the first stereo signal. 25. A transmitter (1101) for transmitting an output data stream, the transmitter comprising:
- means (401) for receiving an M-channel audio signal, where M>2;
- down-mix means (403) for down-mixing the M-channel audio signal into a first stereo signal and associated parametric data;
- forming means (407) for modifying the first stereo signal in order to generate a second stereo signal in response to associated parametric data and spatial parameter data indicating a binaural perception transfer function, the second stereo signal being a binaural signal;
- means (411) for encoding the second stereo signal in order to generate encoded data;
- output means (413) for generating an output data stream containing encoded data and associated parametric data; and
- means (311) for transmitting the output data stream. 26. A transmission system for transmitting an audio signal, wherein the transmission system comprises:
- a transmitter containing:
- means (401) for receiving an M-channel audio signal, where M> 2,
- down-mix means (403) for down-mixing the M-channel audio signal into a first stereo signal and associated parametric data,
- forming means (407) for modifying the first stereo signal in order to generate a second stereo signal in response to associated parametric data and spatial parameter data indicating a binaural perception transfer function, the second stereo signal being a binaural signal,
- means (411) for encoding the second stereo signal in order to generate encoded data,
- output means (413) for generating an output audio data stream comprising encoded data and associated parametric data, and
- means (311) for transmitting the output audio stream; and
- a receiver containing:
- means (701, 703) for receiving the output stream of audio data and
- means (705) for modifying the second stereo signal in order to generate a first stereo signal in response to parametric data and spatial parameter data. 27. The method of receiving an audio signal, the method is that:
- receive (1101) input data containing the first stereo signal and parametric data related to the down-mixed stereo signal of the M-channel audio signal, where M> 2, the first stereo signal being a binaural signal corresponding to the M-channel audio signal; and
- modify (1103) the first stereo signal to generate subjected to down-mixing stereo signal in response to parametric data and spatial parameter data indicating the transfer function of binaural perception, and the spatial parameter data is associated with the first stereo signal. 28. The method of transmitting the output stream of audio data, the method is that:
- receive (1001) M-channel audio signal, where M>2;
- carry out down-mixing (1003) of the M-channel audio signal to the first stereo signal and the associated parametric data;
- modify (1005) the first stereo signal to generate a second stereo signal in response to the associated parametric data and spatial parameter data indicating a transfer function of binaural perception, the second stereo signal being a binaural signal;
- encode (1007) the second stereo signal to generate encoded data and
- form (1009) the output audio data stream containing encoded data and associated parametric data; and
- transmit the output stream of audio data. 29. A method for transmitting and receiving an audio signal, wherein the method is that:
- receive (1001) M-channel audio signal, where M>2;
- carry out down-mixing (1003) of the M-channel audio signal to the first stereo signal and the associated parametric data;
- modify (1005) the first stereo signal to generate a second stereo signal in response to the associated parametric data and spatial parameter data indicating a transfer function of binaural perception, the second stereo signal being a binaural signal;
- encode (1007) the second stereo signal to generate encoded data and
- form (1009) the output audio data stream containing encoded data and associated parametric data;
- transmit the output stream of audio data;
- receive (1101) the output stream of audio data and
- modify (1103) the second stereo signal to generate the first stereo signal in response to parametric data and spatial parameter data. 30. A computer-readable storage medium on which computer-executable instructions are stored which, when executed by a processor, cause the computer to implement the method of claim 22. 31. A computer-readable storage medium on which computer-executable instructions are stored which, when executed by a processor, cause the computer to implement the method of claim 23. 32. A computer-readable storage medium on which computer-executable instructions are stored which, when executed by a processor, cause the computer to implement the method of claim 27. 33. A computer-readable storage medium on which computer-executable instructions are stored that, when executed by a processor, cause the computer to implement the method of claim 28. 34. An audio recording device comprising an encoder (309) according to claim 1. 35. The audio playback device containing a decoder (315) according to item 12.

Images