EP3025514B1 - Sound spatialization with room effect - Google Patents

Description

The invention relates to the processing of sound data, and more particularly to the spatialization (called "3D rendering") of audio signals.

Such an operation is for example performed when decoding a coded 3D audio signal, represented on a number of channels, to a number of different channels, for example two, to allow the reproduction of the 3D audio effects on a headset. listening.

The invention also relates to the transmission and reproduction of multichannel audio signals and their conversion to a rendering device, transducer, imposed by the equipment of a user. This is for example the case for the reproduction of a 5.1 sound stage by an audio headset, or by a pair of loudspeakers.

The invention also relates to the rendering, in the context of a game or video recording, for example, of one or more sound samples stored in files, with a view to their spatialization.

In the case of a static monophonic source, the binauralization is based on the monophonic signal filtering by the transfer function between the desired position of the source and each of the two ears. The binaural signal (two channels) obtained can then feed a headphone and provide the listener with a feeling of the source at the simulated position. Thus, the term "binaural" refers to the reproduction of a sound signal with spatialization effects.

Each of the transfer functions simulating different positions can be measured in a deaf chamber, thus resulting in a set of HRTFs (for "Head Related Transfer Functions") in which no room effect is present.

These transfer functions can also be measured in a "classical" room, resulting in a set of BRIRs ("Binaural Room Impulse Response") in which the room effect, or reverb, is present. The set of BRIRs thus correspond to a set of transfer functions between a given position and the ears of a listener (real or artificial head) placed in a room.

The usual BRIR measurement technique consists of successively sending in each of the actual loudspeakers, positioned around a head (real or artificial) equipped with microphones in the ears, a test signal (for example a sweep signal, a sequence pseudo-random binary or white noise). This test signal makes it possible, during a non-real-time processing, to reconstitute (generally by deconvolution) the impulse response between the position of the loudspeaker and each of the two ears.

The difference between a set of HRTF and BRIR lies mainly in the length of the impulse response, of the order of one millisecond for the HRTF, to the order of one second for the BRIRs.

Since the filtering is based on the convolution between the monophonic signal and the impulse response, the complexity of binauralizing with BRIRs (containing a room effect) is much higher than with HRTFs.

It is possible by this technique to simulate the headphones or a limited number of speakers listening to a multichannel content (L channels) generated by L speakers in a room. Indeed, it suffices to consider each of the L loudspeakers as a virtual source ideally positioned relative to the listener, to measure in the room to simulate the transfer functions (for the left and right ears) of each of these L speakers, then apply to each of the L audio signals (supposed to supply the L actual speakers) BRIR filters corresponding to the speakers. The signals feeding each of the ears are summed to provide a binaural signal feeding an audio headset.

$$O^d={\displaystyle \sum _{l=1}^{L}I\left(l\right)}\ast {\mathit{BRIR}}^d\left(l\right)$$

We denote I (1) (with 1 = [I, L]) the input signal supposed to supply the L loudspeakers. We write BRIR ^{g / d} (1), the BRIRs of each loudspeaker for each of the two ears, and we write O ^{g / d} the binaural output signal. The binauralization of the multichannel signal is therefore written: $$O^g={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}I\left(l\right)}*{\mathit{BRIR}}^g\left(l\right)$$

$$O^d={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}I\left(l\right)}*{\mathit{BRIR}}^d\left(l\right)$$

Where * represents the convolution operator.

Subsequently, the index 1 such that l ∈ [1, L ] refers to one of the L loudspeakers. We have a BRIR for a signal 1.

So, with reference to the figure 1 two convolutions (one for each ear) are present for each speaker (steps S11 to S1L).

For loudspeakers, binauralization therefore requires 2.L convolutions. The complexity C _conv can be calculated in the case of a fast block implementation. A fast implementation by block is for example given by a Fast Fourier Transform (FFT). The document "Submission and Evaluation Procedures for 3D Audio" specifies a possible formula for the calculation of C _conv : $$C_{\mathit{conv}}=\left(\mathrm{The}+2\right).\left(\mathit{nBlocs}\right).\left(6.{\mathit{<figure-callout\; id="2"\; label="log"\; filenames="imgf0001.png,imgf0003.png"\; state="\{\{state\}\}">log</figure-callout>}}_2\left(2\mathit{fs}/\mathit{nBlocs}\right)\right)$$

In this equation, L represents the number of FFTs to frequency transform the input signals (1 FFT per input signal), the 2 represents the number of inverse FFTs to obtain the time binaural signal (2 inverse FFTs for both binaural channels), the 6 indicates a coefficient of complexity per FFT, the second 2 indicates a zero stuffing necessary to avoid the problems due to the circular convolution, Fs indicates the size of each of the BRIRs, and nBlocs represents the fact of use block processing, more realistic in an approach where latency should not be excessively high, and. represents multiplication.

Thus for typical use with nBlocs = 10, Fs = 48000, L = 22, the multichannel signal sample complexity for direct convolution based on FFT is C _conv = 19049 multiplications-additions.

This complexity is too high for a realistic implementation at present on current processors (mobile for example), it is therefore necessary to reduce this complexity without greatly degrading the rendering binauralization.

For spatialization to be of good quality, the entire time signal of the BRIRs must be applied.

US 2006/045294 A1 discloses a sound spatialization system of multichannel signals for delivering at least two output channels, and including partitioning the two-part impulse responses, the first containing the direct arrival and the first reflections, and the second containing the reverb, the second portions of at least two impulse responses being summed and weighted to obtain a unique transfer function representing the reverb to be applied to the input signals, in order to reduce the complexity of the convolution operations. US 2006/045294 A1 also discloses block filtering of FFTs.

STEWART REBECCA AND AL: "Generating a Spatial Average Reverberation Tail Across Multiple Impulse Responses", CONFERENCE: 35TH INTERNATIONAL CONFERENCE: AUDIO FOR GAMES; FEBRUARY 2009, AES, NEW YORK, USA, February 1, 2009 (2009-02-01 ) discloses a partitioning of similar impulse responses, as well as the averaging of the impulse response portions representing the reverberation, to reduce the redundancy in the impulse response databases.

GARDNER WG: "EFFICIENT CONVOLUTION WITHOUT INPUT-OUTPUT DELAY", AUDIO ENGINEERING SOCIETY, NEW YORK, NY, US, vol. 43, no. 3, March 1, 1995 (1995-03-01), pages 127-135 describes an efficient implementation for the convolution of impulse responses, particularly in the context of reverberation simulations, in the form of FFT blocks. This article suggests reusing transformations of input signals as soon as possible, to reduce computational costs, and shows that partitioning an impulse response into sub-blocks of multiple lengths from one another reduces complexity. Calculation.

The present invention improves the situation.

It aims to greatly reduce the complexity of binauralizing a multichannel signal with room effect while maintaining the best audio quality.

The present invention proposes for this purpose a sound spatialization method as defined in the Claims 1 to 10.

The invention also relates to a computer program comprising instructions for implementing the method.

The invention can be implemented by a sound spatialization device as defined in claim 12.

The invention can also be implemented in a sound signal decoding module, as input signals, comprising the spatialization device above.

• la figure 1 illustre un procédé de spatialisation de l'art antérieur,
• la figure 2 illustre schématiquement les étapes d'un procédé au sens de l'invention, dans un exemple de réalisation,
• la figure 3 représente une réponse impulsionnelle binaurale de salle BRIR,
• la figure 4 illustre schématiquement les étapes d'un procédé au sens de l'invention, dans un exemple de réalisation,
• la figure 5 illustre schématiquement les étapes d'un procédé au sens de l'invention, dans un exemple de réalisation,
• la figure 6 représente schématiquement un dispositif comportant des moyens de mise en oeuvre du procédé au sens de l'invention.

Other advantages and characteristics of the invention will appear on reading the following detailed description of embodiments of the invention and on examining the drawings in which:

• the figure 1 illustrates a method of spatialization of the prior art,
• the figure 2 schematically illustrates the steps of a method in the sense of the invention, in an exemplary embodiment,
• the figure 3 represents a binaural impulse response of BRIR room,
• the figure 4 schematically illustrates the steps of a method in the sense of the invention, in an exemplary embodiment,
• the figure 5 schematically illustrates the steps of a method in the sense of the invention, in an exemplary embodiment,
• the figure 6 schematically represents a device comprising means for implementing the method within the meaning of the invention.

We refer to the figure 6 to first illustrate a possible context of implementation of the present invention in a TER terminal connected device (for example a phone, smartphone or other, or a connected tablet, a connected computer, or others). Such a device TER comprises reception means (typically an antenna) of audio signals Xc encoded in compression, a decoding device DECOD delivering decoded signals X ready to be processed by a spatialization device before the audio signals are returned (for example in binaural on a CAS headset). Of course, in some cases, it may be advantageous to keep the partially decoded signals (for example in the subband domain) if the spatialization processing is performed in the same domain (frequency processing in the subband field by example).

• hardware comportant typiquement un ou plusieurs circuits CIR coopérant avec une mémoire de travail MEM et un processeur PROC,
• et software, dont les figures 2 et 4 sont des exemples d'organigrammes en illustrant l'algorithme général.

Still referring to the figure 6 , the spatialization device is presented by a combination of elements:

• hardware typically having one or more CIR circuits cooperating with a working memory MEM and a processor PROC,
• and software, whose figures 2 and 4 are examples of flowcharts illustrating the general algorithm.

Here, the cooperation between the hardware and software elements produces a technical effect providing in particular an economy of complexity of the spatialization for substantially the same audio rendering (same sensation for a listener), as will be seen below.

We now refer to the figure 2 to describe a treatment in the sense of the invention, and implemented by computer means.

In a first step S21, a data preparation is performed. This preparation is optional, the signals can be processed according to steps S22 and following without this pre-treatment.

In particular, this preparation consists in truncating each BRIR to ignore the inaudible samples at the beginning and at the end of the impulse response.

• Une somme cumulée des énergies de chacun des filtres BRIR(1) est calculée. Typiquement, cette énergie est calculée par une somme au carré des amplitudes des échantillons 1 à j, avec j compris dans [1 ; J] avec J le nombre d'échantillon d'un filtre BRIR.
• La valeur d'énergie du filtre d'énergie maximum valMax (parmi les filtres relatifs à l'oreille gauche et à l'oreille droite) est calculée.
• Pour chacun des haut-parleurs 1, on calcule l'indice pour lequel l'énergie de chacun des filtres BRIR(1) dépasse un certain seuil en dB calculé par rapport à valMax (e.g. valMax-50dB).
• L'indice de troncature iT retenu pour toutes les BRIR est l'indice minimum parmi tous les indices des BRIR et il est considéré comme l'instant de début d'ondes sonores directes.

This preparation, for the truncation at the beginning of the TRONC S impulse response, in step S211, consists in determining a start time of direct sound waves and can be implemented by the following steps:

• A cumulative sum of the energies of each of the BRIR filters (1) is calculated. Typically, this energy is computed by a sum squared of the amplitudes of samples 1 to j, with j included in [1; J] with J the sample number of a BRIR filter.
• The energy value of the maximum energy filter valMax (among the filters relating to the left ear and the right ear) is calculated.
• For each of the loudspeakers 1, the index for which the energy of each of the BRIR filters (1) exceeds a certain threshold in dB calculated with respect to valMax (eg valMax-50dB) is calculated.
• The truncation index iT retained for all the BRIRs is the minimum index among all the indices of the BRIRs and is considered as the moment of beginning of direct sound waves.

The index iT obtained therefore corresponds to the number of samples to be ignored for each of the BRIRs. Abrupt truncation at the beginning of an impulse response with a rectangular window can lead to audible artifacts if it is applied in too much energy. It may therefore be preferable to apply a suitable input fade window, however if precautions have been taken in the selected threshold, this windowing becomes useless, because inaudible (just cut the inaudible signal).

The synchronism between BRIR makes it possible to apply a constant delay for all BRIRs for the sake of simplicity of implementation, even if an optimization of complexity is possible.

The truncation of each BRIR to ignore the inaudible samples at the end of the impulse response TRONC E, in step S212, can be performed from steps similar to those described above, adapted to suit the end of the impulse response. Sudden truncation at the end of an impulse response with a rectangular window may lead to audible artifacts on pulse signals where the reverb tail may be audible. Thus, in one embodiment, a suitable output fade window is applied.

In step 22, ISOL A / B synchronism isolation is performed. This isolation in synchronism consists of separating, for each BRIR, the part "direct sound" and "first reflections" (or Direct, noted A) and the part "diffuse sound" (or Diffus, noted B). Indeed, the treatment to be performed on the "diffuse sound" part may advantageously be different from that to be performed on the "direct sound" part, since it is preferable to have a better quality of treatment on the "direct sound" part. Only on the "diffuse sound" part. This makes it possible to optimize the quality / complexity ratio.

In particular, to achieve isolation in synchronism, a single sample "iDD" index common to all BRIRs (hence the term "synchronism") from which the remainder of the impulse response is considered is determined. corresponds to a diffuse field. We therefore partition the BRIR (1) impulse responses into two: A (1) and B (1), where the concatenation of the two corresponds to BRIR (1).

The figure 3 shows the iDD partitioning index at the 2000 sample. The left part of this iDD index corresponds to part A. The right part of this iDD index corresponds to part B.

In one embodiment, these two parts are isolated, without windowing, in order to undergo different treatments. In a variant, a windowing between the parts A (1) and B (1) is applied.

avec V_salle le volume de la salle de mesure.The iDD index may be specific to the room for which the BRIRs were determined. The calculation of this index may therefore depend on the spectral envelope, the correlation of the BRIRs or the echogram of these BRIRs. For example, iDD can be determined by a formula of the type $\mathit{iDD}=\sqrt{V_{\mathit{Salls}}}$

with V _room the volume of the measuring room.

In one embodiment, iDD is a fixed value, typically 2000. In one variant, iDD varies, advantageously dynamically, depending on the environment from which the input signals are captured.

où z ^-iDD correspond au délai de iDD échantillons.The output signal for the left (g) and right (d) ears, represented by O ^{g / d} , is written as follows: $$\begin{array}{l}{O}^{g/d}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}I\left(l\right)}*{\mathit{BRIR}}^{g/d}\left(l\right)=O_{\mathrm{AT}}^{g/d}+z^{-\mathit{iDD}},O_B^{g/d}\\ ={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}I\left(l\right)}*{\mathrm{AT}}^{g/d}\left(l\right)+z^{-\mathit{iDD}}.{\displaystyle \underset{i=1}{\overset{\mathrm{The}}{\Sigma }}I\left(l\right)}*B^{g/d}\left(l\right)\end{array}$$

where z ^{- iDD} is the delay of iDD samples.

dans une mémoire temporaire (par exemple dans un buffer) et en les restituant au moment voulu.The application of this delay to the signals is carried out by storing the values calculated for ${\displaystyle {\Sigma }_{l=1}^{\mathrm{The}}I\left(l\right)*B^{g/d}\left(l\right)}$

in a temporary memory (for example in a buffer) and restoring them at the desired moment.

In one embodiment, the sample indices selected for A and B may also consider frame lengths in the case of integration into an audio encoder. Indeed, typical frame sizes of 1024 samples can lead to a choice such that A makes 1024 and B makes 2048, making sure that B is a diffuse field area for all BRIRs.

In particular, it may be interesting that the size of B is a multiple of the size of A because if the filtering is implemented in blocks of FFT, then the calculation of an FFT for A can be reused for B.

A diffuse field is characterized by the fact that it is statistically identical in all points of the room. Thus, its frequency response varies little depending on the speaker to simulate. The present invention exploits this feature in order to replace all Diffus D (l) filters of all BRIRs with a single and only one "mean" B _mean filter in order to greatly reduce the complexity due to multiple convolutions. For this purpose, it is possible to modify the diffuse field part B in step S23B, again with reference to the figure 2 .

dans le signal d'entrée : $$\begin{array}{l}{O}_B^{g/d}={\displaystyle \sum _{l=1}^L\left[I\left(l\right)\ast B^{g/d}\left(l\right)\right]}={\displaystyle \sum _{l=1}^L\left[I\left(l\right)\ast \left(\sqrt{E_{B^{g/d}\left(l\right)}}.{B_{\mathit{norm}}}^{g/d}\left(l\right)\right)\right]}\\ ={\displaystyle \sum _{l=1}^L\left[\left(\sqrt{E_{B^{g/d}\left(l\right)}}.I\left(l\right)\right)\ast {B_{\mathit{norm}}}^{g/d}\left(l\right)\right]}\end{array}$$

avec ${B_{\mathit{norm}}}^{g/d}\left(l\right)=\frac{B^{g/d}\left(l\right)}{\sqrt{E_{B^{g/d}\left(l\right)}}}$

où E _{B^g / ^d (l)} représente l'énergie de B ^g/d (l).In step S23B1, the value of the mean filter B _mean is calculated. First, it is extremely rare that the complete system is calibrated ideally, so we can apply a weight gain that will be reported in the input signal to perform a single convolution per ear for the diffuse field part. The BRIRs are therefore decomposed into standard energy filters, and the normalization gain is postponed. $\sqrt{E_{B^{g/d}\left(l\right)}}$

in the input signal: $$\begin{array}{l}{O}_B^{g/d}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*B^{g/d}\left(l\right)\right]}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*\left(\sqrt{E_{B^{g/d}\left(l\right)}}.{B_{\mathit{norm}}}^{g/d}\left(l\right)\right)\right]}\\ ={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[\left(\sqrt{E_{B^{g/d}\left(l\right)}}.I\left(l\right)\right)*{B_{\mathit{norm}}}^{g/d}\left(l\right)\right]}\end{array}$$

with ${B_{\mathit{norm}}}^{g/d}\left(l\right)=\frac{B^{g/d}\left(l\right)}{\sqrt{E_{B^{g/d}\left(l\right)}}}$

where E _{B ^g / ^d ( l )} represents the energy of B ^{g / d} ( 1 ).

avec ${B_{\mathit{mean}}}^{g/d}=\frac{1}{L}{\displaystyle {\sum }_{l=1}^L\left[{B_{\mathit{norm}}}^{g/d}\left(l\right)\right]}$

Then, B _norm ^{g / d} ( l ) is approximated by a single mean filter B _mean ^{g / d} which is no longer a function of the loudspeaker 1, but it is also possible to standardize in energy: $$O_B^{g/d}\approx {\widehat{O}}_B^{g/d}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[\left(\sqrt{E_{B^{g/d}\left(l\right)}}.I\left(l\right)\right)*\left(\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)\right]}$$

with ${B_{\mathit{mean}}}^{g/d}=\frac{1}{\mathrm{The}}{\displaystyle {\Sigma }_{l=1}^{\mathrm{The}}\left[{B_{\mathit{norm}}}^{g/d}\left(l\right)\right]}$

In one embodiment, this average filter can be obtained by averaging time samples. In a variant, it can be obtained by any other type of averaging such as averaging power spectral densities.

In one embodiment, the energy of the average filter E _{E mean} ^{g / d} can be measured directly from the constructed filter B _mean ^{g / d} . In a variant, it can also be estimated taking into account the hypothesis that the filters B _norm ^{g / d} ( 1 ) are decorrelated. Indeed, in this case, as we sum unit energy signals, we have: $$E_{{B_{\mathit{mean}}}^{g/d}}={\displaystyle <figure-callout\; id="1"\; label="\Sigma "\; filenames="imgf0001.png,imgf0003.png"\; state="\{\{state\}\}">\Sigma </figure-callout>{\left(\frac{1}{\mathrm{The}}{\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[{B_{\mathit{norm}}}^{g/\mathrm{<figure-callout\; id="2"\; label="d\; l"\; filenames="imgf0001.png,imgf0003.png"\; state="\{\{state\}\}">d\; </figure-callout>}}\left(l\right)\left(\right)\right]}\right)}^2}=\frac{1}{{\mathrm{The}}^2}.\left(\mathrm{The}.E_{{B_{\mathit{norm}}}^{g/d}}\right)=\frac{1}{\mathrm{The}}$$

The energy can be calculated on all the samples corresponding to the diffuse field part.

Avec $W^{g/d}\left(l\right)=\frac{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}{\sqrt{E_{B^{g/d}\left(l\right)}}}$

In step S23B2, the value of the weighting factor W ^{g / d} ( 1 ) is calculated. A single weighting factor to be applied to the input signal is calculated, taking into account the standardizations of the Diffus filters and the average filter: $${\widehat{\mathrm{O}}}_B^{g/d}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[\left(\frac{\sqrt{E_{B^{g/d}\left(l\right)}}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}.I\left(l\right)\right)*{B_{\mathit{mean}}}^{g/d}\right]}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[\left(\frac{1}{W^{g/d}\left(l\right)}.I\left(l\right)\right).{B_{\mathit{mean}}}^{g/d}\right]}$$

With $W^{g/d}\left(l\right)=\frac{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}{\sqrt{E_{B^{g/d}\left(l\right)}}}$

Since the average filter is constant, it can come out of the sum: $${\widehat{\mathrm{O}}}_B^{g/d}={\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[\left(\frac{1}{W^{g/d}\left(l\right)}.I\left(l\right)\right)\right]}*{B_{\mathit{mean}}}^{g/d}$$

Thus, the L convolutions with the diffuse field portion are replaced by a single convolution with a mean filter, with a weighted sum of the input signal.

In step S23B3, it is optionally possible to calculate a gain G correcting the gain of the average filter B _mean ^{g / d} . Indeed, in the case of the convolution between the input signals and the unmatched filters, whatever the correlation values between the input signals, the filtering by decorrelated filters that are the B ^{g / d} ( l ) leads to signals to be summed up which are then also decorrelated. Conversely, in the case of the convolution between the input signals and the approximated average filter, the energy of the signal resulting from the summation of the filtered signals will depend on the correlation value existing between the input signals.

* si tous les signaux d'entrée I(1) sont décorrélés et d'énergie unitaire, et que les filtres B(l) sont tous d'énergie unitaire, mais remplacés par des filtres identiques $\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}},$

on a: $$\begin{array}{l}{E}_{{\widehat{\mathrm{O}}}_B^{g/d}}=\mathit{energie}\left({\displaystyle \sum _{l=1}^L\left[I\left(l\right)\ast \left(\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)\right]}\right)\\ =\mathit{energie}\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}.{\displaystyle \sum _{l=1}^L\left[I\left(l\right)\ast {B_{\mathit{mean}}}^{g/d}\right]}\right)={\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)}^2.\left(L.\frac{1}{L}\right)=L\end{array}$$

For example,
if all the input signals I (1) are identical and of unit energy, and the filters B (1) are all decorrelated (since diffuse fields) and of unit energy, we have: $$E_{O_B^{g/d}}=\mathit{energy}\left({\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*{B_{\mathit{norm}}}^{g/d}\left(l\right)\right]}\right)=\mathrm{The}$$

* if all the input signals I (1) are decorrelated and of unit energy, and the filters B (1) are all of unit energy, but replaced by identical filters $\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}},$

we have: $$\begin{array}{l}{E}_{{\widehat{\mathrm{O}}}_B^{g/d}}=\mathit{energy}\left({\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*\left(\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)\right]}\right)\\ =\mathit{energy}\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}.{\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*{B_{\mathit{mean}}}^{g/d}\right]}\right)={\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/\mathrm{<figure-callout\; id="2"\; label="d"\; filenames="imgf0001.png,imgf0003.png"\; state="\{\{state\}\}">d</figure-callout>}}}}}\right)}^2.\left(\mathrm{The}.\frac{1}{\mathrm{The}}\right)=\mathrm{The}\end{array}$$

Because the energies of the decorrelated signals are added.

on a: $$\begin{array}{l}{E}_{{\widehat{\mathrm{O}}}_B^{g/d}}=\mathit{energie}\left({\displaystyle \sum _{l=1}^L\left[I\left(l\right)\ast \left(\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)\right]}\right)\\ =\mathit{energie}\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}.{\displaystyle \sum _{l=1}^L\left[I\left(l\right)\ast {B_{\mathit{mean}}}^{g/d}\right]}\right)={\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)}^2.\left(L^2.\frac{1}{L}\right)\\ =L^2\end{array}$$

This case is equivalent to the previous one in the sense that the signals coming from the filtering are all decorrelated, thanks to the input signals in the first case, and thanks to the filters in the second case.
* if all the input signals I (1) are identical and unit energy, and the filters B (l) are all of unit energy, but replaced by identical filters $\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}},$

we have: $$\begin{array}{l}{E}_{{\widehat{\mathrm{O}}}_B^{g/d}}=\mathit{energy}\left({\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*\left(\frac{{B_{\mathit{mean}}}^{g/d}}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}\right)\right]}\right)\\ =\mathit{energy}\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/d}}}}.{\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[I\left(l\right)*{B_{\mathit{mean}}}^{g/d}\right]}\right)={\left(\frac{1}{\sqrt{E_{{B_{\mathit{mean}}}^{g/\mathrm{<figure-callout\; id="2"\; label="d"\; filenames="imgf0001.png,imgf0003.png"\; state="\{\{state\}\}">d</figure-callout>}}}}}\right)}^2.\left({\mathrm{The}}^2.\frac{1}{\mathrm{The}}\right)\\ ={\mathrm{The}}^2\end{array}$$

Because the energies of the identical signals are added in quadrature (because their amplitudes are added).

• si deux haut-parleurs sont actifs simultanément, alimentés par des signaux décorrélés, alors aucun gain n'est apporté en appliquant les étapes S23B1 et S23B2 par rapport à la méthode classique.
• si deux haut-parleurs sont actifs simultanément, alimentés par des signaux identiques, alors un gain de 10. log₁₀(L ²/L) =10.log ₁₀(2²/2) = 3.01dB est apporté en appliquant les étapes S23B1 et S23B2 par rapport à la méthode classique.
• si trois haut-parleurs sont actifs simultanément, alimentés par des signaux identiques, alors un gain de 10.log ₁₀(L ²/L) = 10.log ₁₀(3²/3) = 4.77dB est apporté en appliquant les étapes S23B1 et S23B2 par rapport à la méthode classique.

So,

• if two loudspeakers are active simultaneously, powered by decorrelated signals, then no gain is made by applying the steps S23B1 and S23B2 compared to the conventional method.
• if two loudspeakers are active simultaneously, powered by identical signals, then a gain of 10. lo g ₁₀ ( L ² / L ) = 10. log ₁₀ (2 ^2/2 ) = 3.01 dB is provided by applying steps S23B1 and S23B2 compared to the conventional method.
• if three speakers are active simultaneously, fed with identical signals, then a gain of 10 _log10 (L ² / L) = 10 log ^{_{10 (3 2/3) =}} 4.77 dB is made by applying the steps S23B1 and S23B2 compared to the conventional method.

The cases mentioned above correspond to the extreme cases of identical or uncorrelated signals. These cases are however realistic: a source positioned in the middle of two speakers, virtual or real, provide a signal identical to these two speakers (for example with a technique of type VBAP, for "Vector base amplitude panning"). In the case of positioning in a 3D system, the 3 loudspeakers can receive the same signal at the same level.

Thus, compensation can be applied to respect the energy of binauralized signals.

Ideally, this compensation gain G will be determined according to the input signal (ie G (I (1))) and will be applied to the sum of the weighted input signals: $${\widehat{\mathrm{O}}}_B^{g/d}=G.{\displaystyle \underset{l=1}{\overset{\mathrm{The}}{\Sigma }}\left[\frac{1}{W^{g/d}\left(l\right)}.I\left(l\right)\right]}*{B_{\mathit{mean}}}^{g/d}$$

The gain G ( I ( 1 )) can be estimated by a calculation of correlation between each of the signals. It can also be estimated by comparing the energies of the signals before and after summations. In this case, the gain G may vary dynamically over time, depending for example on correlations between the input signals, which vary themselves over time.

), ou au filtre B_mean ^g/d , ce qui évitera l'application d'un gain supplémentaire au vol.In a simplified embodiment, it is possible to set a constant gain, for example G = -3 dB = 10 ^-3/20 , which will avoid having to make a correlation estimate that can be expensive. The constant gain G can then be applied offline to the weighting factors (thus giving $\frac{G}{W^{g/d}\left(l\right)}$

), or the filter B _mean ^{g / d} , which will avoid the application of an additional gain on the fly.

Once the transfer functions A and B isolated and the filters B _mean ^{g / d} (optionally weights W ^{g / d} ( I ) and G) are calculated, these transfer functions and these filters are applied to the input signals.

• On applique (étapes S4A1 à S4AL) au signal multicanal d'entrée un filtrage efficace (par exemple convolution directe basée-FFT) par les filtres Direct (A), comme décrit dans l'état de l'art. On obtient un signal ${\widehat{\mathrm{O}}}_{\mathrm{A}}^{\mathrm{g}/\mathrm{d}}$
  
• En fonction des relations entre les signaux d'entrée, notamment en fonction de leur corrélation, on peut optionnellement corriger à l'étape S4B11 le gain du filtre moyen B_mean ^g/d par application du gain G aux signaux de sortie après sommation des signaux d'entrée préalablement pondérés (étapes M4B1 à M4BL).
• On applique au signal multicanal B à l'étape S4B1 un filtrage efficace par le filtre Diffus moyen B_mean. Cette étape a bien lieu après sommation des signaux d'entrée préalablement pondérés (étapes M4B1 à M4BL). On obtient le signal ${\widehat{\mathrm{O}}}_B^{g/d}.$
  
• On applique au signal ${\widehat{\mathrm{O}}}_B^{g/d}$
  
  un délai iDD afin de compenser le retard introduit lors de l'étape d'isolation du signal B à l'étape S4B2.
• Les signaux ${\widehat{\mathrm{O}}}_{\mathrm{A}}^{\mathrm{g}/\mathrm{d}}$
  
  et ${\widehat{\mathrm{O}}}_B^{g/d}$
  
  sont sommés.
• Si une troncature éliminant les échantillons inaudibles au début des réponses impulsionnelles a été réalisée, alors on applique à l'étape S41 au signal d'entrée un délai iT correspondant aux échantillons inaudibles supprimés.

In a first embodiment, described with reference to the figure 4 , the multichannel signal processing by applying the Direct (A) and Diffus (B) filters for each of the ears is performed as follows:

• The input multichannel signal (S4A1 to S4AL) is efficiently filtered (eg FFT based direct convolution) by the Direct filters (A) as described in the state of the art. We get a signal ${\widehat{\mathrm{O}}}_{\mathrm{AT}}^{\mathrm{g}/\mathrm{d}}$
  
• As a function of the relations between the input signals, in particular as a function of their correlation, it is optionally possible to correct, in step S4B11, the gain of the mean filter B _mean ^{g / d} by applying the gain G to the output signals after summation of the signals previously weighted input (steps M4B1 to M4BL).
• The multichannel signal B in step S4B1 is effectively filtered by means of the _mean diffuse filter B _mean . This step takes place after summing the previously weighted input signals (steps M4B1 to M4BL). We get the signal ${\widehat{\mathrm{O}}}_B^{g/d}.$
  
• We apply to the signal ${\widehat{\mathrm{O}}}_B^{g/d}$
  
  a delay iDD to compensate for the delay introduced during the step of isolating the signal B in step S4B2.
• Signals ${\widehat{\mathrm{O}}}_{\mathrm{AT}}^{\mathrm{g}/\mathrm{d}}$
  
  and ${\widehat{\mathrm{O}}}_B^{g/d}$
  
  are summoned.
• If a truncation eliminating inaudible samples at the beginning of the impulse responses has been performed, then in step S41 the input signal is applied with a delay iT corresponding to the inaudible samples deleted.

In a variant, with reference to the figure 5 , the signals are not only calculated for the left and right ears (indices g and d above) but for k playback devices (typically loudspeakers).

In a second embodiment, the gain G is applied prior to the summing of the input signals, that is to say during the weighting steps (steps M4B1 to M4BL).

In a third embodiment, a decorrelation is applied to the input signals. Thus, the signals are decorrelated after convolution by the B _mean filter regardless of the original correlations between input signals. An efficient implementation of decorrelation (for example using a loopback network) can be used to avoid the use of expensive decorrelating filters.

• tronquées entre l'échantillon 150 et l'échantillon 3222 par la technique décrite à l'étape S21,
• décomposées en deux parties : champ direct A de 1024 échantillons, et champ diffus B de 2048 échantillons, par la technique décrite à l'étape S22,

alors la complexité de binauralisation peut être approximativement donnée par : $${\mathrm{C}}_{\mathrm{inv}}={\mathrm{C}}_{\mathrm{invA}}+{\mathrm{C}}_{\mathrm{invB}}=\left(\mathrm{L}+2\right).\left(6.{\log }_2\left(2.\mathrm{NA}\right)\right)+\left(\mathrm{L}+2\right).\left(6.{\log }_2\left(2.\mathrm{NB}\right)\right)$$

Avec NA et NB les tailles en échantillons de A et BThus, realistically assuming that BRIRs of 48,000 samples length can be:

• truncated between sample 150 and sample 3222 by the technique described in step S21,
• decomposed into two parts: direct field A of 1024 samples, and diffuse field B of 2048 samples, by the technique described in step S22,

then the binauralization complexity can be approximately given by: $${\mathrm{C}}_{\mathrm{inv}}={\mathrm{C}}_{\mathrm{invA}}+{\mathrm{C}}_{\mathrm{INVB}}=\left(\mathrm{The}+2\right).\left(6.{\log }_2\left(2.\mathrm{N\; /\; A}\right)\right)+\left(\mathrm{The}+2\right).\left(6.{\log }_2\left(2.\mathrm{NB}\right)\right)$$

With NA and NB the sizes in samples of A and B

Thus for nBlocks = 10, Fs = 48000, L = 22, NA = 1024 and NB = 2048, the multichannel signal sample complexity for a FFT-based convolution is C _conv = 3312 multiplications-additions. This result is, however, logically compared to a simple solution implementing only truncation, ie for nBlocs = 10, Fs = 3072, L = 22: $${\mathrm{C}}_{\mathrm{trunk}}=\left(\mathrm{The}+2\right).\left(\mathrm{nBlocs}\right).\left(6.{\log }_2\left(2.\mathrm{fs}/\mathrm{nBlocs}\right)\right)=13339$$

There is therefore a factor 19049/3312 = 5.75 of complexity between the state of the art and the present invention, and another factor 13339/3312 = 4 of complexity between the state of the art benefiting from the truncation and the present invention.

If the size of B is a multiple of the size of A, then if the filtering is implemented in blocks of FFT, the calculation of an FFT for A can be reused for B. We therefore need L FFT on NA points, which will serve at the same time for the filtering by A and by B, two inverse FFT on NA points to obtain the binaural time signal, and the multiplication of the spectrums in frequency.

In this case, the complexity can be approximated (the additions are neglected, (L + 1) corresponds to the multiplication of the spectra, L for A and 1 for B) by: $${\mathrm{C}}_{\mathrm{inv}2}=\left(\mathrm{The}+2\right).\left(6.{\log }_2\left(2.\mathrm{N\; /\; A}\right)\right)+\left(\mathrm{The}+1\right)=1607$$

With this approach, we still gain a factor of 2, and therefore a factor of 12 and 8 compared to the state of the art untruncated and truncated.

The invention can find a direct application in the MPEG-H 3D Audio standard.

Of course, the present invention is not limited to the embodiment described above; it extends to other variants while remaining within the scope of the protection defined in the appended claims.

For example, an embodiment has been described above in which the Direct A signal is not approximated by an average filter. Of course, it is possible to use an average filter of A to make the convolutions (steps S4A1 to S4AL) with the signals coming from the loudspeakers.

An embodiment has been described above based on the processing of multichannel content generated for L speakers. Of course, the multichannel content can be generated by any type of audio source such as voice, a musical instrument, any noise, etc.

An embodiment has been described above based on determined BRIR values in a room. Of course, the present invention can be implemented for any type of external environment (eg concert hall, open air, etc.).

An embodiment has been described above based on the application of two transfer functions. Of course, the present invention can be implemented with more than two transfer functions. For example, one can isolate in synchronism a part relating to the sounds emitted directly, a part relating to the first reflections and a part relating to diffuse sounds.

Claims (13)

1. Audio spatializing method wherein at least one filtering operation is applied to at least two input signals (I(1), I(2), ..., I(L)) in order to deliver at least two output signals (O(1), O(2), ..., O(K)), the filtering operation comprising:

   - weighting (M4B1, M4B2, ..., M4BL) said at least two input signals with respective weighting weights (W^k(1), ..., W^k(L)), each weighting weight being specific to each of the input signals;

   - for each impulse response incorporating a room effect among a plurality of impulse responses incorporating a room effect, said impulse response incorporating a room effect being respectively associated with one input signal among said at least two input signals (I(1), 1(2), ..., I(L)) and with one output signal among said at least two output signals (O(1), O(2), ..., O(K)):

   ∘ partitioning (S22), in a time domain, said impulse response into a first portion (A) and a second portion (B), said partitioning being carried out such that:

   said first portion represents direct audio propagations and first audio reflections of said propagations and extends over a first number of samples; and

   said second portion represents a diffuse audio field present after said first reflections and extends over a second number of samples, said second number of samples being a multiple of said first number of samples;

   ∘ determining a first transfer function (A^k(1), A^k(2), ..., A^k(L)) from said first portion;

   ∘ determining a second transfer function from said second portion;

   - for each output signal (O(1), O(2), ..., O(K)) among said at least two output signals (O(1), O(2), ..., O(K)):

   ∘ determining (S23B1) a third transfer function (B_mean ^k) from an average of said second transfer functions corresponding to the output signal (O(1), O(2), ..., O(K));

   ∘ applying (S4A1, S4A2, ..., S4AL) to each input signal (1(1), 1(2), ..., I(L)) the first transfer function (A^k(1), A^k(2), ..., A^k(L)) corresponding to the input signal (I(1), 1(2), ..., I(L)) and to the output signal (O(1), O(2), ..., O(K));

   ∘ applying (S4B1) to each input signal the third transfer function (B_mean ^k) corresponding to the output signal (O(1), O(2), ..., O(K)); wherein the application of the first and third transfer functions is carried out using FFT blocks;

   - summing the signals resulting from applying said first and third transfer functions in order to obtain said at least two output signals (O(1), O(2), ..., O(K)).
2. Method according to Claim 1, characterized in that an energy-compensating gain (G) is applied (S4B11) to the weighting weights (W^k(1), ..., W^k(L)).
3. Method according to one of the preceding claims, characterized in that said partitioning of said impulse response comprises operations for:

   - determining (S211) a start time of presence of direct audio waves,

   - determining a start time of presence of said diffuse audio field after the first reflections and

   - selecting (S22), in said impulse response, a portion of the response that extends temporally between said start time of presence of direct audio waves to said start time of presence of the diffuse field, said selected response portion corresponding to said first transfer function.
4. Method according to Claim 3, characterized in that said filtering comprises applying at least one compensating delay (S4B2) corresponding to a temporal offset between said start time of direct audio waves and said start time of presence of the diffuse field.
5. Method according to Claim 4, characterized in that said first and third transfer functions are applied in parallel to said input signals and in that said at least one compensating delay is applied to the input signals filtered by said third transfer functions.
6. Method according to one of the preceding claims, wherein said third transfer function is given by: $$B_{\mathit{mean}}^k=\frac{1}{L}{\displaystyle \sum _{i=1}^L\left[B_{\mathit{norm}}^k\left(l\right)\right]}$$

   

   with:

   k an index relative to an output signal,

   l ∈ [1; L] an index relative to one input signal among said input signals,

   L a number of input signals, and

   $B_{\mathit{norm}}^k\left(l\right)$

   

   a normalised transfer function obtained from one second transfer function among said second transfer functions.
7. Method according to Claim 6, characterized in that at least one output signal 0^k of said method is given by: $$O^k={\displaystyle \sum _{l=1}^L\left(I\left(l\right)\ast A^k\left(l\right)\right)}+z^{-\mathit{iDD}}\cdot {\displaystyle \sum _{l=1}^L\left(\frac{1}{W^k\left(l\right)}.I\left(l\right)\right)}\ast B_{\mathit{mean}}^k$$

   

   with:

   I(l) one input signal among said input signals,

   A^k (l) one first transfer function among said first transfer functions,

   W^k (l) one weighting weight among said weighting weights,

   z^-iDD corresponds to the application of said compensating delay,

   where · is a multiplication, and

   where * is the convolution operator.
8. Method according to Claim 6, characterized in that it comprises a step of decorrelating the input signals, prior to applying the third transfer functions and in that at least one output signal 0^k of said method is given by: $$O^k={\displaystyle \sum _{l=1}^L\left(I\left(l\right)\ast A^k\left(l\right)\right)}+z^{-\mathit{iDD}}\cdot {\displaystyle \sum _{l=1}^L\left(\frac{1}{W^k\left(l\right)}\cdot I_d\left(l\right)\right)}\ast B_{\mathit{mean}}^k$$

   

   with:

   I(l) one input signal among said input signals,

   I_d (l) one input signal among said input signals having been subjected to said decorrelating step,

   A^k (l) one first transfer function among said first transfer functions,

   W^k (l) one weighting weight among said weighting weights,

   z^-iDD corresponds to the application of said compensating delay,

   where · is a multiplication, and

   where * is the convolution operator.
9. Method according to Claim 6, characterized in that it comprises a step of determining an energy-compensating gain depending on the input signals, and in that at least one output signal is given by: $$O^k={\displaystyle \sum _{l=1}^L\left(I\left(l\right)\ast A^k\left(l\right)\right)}+z^{-\mathit{iDD}}\cdot {\displaystyle \sum _{l=1}^L\left(G\left(I\left(l\right)\right)\cdot \frac{1}{W^k\left(l\right)}\cdot I\left(l\right)\right)}\ast B_{\mathit{mean}}^k$$

   

   with:

   I(l) one input signal among said input signals,

   G(I(l)) said determined energy-compensating gain,

   A^k (l) one first transfer function among said first transfer functions,

   W^k (l) one weighting weight among said weighting weights,

   z^-iDD corresponds to the application of said compensating delay,

   where · is a multiplication, and

   where * is the convolution operator.
10. Method according to one of Claims 6 to 9, characterized in that said weight is given by: $$W^k\left(l\right)=\frac{\sqrt{E_{B_{\mathit{mean}}^k}}}{\sqrt{E_{B^k\left(l\right)}}}$$

    

    with k the index relative to an output signal,

    l ∈ [1; L] the index relative to one input signal among said input signals,

    L the number of input signals,

    with $E_{B_{\mathit{mean}}^k}$

    

    an energy relative to $B_{\mathit{mean}}^k$

    

    E _{B^k (l)} an energy relative to a one second transfer function among said second transfer functions.
11. Computer program containing instructions for implementing the method according to one of Claims 1 to 10, when these instructions are executed by a processor.
12. Audio spatializing device, comprising at least one filter applied to at least two input signals (I(1), I(2), ..., I(L)), the device being able to deliver at least two output signals (O(1), O(2), ..., O(K)),
    the device comprising weighting modules (M4B1, M4B2, ..., M4BL) for weighting said at least two input signals with respective weighting weights (W^k(1), ..., W^k(L)), each weighting weight being specific to each of the input signals;
    the device furthermore being configured to:

    - for each impulse response incorporating a room effect among a plurality of impulse responses incorporating a room effect, said impulse response incorporating a room effect being respectively associated with one input signal among said at least two input signals (I(1), I(2), ..., I(L)) and with one output signal among said at least two output signals (O(1), O(2), ..., O(K)):

    ∘ partition (S22), in a time domain, said impulse response into a first portion (A) and a second portion (B), said partitioning being carried out such that:

    said first portion represents direct audio propagations and first audio reflections of said propagations and extends over a first number of samples; and

    said second portion represents a diffuse audio field present after said first reflections and extends over a second number of samples, said second number of samples being a multiple of said first number of samples;

    ∘ determine a first transfer function (A^k(1), A^k(2), ..., A^k(L)) from said first portion;

    ∘ determine a second transfer function from said second portion;

    the filter comprising:

    - for each output signal (O(1), O(2), ..., O(K)) among said at least two output signals (O(1), O(2), ..., O(K)):

    ∘ determining (S23B1) a third transfer function (B_mean ^k) from an average of said second transfer functions corresponding to the output signal (O(1), O(2), ..., O(K));

    ∘ applying (S4A1, S4A2, ..., S4AL) to each input signal first transfer functions corresponding to the output signal (O(1), O(2), ..., O(K));

    ∘ applying (S4B1) to each input signal the third transfer function (B_mean ^k) corresponding to the output signal (O(1), O(2), ..., O(K));
    wherein the application of the first and third transfer functions is carried out using FFT blocks;

    wherein the signals resulting from applying said first and third transfer functions are summed in order to obtain said at least two output signals (O(1), O(2), ..., O(K)).
13. Module for decoding audio signals, comprising a device according to Claim 12 for spatializing said audio signals when input as input signals.

Images