EP3058564B1 - Sound spatialisation with reverberation, optimised in terms of complexity - Google Patents

Description

The present invention relates to sound spatialization with room effect.

The invention finds an advantageous but non-limiting application to a processing of sound signals respectively originating from L channels associated with virtual loudspeakers (for example in a multichannel representation, or even in an ambiophonic representation, of the sound to be reproduced), for a spatialized restitution on real loudspeakers (for example two earpieces of a helmet in binaural restitution, or two separate loudspeakers in transaural restitution).

For example, the signal from one of these channels can be processed to have a first contribution on the left atrium and a second contribution on the right atrium, in binaural reproduction, by applying in particular a transfer function with room effect to each of these contributions. The application of these hall effect transfer functions then contributes to offering the listener a feeling of immersion allowing him practically to “locate in space” the virtual loudspeaker associated with this channel.

• d'une première fonction de transfert, propre à chaque signal, et
• d'une deuxième fonction de transfert, globale, commune à tous les signaux et caractérisant en particulier un champ diffus, dont la présence advient habituellement dans une salle après un certain temps, typiquement après les premières réflexions d'une onde sonore.

In a particular embodiment, described in particular in the document FR1357299 , a transfer function with room effect is applied to each sound signal of a corresponding channel, in the time domain in the form of an impulse response of the BRIR type (for “Binaural Room Impulse Response” or “Binaural Room Impulse Response”). In particular, in this document, incorporated here by reference, this BRIR transfer function is constructed as the combination:

• a first transfer function, specific to each signal, and
• a second transfer function, global, common to all the signals and characterizing in particular a diffuse field, the presence of which usually occurs in a room after a certain time, typically after the first reflections of a sound wave.

Such an embodiment advantageously makes it possible to apply a common processing to all the signals, which corresponds, in a physical reality, to a "mixing" of the waves acoustic as the reverberations progress, therefore beyond a given duration (characterizing the beginning of the presence of the diffuse field). Such an embodiment then makes it possible to reduce the complexity of the spatialization processings with room effect on several initial channels.

Additionally, the document US 2011/170721 A1 discloses a method of processing at least one input signal through a set of binaural filters such that the outputs can be played over headphones to provide a listening experience of sound in a listening room through one or more virtual speakers.

Nevertheless, in spatialization modules intervening upstream of the restitution, one still seeks to reduce as much as possible the complexity of the spatialization processings. Indeed, for example (but not limitingly), the channel signals are received in encoded form, by a compression decoder. This decoder sends the channel signals, once decoded, to a spatialization module for sound reproduction with room effect, on two loudspeakers. This spatialization step (which follows the decoding of the signals received) should then be of reduced processing complexity so as not to delay the overall set of decoding and spatialization steps on reception of the signals before restitution.

The present invention improves the situation.

The invention proposes for this purpose to reduce the complexity of the application of the room effect transfer function, in particular by reducing this complexity in the spectral domain. Indeed, in the spectral domain, the convolution by a transfer function becomes a multiplication of spectral components of the signal on the one hand, and of a filter representing the transfer function on the other hand ( figure 1 discussed in detail below).

The invention then starts from the advantageous finding that, after direct propagation, a sound wave tends to attenuate at high frequencies due to progressive reflections on surfaces (typically walls, the listener's face, etc.) absorbing the wave, in particular at high frequencies. In addition, the air itself absorbs the highest frequency spectral components of sound as it propagates. This phenomenon is all the more increased for example for the diffuse sound field, for which it is not necessary to have a frequency representation for very high frequencies (for example higher than a frequency in a range of 5 to 15 kHz).

Thus, it is possible to reduce the complexity of the processing of the application of the transfer function with room effect, in the spectral domain, by simply not taking into account, to carry out the aforementioned multiplications of spectral components, the components associated with frequencies higher than a predetermined cut-off frequency (for example higher than 5 to 15 kHz).

The invention therefore aims at a method of sound spatialization, comprising the application of at least one room effect transfer function to at least one sound signal, said application amounting to multiplying, in the spectral domain, spectral components of the sound signal by the spectral components of a filter corresponding to the aforementioned transfer function. Each spectral component of the filter includes a temporal evolution in a time-frequency representation (as detailed later with reference to the picture 3 ).

In particular, these spectral components of the filter are ignored, for the aforementioned component multiplications, beyond a threshold frequency and after at least a given instant in said time-frequency representation. Thus, after this given instant, the spectral components of the filter are taken into account up to a cut-off frequency which can be chosen for example between 5 and 15 kHz (depending on the room effect to be applied and/or the signal to be spatialized, as described below). Beyond the cutoff frequency, the multiplication is not even performed, which mathematically is the same as multiplying the signal by zero.

This given instant typically represents the moment when a sound wave begins to undergo reverberations (by successive reflections, or, later still, from the presence of a diffuse sound field). Thus, in general terms, in an embodiment where the transfer function takes account of reverberations in the room effect (with consideration for example of a diffuse sound field), the aforementioned given instant can be chosen as a function of such reverberations. For example, the aforesaid given instant may be later, in the room effect, than direct sound propagation with first reflections, and then correspond to the beginning of the presence of a diffuse sound field.

Furthermore, an embodiment can be provided in which the aforementioned threshold frequency decreases as a function of time in said time-frequency representation. For example, if the signal is sampled over several successive blocks, provision may be made, by way of example, to retain the spectral components present in the signal, in the multiplication of the components, for a first block, then to ignore them beyond a first threshold frequency for a second block which follows the first block, then to ignore them beyond a second threshold frequency for a third block which follows the second block, etc., the second threshold frequency being lower than the first.

• au-delà d'une première fréquence seuil pour un bloc donné,
• puis, au-delà d'une deuxième fréquence seuil, pour un bloc qui suit le bloc donné,

la deuxième fréquence seuil étant plus basse que la première fréquence seuil. Le bloc donné précité peut inclure par exemple des échantillons situés temporellement à des instants qui correspondent à des moments où une onde sonore a subi une ou plusieurs réflexions, avec même un début de présence de champ sonore diffus. Le bloc qui suit ce bloc donné (immédiatement ou quelques blocs plus tard) peut inclure par exemple des échantillons situés temporellement après ou à partir d'un début de présence de champ sonore diffus.Thus, in more generic terms, in a realization where the signal is sampled over several successive blocks, the spectral components of the filter can be ignored, for the multiplication of the components:

• beyond a first threshold frequency for a given block,
• then, beyond a second threshold frequency, for a block following the given block,

the second threshold frequency being lower than the first threshold frequency. The aforementioned given block can include, for example, samples located temporally at times that correspond to times when a sound wave has undergone one or more reflections, with even the beginning of the presence of a diffuse sound field. The block which follows this given block (immediately or a few blocks later) can include for example samples located temporally after or from a beginning of presence of diffuse sound field.

, m étant un indice de bloc) caractérisant un champ sonore diffus. En effet, il est possible par exemple d'appliquer une fonction de transfert B^k _mean à un bloc donné précité, et d'appliquer une fenêtre de coupure temporellement progressive (de type « fade out ») à cette fonction de transfert B^k _mean, pour le bloc qui suit, pour « terminer » la présence du champ sonore diffus.Such an embodiment makes it possible, for example, to limit possibly audible signal limiting artefacts in the high frequencies for the reverberations, this embodiment being accomplished progressively over several blocks. It also makes it possible to consider several forms of transfer functions (denoted below $B_{\mathit{mean}}^k\left(m\right)$

, m being a block index) characterizing a diffuse sound field. Indeed, it is possible for example to apply a transfer function B ^k _mean to a given block mentioned above, and to apply a temporally progressive cut-off window (of the “fade out” type) to this transfer function B ^k _mean , for the block which follows, to “terminate” the presence of the diffuse sound field.

• O^k étant un signal de sortie, et k étant l'indice relatif à un signal de sortie,
• l ∈ [1;L], étant l'indice relatif à un signal d'entrée parmi lesdits signaux d'entrée, L étant le nombre de signaux d'entrée, et I(l) étant un signal d'entrée parmi lesdits signaux d'entrée,
• A^k (l) étant une fonction de transfert avec effet de salle spécifique à un signal d'entrée,
• $B_{\mathit{mean}}^k\left(m\right)$
  
  étant une fonction de transfert globale, avec effet de salle, commune aux signaux d'entrée,
• W^k(l) étant un poids de pondération choisi, et G(I(l)), un gain de compensation d'énergie prédéterminé,
• z^-iDDm étant une application de délai, compté en nombre de blocs d'échantillons, correspondant à un écart temporel entre une émission sonore dans une salle correspondant à l'effet de salle, et un début de présence de champ diffus dans cette salle, l'indice m correspondant à un nombre de blocs d'échantillons de durée correspondant à ce délai, M étant le nombre total de blocs que dure une fonction de transfert dans une représentation temps-fréquence,
• le signe « . » désignant la multiplication,
• le signe «^∗[0f^k (l)]» désignant l'opérateur de convolution sur un nombre de fréquences limité et allant d'une plus basse fréquence à une fréquence maximale f ^k(l) qui est fonction au moins du signal d'entrée d'indice l, et
• le signe « ^∗[0f^k (m)] » désignant l'opérateur de convolution sur un nombre de fréquences limité et allant d'une plus basse fréquence à une fréquence f^k (m) qui est fonction du bloc d'échantillons d'indice m.

In one embodiment where the method is implemented by a sound spatialization module receiving a plurality of input signals and delivering at least two output signals, to deliver each output signal, a room effect transfer function is applied to each input signal,
each of said output signals being given by application of the following formula: $$\begin{array}{c}{O}^k={\displaystyle \sum _{I=1}^I\left(I\left(I\right){\ast }_{\left[0;\dots f^k\left(I\right)\right]}{\mathrm{AT}}^k\left(I\right)\right)}+\\ {\displaystyle \sum _{m=1}^M\left(z^{-\mathit{iDDm}}.{\displaystyle \sum _{I=1}^I\left(G\left(I\left(I\right)\right).\frac{1}{W^k\left(I\right)}.I\left(I\right)\right)}\right){\ast }_{\left[0;\dots ;f^k\left(m\right)\right]}{B}_{\mathit{mean}}^k\left(m\right)}\end{array}$$

• O ^k being an output signal, and k being the index relating to an output signal,
• l ∈ [1; L ], being the index relating to an input signal among said input signals, L being the number of input signals, and I ( l ) being an input signal among said input signals,
• A ^k ( l ) being a transfer function with room effect specific to an input signal,
• $B_{\mathit{mean}}^k\left(m\right)$
  
  being a global transfer function, with room effect, common to the input signals,
• W ^k (l) being a chosen weighting weight, and G ( I ( l )), a predetermined energy compensation gain,
• z ^-iDDm being an application of delay, counted in number of blocks of samples, corresponding to a time difference between a sound emission in a room corresponding to the room effect, and the beginning of the presence of a diffuse field in this room, the index m corresponding to a number of blocks of samples of duration corresponding to this delay, M being the total number of blocks that a transfer function lasts in a time-frequency representation,
• the sign " . denoting multiplication,
• the sign " ^∗ [0 f ^k ( l )]" designating the convolution operator on a limited number of frequencies and going from a lowest frequency to a maximum frequency f ^k ( l ) which is a function at least of the input signal of index l , and
• the sign “ ^∗ [0 f ^k ( m )]” designating the convolution operator on a limited number of frequencies and going from a lowest frequency to a frequency f ^k ( m ) which is a function of the block of samples of index m .

This realization will be described in detail later with reference to the figures 2 and 5 , notably.

It is also possible to apply a limitation of multiplication calculations beyond a first threshold frequency, from the first block or blocks of samples, according to the characteristics of the signal (for example its sampling frequency, or the highest frequency represented in the spectral components of the signal) or according to the spatialization characteristics applied (with for example a limitation of the high frequency components for a contra-lateral acoustic path as detailed below).

In this case, the signal resulting from the reverberations (after reflections or in the diffuse field) does not normally comprise spectral components of higher frequency than the initial signal. Thus, the aforementioned threshold frequency cannot be greater than this highest frequency.

Thus, in more generic terms, in one embodiment, information is obtained on the highest frequency spectral component in the sound signal, and the aforementioned threshold frequency is chosen as being the minimum among a predetermined threshold frequency (for example between 5 and 15 kHz) and said highest frequency.

Typically, in an embodiment where the sound signal comes from a compression decoder, the highest frequency spectral component information can be provided by the decoder.

Similarly, if the spatialization is carried out with a module capable of supporting different signal formats, in particular in terms of sampling frequency of such signals, the highest frequency, mentioned above, cannot be greater than half the sampling frequency, and, thus, the threshold frequency for the implementation of the invention can be chosen in addition according to this sampling frequency.

In an embodiment where the sound signal is spatialized on at least first and second virtual loudspeakers, associated respectively with a first and a second channel, first and second transfer functions with room effect are respectively applied to these first and second channels, as explained above in the introduction (for example by adapting signals on surround channels to switch to binaural or transaural reproduction). In particular, in the case where one of the first and second transfer functions applies an ipsi-lateral acoustic path effect, while the other of the first and second transfer functions applies a contra-lateral acoustic path effect, it is possible to provide for an elimination of the spectral components of the sound signal beyond a given screening frequency. This "screening" frequency is explained by the fact that for a contra-lateral path between a virtual loudspeaker and a considered ear of the listener, the listener's head masks the acoustic path and absorbs the highest tones of the acoustic wave (thus eliminating the spectral components associated with the highest frequencies of the acoustic wave). Thus, the aforementioned threshold frequency, for the transfer function applying a contra-lateral path effect, can be chosen as a minimum from among a predetermined threshold frequency (for example chosen between 5 and 15 kHz) and this screening frequency. This embodiment is advantageous for being applied already for the first block of samples. On the other hand, it does not exclude the possibility of again increasing the threshold frequency for the following block to simulate a first reflection on a wall located opposite the ear in question, this first reflection being received at this ear by an ipsi-lateral path.

It will be understood in any case that the cut-off frequency can be chosen common to all the signals, in one possible embodiment, after a given instant which corresponds for example to the presence of the diffuse field.

• une fonction de transfert spécifique à ce signal, additionnée à
• une fonction de transfert globale, commune à tous les signaux, et représentative d'une présence de champ diffus,

alors l'instant donné précité peut être commun à l'ensemble des signaux et correspondre par exemple à un début de présence de champ sonore diffus.Thus, the realization described in the document FR1357299 introduced above can be advantageous in the context of the invention, and in particular if each transfer function applied to a signal comprises:

• a transfer function specific to this signal, added to
• a global transfer function, common to all the signals, and representative of the presence of a diffuse field,

then the aforementioned given instant can be common to all the signals and correspond for example to the beginning of the presence of a diffuse sound field.

In an embodiment where the signals comprise successive blocks of samples, of the same sizes between signals, at least one given instant is provided for the limitation of the consideration of the frequency components up to a cutoff frequency, this given instant being located temporally at the start of a block different from a first block in a succession of blocks. This given instant therefore occurs after direct propagation, and at the time of sound reflections or the presence of a diffuse field.

This achievement will be detailed later with reference to the figure 5 , also illustrating, in an exemplary embodiment, a possible algorithm of a computer program that would be executed by a processor of a spatialization module operating the method within the meaning of the invention. In this respect, the present invention is also aimed, in general, at a computer program comprising instructions for implementing the above method, when they are executed by a processor.

This disclosure also presents an example of implementation of a sound spatialization module, comprising calculation means for applying at least one room-effect transfer function to at least one input sound signal, said application amounting to multiplying, in the spectral domain, spectral components of the sound signal by the spectral components of a filter corresponding to said transfer function, each spectral component of the filter comprising a temporal evolution in a time-frequency representation. In particular, these calculation means are configured to ignore said spectral components of the filter for said multiplications of components, beyond a threshold frequency and after at least a given instant in said time-frequency representation.

• O^k étant un signal de sortie, et k étant l'indice relatif à un signal de sortie,
• l ∈ [1;L], étant l'indice relatif à un signal d'entrée parmi lesdits signaux d'entrée, L étant le nombre de signaux d'entrée, et I(l) étant un signal d'entrée parmi lesdits signaux d'entrée,
• A^k (l) étant une fonction de transfert avec effet de salle spécifique à un signal d'entrée,
• $B_{\mathit{mean}}^k\left(m\right)$
  
  étant une fonction de transfert globale, avec effet de salle, commune aux signaux d'entrée,
• W^k(l) étant un poids de pondération choisi, et G(I(l)), un gain de compensation d'énergie prédéterminé,
• z^-iDDm étant une application de délai, compté en nombre de blocs d'échantillons, correspondant à un écart temporel entre une émission sonore dans une salle correspondant à l'effet de salle, et un début de présence de champ diffus dans cette salle, l'indice m correspondant à un nombre de blocs d'échantillons de durée correspondant à ce délai, M étant le nombre total de blocs que dure une fonction de transfert dans une représentation temps-fréquence,
• le signe « . » désignant la multiplication,
• le signe « ^∗[0f^k (l)] » désignant l'opérateur de convolution sur un nombre de fréquences limité et allant d'une plus basse fréquence à une fréquence maximale f^k (l) qui est fonction au moins du signal d'entrée d'indice l, et
• le signe « ^∗[0f^k (m)] » désignant l'opérateur de convolution sur un nombre de fréquences limité et allant d'une plus basse fréquence à une fréquence f^k (m) qui est fonction du bloc d'échantillons d'indice m.

The sound spatialization module receiving a plurality of input signals, delivers at least two output signals, the calculation means being configured to apply a room effect transfer function, at each input signal, each of said output signals being given by application of the following formula: $$\begin{array}{c}{O}^k={\displaystyle \sum _{I=1}^I\left(I\left(I\right){\ast }_{\left[0;\dots f^k\left(I\right)\right]}{\mathrm{AT}}^k\left(I\right)\right)}+\\ {\displaystyle \sum _{m=1}^M\left(z^{-\mathit{iDDm}}.{\displaystyle \sum _{I=1}^I\left(G\left(I\left(I\right)\right).\frac{1}{W^k\left(I\right)}.I\left(I\right)\right)}\right){\ast }_{\left[0;\dots ;f^k\left(m\right)\right]}{B}_{\mathit{mean}}^k\left(m\right)}\end{array}$$

• O ^k being an output signal, and k being the index relating to an output signal,
• l ∈ [1; L ], being the index relating to an input signal among said input signals, L being the number of input signals, and I ( l ) being an input signal among said input signals,
• A ^k ( l ) being a transfer function with room effect specific to an input signal,
• $B_{\mathit{mean}}^k\left(m\right)$
  
  being a global transfer function, with room effect, common to the input signals,
• W ^k ( l ) being a chosen weighting weight, and G ( I ( l )), a predetermined energy compensation gain,
• ^{z- iDDm} being an application of delay, counted in number of blocks of samples, corresponding to a time difference between a sound emission in a room corresponding to the room effect, and the beginning of the presence of a diffuse field in this room, the index m corresponding to a number of blocks of samples of duration corresponding to this delay, M being the total number of blocks that a transfer function lasts in a time-frequency representation,
• the sign " . denoting multiplication,
• the sign " ^∗ [0 f ^k ( l )]" designating the convolution operator on a limited number of frequencies and going from a lowest frequency to a maximum frequency f ^k ( l ) which is a function at least of the input signal of index l , and
• the sign “ ^∗ [0 f ^k ( m )]” designating the convolution operator on a limited number of frequencies and going from a lowest frequency to a frequency f ^k ( m ) which is a function of the block of samples of index m .

This module can be integrated into a compression decoding device, or more generally into a restitution system.

We represented on the figure 6 such a spatialization module SPAT, as well as a decoding device DECOD which receives from a network RES, in the example represented, signals coded in compression I′( l ) (with I=1,...,L) and decodes them before restitution by transmitting the decoded signals I( l ) (with I=1,...,L) to the spatialization module. The latter comprises in the example represented an input interface IN to receive the decoded signals, as well as calculation means such as a processor PROC and a working memory MEM cooperating with the interfaces IN / OUT to spatialize the signals I ( I ) and deliver by the output interface OUT only two signals O ^d and O ^g intended to supply the respective earpieces of a CAS helmet.

• la figure 1 illustre une réalisation générale du procédé selon l'invention ;
• la figure 2 illustre un exemple d'application du procédé selon un mode de réalisation où les fonctions de transfert sont sous la forme d'une combinaison de deux fonctions de transfert dont l'une est appliquée avec un retard sur le signal à traiter ;
• la figure 3 représente un exemple de représentation temps-fréquence d'une fonction de transfert avec des fréquences de coupure (ou « fréquences seuils » précitées) variables notamment en fonction du temps ;
• la figure 4 illustre un organigramme correspondant à un algorithme général possible du programme informatique au sens de l'invention,
• la figure 5 représente une réalisation particulière issue du mode représenté sur la figure 2, mais sur plus de deux blocs temporels successifs, avec une évolution de la fonction de transfert $B_{\mathit{mean}}^k\left(m\right)$
  
  représentant le champ diffus, en fonction des blocs m ;
• la figure 6 illustre un exemple de module de spatialisation au sens de l'invention ;
• la figure 7 illustre schématiquement les haut-parleurs virtuels et l'effet de salle pour appliquer une fonction de transfert appropriée, avec limitation des composantes fréquentielles de cette fonction de transfert jusqu'à une fréquence de coupure appropriée.

Other characteristics and advantages of the invention will appear on examination of the detailed description below, and of the appended drawings in which:

• there figure 1 illustrates a general embodiment of the method according to the invention;
• there figure 2 illustrates an example of application of the method according to an embodiment where the transfer functions are in the form of a combination of two transfer functions, one of which is applied with a delay to the signal to be processed;
• there picture 3 represents an example of time-frequency representation of a transfer function with cut-off frequencies (or aforementioned “threshold frequencies”) which vary in particular as a function of time;
• there figure 4 illustrates a flowchart corresponding to a possible general algorithm of the computer program within the meaning of the invention,
• there figure 5 represents a particular realization resulting from the mode represented on the picture 2 , but over more than two successive time blocks, with an evolution of the transfer function $B_{\mathit{mean}}^k\left(m\right)$
  
  representing the diffuse field, as a function of the blocks m ;
• there figure 6 illustrates an example of a spatialization module within the meaning of the invention;
• there figure 7 schematically illustrates the virtual loudspeakers and the room effect to apply an appropriate transfer function, with component limitation frequencies of this transfer function up to an appropriate cut-off frequency.

Before describing the figure 1 and the general principle of the invention, reference is made to the figure 7 to explain the physical phenomena underlying the present invention.

A plurality of virtual loudspeakers surround, in the example represented, the head TE of a listener. Each of the HPV virtual loudspeakers is initially fed by a signal I ( l ) with l ∈ [1; L ] for example previously decoded as indicated above with reference to the figure 6 . The layout of the virtual loudspeakers can relate to a multichannel or also surround sound representation of the signals I ( l ) to be processed in order to restore them together in a spatialized way with a room effect on a headset with CAS earphones ( figure 6 ). For this purpose, a room effect transfer function is usually applied to each signal for each atrial signal to be delivered O ^k , with k=d (for right), g (for left). So, with reference to the figure 7 , we consider for each virtual loudspeaker HPV the acoustic path (ipsi-lateral TIL in the example represented) from the loudspeaker HPV to the left ear OG, as well as the acoustic path (contra-lateral TCL in the example represented) from the loudspeaker HPV to the right ear OD, as well as reflections on the walls MUR (path RIL), and finally a diffuse field after several reflections. With each reflection, it is considered that the acoustic wave is attenuated in the highest frequencies.

So, with reference to the picture 3 relating to a time-frequency representation of a suitable transfer function for the HPV virtual loudspeaker represented on the figure 7 , it already appears that the listener's head naturally masks the contra-lateral path and the highest frequencies to be considered for the transfer function specific to the right ear OD are lower than those to be considered for the transfer function specific to the left ear OG (which is facing the virtual loudspeaker HPV along an ipsi-lateral path). Thus, by considering a first block of time from 0 to N-1, denoted m=0, the maximum frequency F _c ^d (0) of a filter representing the transfer function specific to the right ear can be lower than the maximum frequency F _c ^g (0) of a filter representing the transfer function specific to the left ear. A designer of such a filter can thus limit the components of his filter for the right ear up to the cutoff frequency F _c ^d (0) (corresponding to a head screening frequency) even though the signal to be processed I( l ) may have higher spectral components and up to the frequency F _c ^g (0) at least.

Then, after reflections, the acoustic wave tends to attenuate in the high frequencies, which is well respected by the time-frequency representation of the transfer function for the left ear, as for the right ear, for the instants N to 2N-1, corresponding to the following block noted m=1. Thus, a designer of filters representing these transfer functions can plan to limit the components of the filters for the right ear up to the cutoff frequency F _c ^d (1) and for the left ear up to the cutoff frequency F _c ^g (1). In an embodiment illustrated in particular on the figure 5 , we can consider that in the block m=1, the transfer function typically characterizes a diffuse field for the right ear as for the left ear, and, thus, it can be established (possibly but not limitingly) that F _c ^d (1)=F _c ^g (1).

Then, in the presence of a diffuse field with overall attenuation of the sound ("fade out"), the acoustic wave tends to be attenuated more in the high frequencies, which is still well respected by the time-frequency representation of the transfer function for the left ear, as for the right ear, on the picture 3 , for times 2N to 3N-1, corresponding to the block noted m=2. Thus, a designer of filters representing these transfer functions can plan to limit the components of the filters for the right ear up to the cutoff frequency F _c ^d (2) and for the left ear up to the cutoff frequency F _c ⁸ (2).

It will be noted that shorter blocks would make it possible to vary the highest frequency to be considered more finely, for example to take account of a first RIL reflection for which the highest frequency increases for the right ear (dotted lines around F _c ^d (0) in the picture 3 ) in the first moments of the block m=0.

Thus, it will be noted that it is possible not to take into account all the spectral components of a filter representing a transfer function, in particular beyond a cut-off frequency F _c . Consequently, it is advantageous to treat the application of the function of transfer, in the spectral domain. Indeed, the convolution of a signal I( l ) by a transfer function becomes, in the spectral domain, a multiplication of the spectral components of the signal I( l ) by the spectral components of the filter representing the transfer function in the spectral domain, and, in particular, this multiplication can be operated up to a cutoff frequency only, which is a function of a given block, for example, and of the signal to be processed.

So, with reference to the figure 1 , L input signals I(1), I(2), ..., I(L) are transformed into the frequency domain, respectively at steps TF11, TF12, ..., TF1L. As a variant, such input signals may already be available in frequency form (for example from the decoder).

At step BA11, a complete spatialization impulse response (typically of the BRIR type for “Binaural Room Impulse Response”) in temporal form corresponding to signal I(1) of channel 1 is stored in memory. In step TFA11, this impulse response is transformed into frequency form to obtain a corresponding filter in the spectral domain. In an advantageous embodiment, the filter is stored in its spectral form to avoid repeating the calculation of the transform. This filter is then multiplied with the input signal in frequency form of channel 1 (which is equivalent to a convolution in the time domain). We therefore have the spatialized signal for the signal I(1) of channel 1.

The same operations are carried out for the other L-1 channels. There is thus a total of L spatialized channels. These channels are then summed to obtain a single output signal representing the L channels, and the time domain is returned to step ITF11, to deliver one of the signals O ^k (with k=d,g) supplying an earpiece. A similar processing is carried out for the other auricle. In an embodiment described in detail below with reference to the figures 2 and 5 , the L spatialized channels are not independently accessible before summing: the single output signal is constructed by summing each spatialized channel with the previous output signal as it goes along.

These operations are carried out for each output signal O ^k to be constructed. Typically, if it is a binaural playback, these steps are performed twice, once for the output signal intended to feed a left earpiece of a headset and once for the output signal intended to feed the right earpiece of the headset. Thus, two spatialized signals O ^d and O ^g are finally obtained, each corresponding to an ear.

The L input signals can typically correspond to the L channels of a multi-channel audio content supposed to feed (“virtual”) loudspeakers. The L input signals can for example correspond to the L surround signals of an audio content in surround representation.

Referring now to the figure 2 illustrating an implementation within the meaning of the invention, the principle of a spatialization of L channels as presented on the figure 1 . Nevertheless, the presentation of the picture 2 is simplified in that the L input signals are combined into a single channel I( l ). Thus, L input signals I(1), I(2), ..., I(L) are transformed into the frequency domain at step S21. As indicated previously, such input signals may alternatively already be available in frequency form. At step S22, a spatialization impulse response A ^k (/) (typically of the BRIR type) corresponding to the signal I( 1 ) of the channel / is transformed into the spectral domain to obtain a frequency filter. This impulse response A ^k ( I ) is incomplete in the representation of the picture 2 because it corresponds to a first temporal block of samples m=0. As indicated above, this impulse response may already be available in frequency form. The components of this filter are then multiplied by the spectral signal of the corresponding channel I( l ). This multiplication is parameterized (as shown below with reference to the figure 4 ) so that certain frequency components are ignored, within the meaning of the invention. Typically, the highest frequency components will be ignored to limit the complexity of the calculations. On the figures 2 and 5 , we then note the multiplication of the components limited to a cut-off frequency by the sign: X

A cutoff frequency f _cA(I) from which the frequency components are ignored is defined (for example the maximum frequency represented in the channel signal I( l ), or half of its sampling frequency). Furthermore, this cutoff frequency is specific to each filter and for each block (it decreases for example for the blocks m=1, m=2). As the filters here are specific to each input signal and to each ear, a cutoff frequency is specific to an input signal, to an ear (therefore an output signal) and to a temporal block.

The spatialized signal is then available for the channel l for this first time block. These operations are carried out for all the L channels: l = 1,...,L. There are thus L spatialized channels. These channels are then summed at step S23 to obtain a single signal representing the L channels over the first time block.

In practice, the summation is carried out in a special way, because it takes account of a delay on the channels to characterize the reverberations (reflections and diffuse field), as detailed below. Indeed, in one embodiment, the L spatialized channels are not independently accessible before summing: the single output signal is constructed by summing each spatialized channel with the preceding output signal as it goes along. To this end, in the DBD step, the input signals I( 1 ) are delayed by a certain delay given by z ^-iDD.m specific to each block m=1, ..., M. Note that for the first block, the delay m is zero. In the case of a frequency representation, this delay generally corresponds to the size of a signal frame processed for the first block, and is interpreted as taking the previous input block in its frequency form.

At step S24, an incomplete spatialization impulse response B ^k _m ( l ) (typically of BRIR type) corresponding to signal I( l ) of channel l is transformed into the spectral domain to obtain a frequency filter. This impulse response B ^k _m (/) is incomplete because it corresponds to a second temporal block of samples (then to a third block and so on, for m=1, ..., M). As indicated above, this impulse response may alternatively already be available in frequency form. By applying the principle described in the document FR1357299 , it is possible to reduce the complexity of the processing by setting B ^k _m (1)= ... = B ^k _m ( l ) = ... = B ^k _m (L)= B ^k _mean ( m ) and finally to make this transfer function depend only on the block m considered (main diffuse field, or diffuse field secondary with “fade out” attenuation) and ear k. Similarly, the diffuse field does not depend on the channels and it is possible to fix the cut-off frequency f _c as being identical for each channel (but which can decrease further from one block to the next, as we saw previously with reference to the picture 3 ). This achievement is presented on the figure 5 .

Referring again to the picture 2 , this filter B ^k _m (/) is then multiplied with the signal I( l ) of channel l . The cutoff frequencies are different for this second time block. As presented with reference to picture 3 , measurements show that the high frequencies are more attenuated in the distant temporal blocks (corresponding to diffuse sounds and multiple reverberations). The cutoff frequencies for these distant blocks can therefore be lower than for the first blocks. However, the lower the cut-off frequency, the more limited the number of operations. Thus, the complexity of the calculations is advantageously reduced.

The same operations are carried out for the L channels and the filter multiplication operations are repeated on the progressively delayed spectral signals by summing the contributions at step S25 repeated for each delay m until a single signal representing the L channels is obtained over the set M of the temporal blocks m considered. The single output signal is constructed by gradually summing each spatialized channel with the previous output signal as will now be seen with reference to the figure 4 .

Finally, we go back to the time domain at step S26 to obtain an output signal intended to supply one of the earpieces of the helmet.

With reference to the figure 4 , we now describe a spatialization method for a given time block (for example for the block representing the direct sound field with values in the time interval [0; N-1]) and for a signal corresponding, for example, to the right ear. Of course, the same process is applied for the signal corresponds to the left ear. The distinction between the two ears is introduced by the application of filters specific to each of these ears.

At step S40, the output signal S is initialized to 0. This output signal is expressed in the frequency domain. It has a limited size with a length greater than the cutoff frequency fc( l ). For example, this signal is set to [0; fs( l )/2], fs( l ) being the sampling frequency of this signal I( l ). A first count variable l is also initialized to 1. This first count variable identifies one of the channel signals I(1), I(2), ..., I( l ), ..., I(L) on the temporal block [0; N-1] for the right ear. At step S41, a second count variable j is initialized to 0. This second count variable identifies a frequency component of a signal I( 1 ) on the temporal block [0; N-1] for the right ear.

At step S42, the coefficient c _BRIR (j; l ) is stored. This coefficient corresponds to the frequency component j of the BRIR( l ) filter on the temporal block [0; N-1] for the right ear. Similarly, the coefficient c _I (j; l ) is stored. This coefficient corresponds to the frequency component j of the signal I( l ) on the temporal block [0; N-1] for the right ear. Thus, the coefficients c _BRIR (j; l ) and c _I (j; l ) correspond to the same frequency component (identified by the variable j) and can thus be subsequently multiplied term by term (step S44).

In test T47, it is checked that the frequency corresponding to variable j is lower (for example strictly) than the cut-off frequency fc(/). This cutoff frequency corresponds to the cutoff frequency of the signal I( l ) for the temporal block [0; N-1] for the right ear. If frequency j is lower than cutoff frequency fc(/), step S44 proceeds.

At step S44, a value MULT(j) corresponding to the multiplication of the coefficients c _BRIR (j; l ) and c _i (j; 1) is calculated. These coefficients are indeed multiplied term by term because they correspond to the same frequency component j (for the same channel, on the same block and for the same ear).

At step S45, this value MULT(j) is incremented at signal S at the position of frequency j.

Thus, we proceed to a step-by-step construction of a signal S which includes (at the end of the loop of length fc(/)) all the frequency components up to the cutoff frequency fc(t) (for this signal I( l ) on the block [0; N-1] and for a right ear). As at the beginning of the loop of the figure 4 , we already have all the components initialized to 0, at the end of the loop, we have finally filled a buffer (initially null) up to the cutoff frequency to successively build the signal S. Thus, each multiplication MULT(j) of coefficients is added step by step to the signal S under construction.

At step S46, variable j is incremented and return to step S42. If the variable j is greater than (for example or equal to) the cut-off frequency fc(/), test T48 is passed on to. Thus, the signal S has been filled over the interval [0; fc( l )].

As specified above, this signal can be defined on an interval larger than [0; fc(/)] (e.g. [0; fs( l )/2]). Moreover, this signal had been initialized to 0 over its entire definition interval. Therefore, it is zero on the rest of the interval which has not been filled (for example [fc( l ); fs( l )/2]). The complexity is therefore improved here because steps of filling the signal S have not been carried out, which reduces the number of calculations necessary.

In test T48, it is checked that the counting variable / corresponding to the signal I( 1 ) of the channel / is less (for example strictly) than the number L of channels. If the variable / is less than or equal to L, the variable / is incremented at step S49 and the process is resumed at step S41. If the variable / is greater than L, the signal S corresponding to the spatialized signal for the temporal block [0; N-1] for the right ear is available at step S50.

This signal S corresponding to the temporal block [0; N-1] is then summed with the other signals generated in a similar way for other time blocks [N; 2N-1], [2N; 3N-1], etc., (and for which an appropriate delay has been applied according to the above DBD step of the figure 2 For example).

• une fréquence maximale de champ diffus Fc (diffus) comme illustrée sur la figure 3 décrite ci-avant (choisie par exemple entre 10 à 15 kHz pour le bloc m=1 et entre 5 à 10 kHz pour le bloc m=2), et
• la fréquence maximale fmax représentée dans chaque signal d'entrée (par exemple sa fréquence d'échantillonnage ou la fréquence maximale dont la composante spectrale n'est pas nulle, cette valeur étant habituellement donnée par un décodeur en compression).

Typically, to construct the block [N; 2N-1], a filter is applied in the frequency domain corresponding to a transfer function common to all the input signals I( l ), representing the diffuse field, with a cut-off frequency fc in the multiplication of the spectral components which corresponds to the minimum between:

• a maximum diffuse field frequency Fc (diffuse) as illustrated in the picture 3 described above (chosen for example between 10 to 15 kHz for the block m=1 and between 5 to 10 kHz for the block m=2), and
• the maximum frequency fmax represented in each input signal (for example its sampling frequency or the maximum frequency whose spectral component is not zero, this value usually being given by a compression decoder).

It should be noted that the frequency multiplication by stopping at a given cut-off frequency (which mathematically amounts to multiplying by 0 beyond) is not trivial for those skilled in the art. Indeed, in the context of filtering an audio signal, this type of very violent low-pass filter generally brings audible artifacts (known as "aliasing"), due to echo or pre-echo phenomena originating from the temporal aliasing generated by circular convolution, which it is generally desirable to avoid. However, in the context of the invention, this low-pass filter is not applied to the audio signal, but to the BRIR filter (which is itself convoluted to the audio signal) which is already composed of multiple reflections; the artefacts produced will therefore, at worst, be perceived as additional reflections of the original BRIR filter, and in practice rarely perceptible. It is however possible to attenuate these artefacts by slightly modifying the frequencies of the filter preceding the cutoff frequency (for example by soft attenuation by applying a half Hanning window (fade out type)).

In general, with reference to the figure 4 , it will be noted that two operations are carried out in the same loop instance (typically a clock stroke): the multiplication MULT(k) and its addition to the output signal S. This notably makes it possible to implement this method on processors which have the possibility of carrying out several operations on the same loop instance (typically a clock stroke) and thus of reducing the time necessary for the calculations.

We have illustrated on the figure 5 a complete algorithmic form of the processing, conforming to the formula giving an output signal O ^k presented above: $$\begin{array}{c}{O}^k={\displaystyle \sum _{I=1}^I\left(I\left(I\right){\ast }_{\left[0;\dots f^k\left(I\right)\right]}{\mathrm{AT}}^k\left(I\right)\right)}+\\ {\displaystyle \sum _{m=1}^M\left(z^{-\mathit{iDDm}}.{\displaystyle \sum _{I=1}^I\left(G\left(I\left(I\right)\right).\frac{1}{W^k\left(I\right)}.I\left(I\right)\right)}\right){\ast }_{\left[0;\dots ;f^k\left(m\right)\right]}{B}_{\mathit{mean}}^k\left(m\right)}\end{array}$$

As indicated above, the weightings W ^k ( l ) and the gains G(I( l )) can be fixed at 1. We have not represented on the figure 5 the gains G(I( l )) because this figure should be read as an integration of the gains at the weights 1/W ^k ( l ). Moreover, when designing filters, these two parameters are determined, fixed and multiplied together once and for all.

Claims (8)

1. Method for achieving audio spatialization, comprising the application of at least one room-effect transfer function to at least one audio signal, said application amounting to multiplying, in the spectral domain, spectral components of the audio signal by spectral components of a filter corresponding to said transfer function, each spectral component of the filter comprising a variation as a function of time in a time-frequency representation,
   in which method said spectral components of the filter are ignored, in said multiplications of components, beyond a threshold frequency and after at least one given time in said time-frequency representation and in which method, in an implementation by a module for achieving audio spatialization that receives a plurality of input signals and that delivers at least two output signals, to deliver each output signal, a room-effect transfer function is applied to each input signal, the method being characterized in that each of said output signals is given by application of the following formula: $$\begin{array}{c}{O}^k={\displaystyle \sum _{l=\mathrm{1}}^L\left(I\left(l\right){\ast }_{\left[\mathrm{0};\dots ;f^k\left(l\right)\right]}{A}^k\left(l\right)\right)}\\ +{\displaystyle \sum _{m=1}^M\left(\begin{array}{l}{z}^{-\mathit{iDDm}}\\ \cdot {\displaystyle \sum _{l=1}^L\left(G\left(I\left(l\right)\right)\cdot \frac{\mathrm{1}}{W^k\left(l\right)}\cdot I\left(l\right)\right)}\end{array}\right){\ast }_{\left[\mathrm{0};\dots ;f^k\left(m\right)\right]}{B}_{\mathit{mean}}^k\left(m\right)}\end{array}$$

   

   - O^k being an output signal, and k being the index relative to an output signal,

   - l ∈ [1;L], being the index relative to one input signal among said input signals, L being the number of input signals, and I(l) being one input signal among said input signals,

   - A^k (l) being a transfer function with room effect specific to one input signal,

   - $B_{\mathit{mean}}^k\left(m\right)$

   

   being an overall transfer function, with room effect, common to the input signals,

   - W^k (l) being a selected weighting weight, and G(I(l)) being a predetermined energy compensation gain,

   - z^-iDDm being an application of a delay, counted in number of blocks of samples, corresponding to a time difference between an audio emission in a room corresponding to the room effect and an onset of presence of a diffuse field in this room, the index m corresponding to a number of blocks of samples of duration corresponding to this delay, M being the total number of blocks that a transfer function lasts in a time-frequency representation,

   - the sign "·" designating multiplication,

   - the sign "^∗[0;...;f^k (l)]" designating convolution over a limited number of frequencies ranging from a lowest frequency to a maximum frequency f^k(l) that is dependent at least on the input signal of index l, and

   - the sign "^∗[0;...;f^k (m)]" designating convolution over a limited number of frequencies ranging from a lowest frequency to a frequency f^k (m) that is dependent on the block of samples of index m.
2. Method according to Claim 1, wherein the threshold frequency decreases as a function of time in said time-frequency representation.
3. Method according to one of the preceding claims, wherein information is obtained on the spectral component of highest frequency in the audio signal, and wherein said threshold frequency is a minimum among a predetermined threshold frequency and said highest frequency.
4. Method according to Claim 3, wherein the audio signal is generated by a compression decoder and the information on the spectral component of highest frequency is delivered by the decoder.
5. Method according to one of Claims 3 and 4, wherein the audio signal is sampled at a given sampling frequency, said threshold frequency being chosen depending on said sampling frequency.
6. Method according to one of the preceding claims, wherein the audio signal is spatialized via at least a first virtual loudspeaker and a second virtual loudspeaker that are associated with a first channel and a second channel, respectively, and first and second transfer functions with room effect are applied to said first and second channels, respectively,

   one of the first and second transfer functions applying an ipsi-lateral acoustic pathway effect, and the other of the first and second transfer functions applying a contra-lateral acoustic pathway effect with removal of spectral components above a given screening frequency from the audio signal,

   and wherein said threshold frequency for the transfer function applying a contra-lateral pathway effect is a minimum among a predetermined threshold frequency and said screening frequency.
7. Method according to Claim 1, wherein the signals comprise successive blocks of samples, of same sizes between signals, and wherein said at least one given time is located temporally at the start of a block other than a first block in a succession of blocks.
8. Computer program, containing instructions for implementing the method according to one of the preceding claims, when they are executed by a processor.

Images