EP1994526B1 - Joint sound synthesis and spatialization - Google Patents

Abstract

The invention concerns a process for joint synthesis and spatialization of multiple sound sources in associated spatial positions, including: a) a step of assigning to each source at least one parameter (pi) representing an amplitude; b) a step of spatialization consisting in implementing an encoding into a plurality of channels, wherein each amplitude (pi) is duplicated to be multiplied to a specialization gain (gim), each spatialization gain being determined for one encoding channel (pgm) and for a source to be spatialized (Si); c) a step of grouping (R) the parameters multiplied by the gains (Pim), in respective channels (pg1, . . . , pgM), by applying a sum of said multiplied parameters (pim) on all the sources (Si) for each channel (pgm), and d) a step of parametric synthesis (SYNTH(I), . . . , SYNTH(M)) applied to each of the channels (pgm).

Description

The present invention relates to audio processing and, more particularly, to three-dimensional spatialization of synthetic sound sources.

Currently, the spatialization of a synthetic sound source is often performed without taking into account the sound production mode, that is to say, the way in which the sound is synthesized. Thus, many models, including parametric, have been proposed for synthesis. At the same time, many spatialization techniques have also been proposed, without proposing a cross-check with the technique chosen for a synthesis.

Among the synthetic techniques, the so-called "non- parametric" methods are known . No particular parameter is used a priori to modify samples previously stored in memory. The best-known representative of these methods is classical wave table synthesis.

This type of technique is opposed by " parametric " synthesis methods, which rely on the use of a model that makes it possible to manipulate a small number of parameters, compared to the number of signal samples produced in the sense of the non-parametric methods. Parametric synthesis techniques typically rely on additive, subtractive, source / filter or non-linear models.

Among these parametric methods, those which make it possible to manipulate parameters corresponding to different sound sources, so as to use only one synthesis process, nevertheless for all sources, are called " mutuals ". In so-called " sinusoidal " methods , typically, a frequency spectrum is constructed from parameters such as the amplitude and frequency of each component. partial of the overall sound spectrum of the sources. Indeed, an implementation by inverse Fourier transform, followed by an addition / overlap, ensures an extremely efficient synthesis of several sound sources simultaneously.

Concerning the spatialization of sound sources, different techniques are currently known. Some techniques (such as " transaural " or " binaural ") are based on the consideration of HRTFs (" Head Related Transfer Function ") transfer functions representing the disturbance of acoustic waves by the morphology of an individual, these HRTFs functions being specific to this individual. The sound reproduction is done in a manner adapted to the HRTFs of the listener, typically on two remote speakers (" transaural ") or from the two earpieces of a headset (" binaural ") . Other techniques (for example " surround " or "multichannel" (5.1 to 10.1 or more) rather provide a restitution on more than two speakers.

More specifically, some HRTFs techniques use the separation of the frequency and position variables of the HRTFs, thus giving a set of p basic filters (corresponding to the first p eigenvalues of the covariance matrix of the HRTFs whose variables statistics are the frequencies), these filters being weighted by spatial functions (obtained by projection of the HRTFs on basic filters). The spatial functions can then be interpolated, as described in the document US 5500900 .

Spatialization of many sound sources can be achieved through a multichannel implementation applied to the signal of each of the sound sources. The gains of the spatialization channels are applied directly to the sound samples of the signal, often described in the time domain (but possibly also in the frequency domain). These samples sound are treated by a spatialization algorithm (with application of gains that depend on the desired position), regardless of the origin of these samples. Thus, the proposed spatialization could apply to both natural and synthetic sounds.

On the one hand, each sound source must be synthesized independently (with a temporal or frequency signal), in order to then be able to apply independent spatialization gains. For N sound sources, it is therefore necessary to perform N synthesis calculations.
On the other hand, the application of the gains to sound samples, whether they come from the time or frequency domain, requires at least as many multiplications as there are samples. For a block of Q samples, it is therefore necessary to apply at least NMQ gains, M being the number of intermediate channels (surround channels for example) and N being the number of sources.
Thus, this technique requires a high calculation cost in the case of the spatialization of many sound sources.

Among the ambiophonic techniques, the so-called " virtual loudspeakers " method makes it possible to encode the signals to be spatialized by applying them in particular gains, the decoding being done by convolution of the signals encoded by pre-calculated filters (Jérôme Daniel, " Representation of acoustic fields, application to the transmission and reproduction of complex sound scenes in a multimedia context " , PhD Thesis, 2000).

A very promising technique, combining synthesis and spatialization, was presented in the document WO-05/069272 .

It consists in determining amplitudes to be assigned to signals representing sound sources to define, at the same time, the loudness (by example a " volume ") of a source to synthesize and a gain of spatialization of this source. This document notably discloses a binaural spatialization taking into account delays and gains (or " spatial functions ") and, in particular, a mixing of the sources synthesized in the encoding part of spatialization.

More particularly, an example embodiment that is referred to in this document WO-05/069272 and in which the sources are synthesized by associating amplitudes with frequencies constituting a " sound timbre " (for example a fundamental frequency and its harmonics), provides for grouping by identical frequencies synthesis signals, with a view to spatialization subsequent operating on the frequencies.

This exemplary embodiment is illustrated on the figure 1 . In a synthesis block SYNTH (represented in dashed lines), at frequencies f ₀ , f ₁ , f ₂ ,..., F _p of each source to be synthesized S ₁ ,..., S _N, respective amplitudes are assigned. a ₀ ¹ , a ₁ ¹ , ..., a _p ¹ , ..., a _i ^j , ..., a ₀ ^N , a ₁ ^N , ..., a _p ^N , where in the general notation a _i ^j , j is a source index between 1 and N and i is a frequency index between 0 and p. Of course, certain amplitudes of a set at ₀ ^j , a ₁ ^j , ..., a _p ^j to be assigned to a same source j can be zero if the corresponding frequencies are not represented in the sound signal of this source j .
The amplitudes a _i ¹ , ..., a _i ^N relative to each frequency f _i are grouped (" mixed ") to be applied, frequency by frequency, to the SPAT spatialization block for a frequency-based encoding (in binaural by example, then providing an interaural delay to apply to each source). The signals of the channels c ₁ ,..., C _k , originating from the spatialization block SPAT, are then intended to be transmitted through one or more networks, or else stored, or other, for the purpose of a subsequent restitution (preceded by where appropriate, a suitable spatialization decoding).

This technique, although very promising, still deserves some optimizations.

In general, current methods require significant computing power to spatialize many synthesized sound sources.

The present invention improves the situation.

1. a) une étape d'affectation à chaque source d'au moins un paramètre de synthèse p_i, représentatif d'une amplitude d'au moins une composante fréquentielle de la source,
2. b) une étape de spatialisation mettant en oeuvre un encodage en une pluralité de canaux, dans laquelle on duplique chaque paramètre d'amplitude pour le multiplier par un gain de spatialisation, chaque gain de spatialisation étant déterminé, d'une part, pour un canal d'encodage et, d'autre part, pour une source à spatialiser,
3. c) une étape de regroupement des paramètres multipliés par les gains, dans des canaux respectifs, en appliquant une somme desdits paramètres multipliés sur toutes les sources pour chaque canal, et
4. d) une étape de synthèse paramétrique appliquée à chacun des canaux.

it proposes for this purpose a method for synthesizing and spatially co-ordinating a plurality of sound sources in associated positions of the space, the method comprising:

1. a) a step of assigning to each source at least one synthesis parameter p _i , representative of an amplitude of at least one frequency component of the source,
2. b) a spatialization step implementing an encoding in a plurality of channels, in which each amplitude parameter is duplicated to multiply it by a spatialization gain, each spatialization gain being determined, on the one hand, for a channel encoding and, on the other hand, for a source to be spatialised,
3. c) a step of grouping the parameters multiplied by the gains, in respective channels, by applying a sum of said multiplied parameters on all the sources for each channel, and
4. d) a parametric synthesis step applied to each of the channels.

There is also provided a computer program according to claim 6 and a module according to claim 7.

Thus, the present invention proposes for this purpose to first apply a spatialization encoding, then a " pseudo-synthesis " , the term " pseudo " aiming at the fact that the synthesis applies in particular to the encoded parameters, derived from spatialization and not to usual synthetic sound signals.

Indeed, a feature that the invention proposes is the spatial encoding of some synthesis parameters, rather than performing a spatial encoding of the signals corresponding directly to the sources. This spatial encoding applies more particularly to synthesis parameters which are representative of an amplitude and it advantageously consists in applying to these few synthesis parameters spatialization gains which are calculated as a function of respective desired positions of the sources. It will thus be understood that the parameters multiplied by the gains in step b) and grouped in step c) are not really sound signals, as in the general prior art described above.

The present invention then uses a mutual parametric synthesis where one of the parameters has the dimension of an amplitude. Unlike techniques of the prior art, it thus takes advantage of the advantages of such a synthesis to perform the spatialization. The combination of synthesis parameter sets obtained for each of the sources advantageously makes it possible to globally control the encoded blocks of mutual parametric synthesis.

The present invention then makes it possible to spatialize simultaneously and independently of numerous synthesized sound sources from a parametric synthesis model, the spatialization gains being applied to the synthesis parameters rather than to the samples of the time or frequency domain. This embodiment thus ensures a substantial saving of the computing power required because it implies a low calculation cost.

According to one of the advantages provided by the invention, since the number of steps in the synthesis is made independent with respect to the number of sources, only one intermediate channel synthesis can be applied. Regardless of the number of sound sources, only a constant number M of synthesis calculations is planned. Typically, since the number of sources N becomes larger than the number M of intermediate channels, the technique in the sense of the invention requires fewer calculations than the usual techniques in the sense of the prior art. For example, at the surround order 1 and in two dimensions (ie three intermediate channels), the invention already allows a calculation gain for only four sources to spatialize.

The present invention also makes it possible to reduce the number of gains to be applied. Indeed, the gains are applied to the synthesis parameters and not to the sound samples. Updating parameters such as the volume is generally less frequent than the sampling frequency of a signal, a calculation economy is thus achieved. For example, for a parameter update frequency (such as the volume in particular) of 200 Hz, a substantial saving in multiplication is achieved for a signal sampling frequency of 44100 Hz (in a ratio of about 200).

The fields of application of the present invention may concern both the musical field (including polyphonic ringtones of mobiles), the field of multimedia (including video game sound systems), the field of virtual reality (rendering of sound scenes). , simulators (synthesis of engine noise), or others.

• la figure 2 illustre le traitement général de spatialisation et synthèse prévu dans un procédé au sens de l'invention,
• la figure 3 illustre un traitement des signaux spatialisés et synthétisés, pour un décodage spatial en vue d'une restitution,
• la figure 4 illustre un mode de réalisation particulier dans lequel on affecte plusieurs paramètres d'amplitude à chaque source, chaque paramètre étant associé à une composante fréquentielle,
• la figure 5 illustre les étapes d'un procédé au sens de l'invention, et peut correspondre à un organigramme d'un programme d'ordinateur pour la mise en oeuvre de l'invention.

Other features and advantages of the invention will appear on examining the detailed description below, and the accompanying drawings in which, in addition to the figure 1 relating to the prior art and described above:

• the figure 2 illustrates the general treatment of spatialisation and synthesis provided for in a process within the meaning of the invention,
• the figure 3 illustrates a processing of the spatialized and synthesized signals, for spatial decoding with a view to restitution,
• the figure 4 illustrates a particular embodiment in which several amplitude parameters are assigned to each source, each parameter being associated with a frequency component,
• the figure 5 illustrates the steps of a method in the sense of the invention, and may correspond to a flowchart of a computer program for the implementation of the invention.

• p₁g₁ ¹, ..., p_ig_i ¹, ..., p_Ng_N ¹ regroupés dans un premier canal de spatialisation p_g ¹,
  et ce, jusqu'à :

• p₁g₁ ^M, ..., p_ig_i ^M, ..., p_Ng_N ^M regroupés dans un M^ième canal de spatialisation p_g ^M,

la lettre g de l'indice désignant le terme "global". With reference to the figure 2 at least one parameter p _i , representative of an amplitude, is assigned to a source S _{i out} of a plurality of sources S ₁ , ..., S _N to be synthesized and spatialized (i being between 1 and N). Each parameter p _i is duplicated in as many spatialization channels provided in the spatialization block SPAT. In the example shown, where M encoding channels are provided for the spatialization, each parameter p _i is duplicated M to apply respective spatialization gains g _i ¹ , g _i ^M (i being, as a reminder, a subscript from source S _i ).
We then obtain NM parameters each multiplied by a gain: p ₁ g ₁ ¹ , ..., P ₁ g ₁ ^M , ..., p _i g _i ¹ , ..., p _i g _i ^M , ... , p _N g _N ¹ , ..., p _N g _N ^M.
We then regroup these multiplied parameters (reference R of the figure 2 ) by spatialization channels (M channels in all), that is:

• p ₁ g ₁ ¹ , ..., p _i g _i ¹ , ..., p _N g _N ¹ grouped in a first spatialization channel p _g ¹ ,
  until:

• p ₁ g ₁ ^M , ..., p _i g _i ^M , ..., p _N g _N ^M grouped in an M ^th spatialization channel p _g ^M ,

the letter g of the index designating the term " global " .

Thus, new parameters p _i ^m (i varying from 1 to N and m varying from 1 to M) are calculated by multiplying the parameters p _i by the encoding gains g _i ^m , obtained from the position of each of the sources. The parameters p _i ^m are combined (by summation in the example described) to provide the parameters p _g ^m which feed M mutual parametric synthesis blocks. These M blocks (referenced SYNTH (1) to SYNTH (M) on the figure 2 ) are constitutive SYNTH synthesis module, which delivers M time signals or frequency ss ^m (m ranging from 1 to M), obtained by synthesis from the parameters p _g ^m . These signals ss ^m can then feed a conventional block of spatial decoding, as will be seen later with reference to the figure 3 .

In a particular embodiment, the synthesis used is an additive synthesis with application of an inverse Fourier transform (IFFT).

où p_i,k est l'amplitude de la composante de fréquence f_i,k et dont la phase est donnée par ϕ_i,k pour la source S_i, à l'instant n.
Il est possible de réaliser la synthèse additive dans le domaine fréquentiel à partir des seuls paramètres p_i,k, f_i,k et ϕ_i,k donnés, en utilisant par exemple la technique exposée dans le document FR-2 679 689 .
Le paramètre p_i,k représente l'amplitude d'une composante fréquentielle k donnée pour une source S_i donnée. On en déduit donc les paramètres p^m _i,k pour chaque source et chacun des M canaux grâce à la relation : $${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i},\mathrm{k}}={{\mathrm{g}}^{\mathrm{m}}}_{\mathrm{i}-}{\mathrm{p}}_{\mathrm{i},\mathrm{k}},$$

m variant de 1 à M. Les gains g^m _i sont prédéterminés pour une position désirée pour la source S_i et en fonction de l'encodage de spatialisation choisi.For this purpose, a set of N sources is characterized by a plurality of parameters p _{i, k} representing the amplitude in the frequency domain of the k ^th frequency component for the i ^th source S _i .
The time signal s _i (n) which corresponds to this source S _i , if it were synthesized independently of the other sources, would be given by: $${\mathit{s}}_{\mathit{i}}\left(\mathit{not}\right)\mathit{=}\underset{\mathit{k}\mathit{=}\mathit{1}}{\overset{\mathit{K}}{\mathit{\Sigma }}}{\mathit{vs}}_{\mathit{i}\mathit{,}\mathit{k}}\left(\mathit{not}\right)\mathit{,}{\mathit{with\; c}}_{\mathit{i}\mathit{,}\mathit{k}}\left(\mathit{not}\right)\mathit{=}{\mathit{p}}_{\mathit{i}\mathit{,}\mathit{k}}\left(\mathit{not}\right)\mathit{<figure-callout\; id="2"\; label="cos"\; filenames="imgf0001.png"\; state="\{\{state\}\}">cos</figure-callout>}\left[\mathrm{2}{\mathit{\pi f}}_{\mathit{i}\mathit{,}\mathit{k}}\left(\mathit{not}\right)\mathit{not}\mathit{/}{\mathit{F}}_{\mathit{e}}\mathit{+}{\mathit{\phi }}_{\mathit{i}\mathit{,}\mathit{k}}\left(\mathit{not}\right)\right]$$

where p _{i, k} is the amplitude of the frequency component f _{i, k} and whose phase is given by φ _{i, k} for the source S _i , at time n.
It is possible to carry out the additive synthesis in the frequency domain from the only given parameters p _{i, k} , f _{i, k} and φ _{i, k} , using for example the technique described in the document FR-2,679,689 .
The parameter p _{i, k} represents the amplitude of a given frequency component k for a given source S _i . We thus deduce the parameters p ^m _{i, k} for each source and each of the M channels thanks to the relation: $${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i},\mathrm{k}}={{\mathrm{boy\; Wut}}^{\mathrm{m}}}_{\mathrm{i}-}{\mathrm{p}}_{\mathrm{i},\mathrm{k}},$$

m varies from 1 to M. The gains g ^m _i are predetermined for a desired position for the source S _i and according to the chosen spatialization encoding.

• Y_m est un harmonique sphérique d'ordre m,
• θ_i et δ_i sont respectivement l'azimut et le site souhaités pour la source S_i.

In the case of an ambiophonic encoding for example, these gains correspond to spherical harmonics and can be written g ^m _i = Y _m (θ _i , δ _i ), where:

• Y _m is a spherical harmonic of order m,
• θ _i and δ _i are respectively the azimuth and the desired site for the source S _i .

• ${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{g}\mathrm{,}\mathrm{kʹ}}\mathrm{=}{{\displaystyle \mathrm{\sum }_{i\mathrm{=}\mathrm{1}}^N}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i}\mathrm{,}\mathrm{k}}}_{},$
  
  , où k' décrit toutes les fréquences f_i,k présentes dans toutes les sources S_i.

En pratique, la valeur de k' est inférieure à k.i car des fréquences communes peuvent caractériser plusieurs sources à la fois. Dans une réalisation, il peut être prévu d'associer un même jeu global de fréquences à toutes les sources, quitte à ce que certains paramètres d'amplitude pour certaines fréquences de sources soient nuls.
Dans ce cas, les valeurs de k et k' sont égales et la relation précédente s'écrit simplement: $${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{g}\mathrm{,}\mathrm{k}}\mathrm{=}{{\displaystyle \mathrm{\sum }_{i\mathrm{=}\mathrm{1}}^N}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i}\mathrm{,}\mathrm{k}}}_{}\mathrm{.}$$

L'étape de synthèse consiste à utiliser ces paramètres p^m _g,k (m variant de 1 à M) pour synthétiser chacun des M spectres en fréquence ss^m(ω) issus du module de synthèse SYNTH. Il peut être prévu à cet effet d'appliquer la technique décrite dans FR-2 679 689 , en ajoutant itérativement des enveloppes spectrales correspondant à la transformée de Fourier d'une fenêtre temporelle (par exemple de Hanning), ces enveloppes spectrales étant précédemment échantillonnées, tabulées, centrées aux fréquences f_k et pondérées alors par p^m _g,k, ce qui s'écrit :

• ${\mathrm{SS}}^m\left(\omega \right)={\displaystyle \sum _{k=1}^K}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{g}\mathrm{,}\mathrm{k}}\cdot {\mathit{env}}_k\left(\omega \right),$
  
  où env_k(ω) est l'enveloppe spectrale centrée à la fréquence f_k.

The parameters p ^m _{i, k} are then combined frequency by frequency, so as to obtain a single global parameter:

• ${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{boy\; Wut}\mathrm{,}\mathrm{k\; \text{'}}}\mathrm{=}{{\displaystyle \underset{i\mathrm{=}\mathrm{1}}{\overset{\mathrm{NOT}}{\mathrm{\Sigma }}}}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i}\mathrm{,}\mathrm{k}}}_{},$
  
  , where k 'describes all the frequencies f _{i, k} present in all the sources S _i .

In practice, the value of k 'is less than ki because common frequencies can characterize several sources at once. In one embodiment, it may be provided to associate the same global set of frequencies to all sources, even if certain amplitude parameters for certain source frequencies are zero.
In this case, the values of k and k 'are equal and the preceding relation is written simply: $${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{boy\; Wut}\mathrm{,}\mathrm{k}}\mathrm{=}{{\displaystyle \underset{i\mathrm{=}\mathrm{1}}{\overset{\mathrm{NOT}}{\mathrm{\Sigma }}}}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i}\mathrm{,}\mathrm{k}}}_{}\mathrm{.}$$

The synthesis step consists of using these parameters p ^m _{g, k} (m varying from 1 to M) to synthesize each of the M frequency spectra ss ^m (ω) from the SYNTH synthesis module. It may be provided for this purpose to apply the technique described in FR-2,679,689 , iteratively adding spectral envelopes corresponding to the Fourier transform of a time window (for example Hanning), these spectral envelopes being previously sampled, tabulated, centered at frequencies f _k and then weighted by p ^m _{g, k} , ce which is written:

• ${\mathrm{SS}}^m\left(\omega \right)={\displaystyle \underset{k=1}{\overset{K}{\Sigma }}}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{boy\; Wut}\mathrm{,}\mathrm{k}}\cdot {\mathit{ca.}}_k\left(\omega \right),$
  
  where env _k (ω) is the spectral envelope centered at the frequency f _k .

This achievement is illustrated on the figure 4 . We assign K amplitude parameters p _{i, k} to each source S _i . The index i, of source, is between 1 and N. The index k, of frequency, is between 1 and K. For each source S _i , these K parameters are duplicated, M times, to be multiplied each by a gain of spatialization g _i ^m . The index m, of spatialization encoding channel, is between 1 and M.

où k varie de 1 à K dans chaque canal m, et m varie globalement de 1 à M.
On comprendra ainsi que dans chaque canal m, il est prévu des sous-canaux p^m _g,k associés chacun à une composante fréquentielle k, l'indice g désignant, pour rappel, le terme "global".
Le traitement se poursuit alors en multipliant le paramètre global de chaque sous-canal p^m _g,k associé à une fréquence f_k par une enveloppe spectrale env_k(ω) centrée en cette fréquence f_k, et ce, pour tous les K sous-canaux (k compris entre 1 et K), et globalement, pour tous les M canaux (m étant compris entre 1 et M). Ensuite, les K sous-canaux sont sommés dans chaque canal m, conformément à la relation ci-après :

• ${\mathrm{SS}}^m\left(\omega \right)={\displaystyle \sum _{k=1}^K}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{g}\mathrm{,}\mathrm{k}\cdot }{\mathit{env}}_k\left(\omega \right),$
  
  pour m allant de 1 à M canaux au total. On obtient alors les signaux ss^m(ω) encodés pour leur spatialisation et synthétisés au sens de l'invention. Ils sont exprimés dans le domaine fréquentiel.

In each channel m, the frequency of the K results of the products g _i ^m .p _{i, k} is grouped, frequency by frequency, according to the expression given above: $${{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{boy\; Wut}\mathrm{,}\mathrm{k}}\mathrm{=}{\displaystyle \underset{i=1}{\overset{\mathrm{NOT}}{\Sigma }}}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i}\mathrm{,}\mathrm{k}}\mathrm{,}\mathrm{with}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{i}\mathrm{,}\mathrm{k}}={{\mathrm{boy\; Wut}}^{\mathrm{m}}}_{\mathrm{i}}\cdot {\mathrm{p}}_{\mathrm{i}\mathrm{,}\mathrm{k}}\mathrm{,}$$

where k varies from 1 to K in each channel m, and m varies globally from 1 to M.
It will thus be understood that in each channel m, there are sub-channels p ^m _{g, k} each associated with a frequency component k, the index g designating, as a reminder, the term " global " .
The processing then continues by multiplying the global parameter of each subchannel p ^m _{g, k} associated with a frequency f _k by a spectral envelope env _k (ω) centered at this frequency f _k , and this, for all the K under -channels (k between 1 and K), and globally, for all M channels (m being between 1 and M). Then the sub-channels K are summed in each channel m, according to the following relation:

• ${\mathrm{SS}}^m\left(\omega \right)={\displaystyle \underset{k=1}{\overset{K}{\Sigma }}}{{\mathrm{p}}^{\mathrm{m}}}_{\mathrm{boy\; Wut}\mathrm{,}\mathrm{k}\cdot }{\mathit{ca.}}_k\left(\omega \right),$
  
  for m ranging from 1 to M channels in total. The signals ss ^m (ω) encoded for their spatialization and synthesized in the sense of the invention are then obtained. They are expressed in the frequency domain.

Le traitement par trames successives peut être réalisé par une technique classique d'addition/recouvrement.To bring back these M signals in the time domain (then noted SS ^m (n)), we can then apply them an inverse Fourier transform (IFFT): $${\mathrm{SS}}^{\mathrm{m}}\left(\mathrm{not}\right)=\mathit{IFFT}\left({\mathrm{ss}}^{\mathrm{m}}\left(\omega \right)\right)$$

The treatment by successive frames can be achieved by a conventional technique of addition / recovery.

Each of the M time signals SS ^m (n) can then be supplied to a spatialization decoding block.

For this purpose, it is possible, for example, to provide a pair of matched filters Fg ^m (n), Fd ^m (n) to convolutionally apply to each signal SS ^m (n), as shown in FIG. figure 3 , for an adaptation of an ambiophonic encoding to a left and right two-way binaural rendition. These filters for such an ambiophonic / binaural transition can be obtained by applying the virtual speaker technique mentioned above.

$${\mathrm{S}{S}^{\mathrm{m}}}_d\left(n\right)=\left(\mathrm{S}{S}^{\mathrm{m}}*{\mathit{Fd}}^m\right)\left(n\right)$$

Après filtrage, tous les signaux destinés aux oreilles gauche et droite sont sommés respectivement, et on obtient ainsi une paire de signaux binauraux : $${\mathrm{S}}_g\left(n\right)={\displaystyle \sum _{m=1}^{\mathrm{M}}}\mathrm{S}{S}_{\mathrm{g}}^{\mathrm{m}}\left(n\right)$$

$${\mathrm{S}}_d\left(n\right)={\displaystyle \sum _{m=1}^{\mathrm{M}}}\mathrm{S}{S}_{\mathrm{d}}^{\mathrm{m}}\left(n\right)$$

qui viennent alimenter alors les écouteurs d'un casque à deux oreillettes.The processing carried out by the DECOD block of spatial decoding of the figure 3 can be of type: $${\mathrm{S}{S}^{\mathrm{m}}}_{\mathrm{boy\; Wut}}\left(\mathrm{not}\right)=\left(\mathrm{S}{S}^{\mathrm{m}}*{\mathit{fg}}^m\right)\left(\mathrm{not}\right)$$

$${\mathrm{S}{S}^{\mathrm{m}}}_d\left(\mathrm{not}\right)=\left(\mathrm{S}{S}^{\mathrm{m}}*{\mathit{fd}}^m\right)\left(\mathrm{not}\right)$$

After filtering, all the signals intended for the left and right ears are summed respectively, and thus a pair of binaural signals is obtained: $${\mathrm{S}}_{\mathrm{boy\; Wut}}\left(\mathrm{not}\right)={\displaystyle \underset{m=1}{\overset{\mathrm{M}}{\Sigma }}}\mathrm{S}{S}_{\mathrm{boy\; Wut}}^{\mathrm{m}}\left(\mathrm{not}\right)$$

$${\mathrm{S}}_d\left(\mathrm{not}\right)={\displaystyle \underset{m=1}{\overset{\mathrm{M}}{\Sigma }}}\mathrm{S}{S}_{\mathrm{d}}^{\mathrm{m}}\left(\mathrm{not}\right)$$

which then feed the headphones of a headset with two earpieces.

However, a more advantageous variant is described below. The adaptation filters from the surround format to the binaural format can be applied directly in the frequency domain, thus avoiding convolution in the time domain and a corresponding calculation cost.

$${{\mathrm{S}S}^m}_d\left(\omega \right)={\mathrm{S}S}^m\left(\omega \right)\mathrm{.}\mathrm{F}{\mathrm{d}}^m\left(\omega \right)$$

Les spectres sont ensuite sommés par oreille avant d'effectuer la transformée de Fourier inverse et l'opération d'addition/recouvrement, soit : $${\mathrm{S}}_{\mathrm{g}}\left(\omega \right)={\displaystyle \sum _{m=1}^{\mathrm{M}}}{{\mathrm{S}}^m}_g\left(\omega \right)$$

$${\mathrm{S}}_{\mathrm{d}}\left(\omega \right)={\displaystyle \sum _{m=1}^{\mathrm{M}}}{{\mathrm{S}}^m}_d\left(\omega \right)$$

Puis, pour exprimer les signaux alimentant le dispositif de restitution dans le domaine temporel, on applique la transformée de Fourier inverse : $${\mathrm{S}}_{\mathrm{g}}\left(n\right)=\mathit{IFFT}\left(s_g\left(\omega \right)\right)$$

$${\mathrm{S}}_{\mathrm{d}}\left(n\right)=\mathit{IFFT}\left(s_d\left(\omega \right)\right)$$

For this purpose, each of the M frequency spectra ss ^m (ω) is directly multiplied by the respective Fourier transforms of the temporal filters, noted Fg ^m (ω) and Fd ^m (ω) (adapted if necessary to have a coherent number of points), which is written: $${{\mathrm{S}S}^m}_{\mathrm{boy\; Wut}}\left(\omega \right)={\mathrm{S}S}^m\left(\omega \right)\mathrm{.}\mathrm{F}{\mathrm{boy\; Wut}}^m\left(\omega \right)$$

$${{\mathrm{S}S}^m}_d\left(\omega \right)={\mathrm{S}S}^m\left(\omega \right)\mathrm{.}\mathrm{F}{\mathrm{d}}^m\left(\omega \right)$$

The spectra are then summed by ear before performing the inverse Fourier transform and the addition / recovery operation, ie: $${\mathrm{S}}_{\mathrm{boy\; Wut}}\left(\omega \right)={\displaystyle \underset{m=1}{\overset{\mathrm{M}}{\Sigma }}}{{\mathrm{S}}^m}_{\mathrm{boy\; Wut}}\left(\omega \right)$$

$${\mathrm{S}}_{\mathrm{d}}\left(\omega \right)={\displaystyle \underset{m=1}{\overset{\mathrm{M}}{\Sigma }}}{{\mathrm{S}}^m}_d\left(\omega \right)$$

Then, to express the signals supplying the reproduction device in the time domain, the inverse Fourier transform is applied: $${\mathrm{S}}_{\mathrm{boy\; Wut}}\left(\mathrm{not}\right)=\mathit{IFFT}\left(s_{\mathrm{boy\; Wut}}\left(\omega \right)\right)$$

$${\mathrm{S}}_{\mathrm{d}}\left(\mathrm{not}\right)=\mathit{IFFT}\left(s_d\left(\omega \right)\right)$$

• L'étape a) vise l'affectation des paramètres représentatifs d'une amplitude à chaque source S_i. Dans l'exemple représenté, on affecte un paramètre p_i,k par composante fréquentielle f_k comme décrit ci-avant.
• L'étape b) vise la duplication de ces paramètres et leur multiplication par les gains g_i ^m des canaux d'encodage.
• L'étape c) vise le regroupement des produits obtenus à l'étape b), avec en particulier le calcul de leur somme sur toutes les sources S_i.
• L'étape d) vise la synthèse paramétrique avec multiplication par une enveloppe spectrale env_k comme décrit ci-avant, suivi d'un regroupement des sous-canaux par application, dans chaque canal, d'une somme sur toutes les composantes fréquentielles (d'indice k allant de 1 à K).
• L'étape e) vise un décodage de spatialisation des signaux ss^m issus des canaux respectifs, synthétisés, spatialisés et représentés dans le domaine fréquentiel, pour une restitution sur deux haut-parleurs par exemple au format binaural.

The present invention also relates to a computer program product, whether stored in a memory of a central unit or a terminal, or on a removable support adapted to cooperate with a drive of this central unit (CD-ROM , diskette or other), or downloadable via a telecommunications network. This program comprises in particular instructions for the implementation of the method described above and a flow chart of which can be illustrated by way of example on the figure 5 , summarizing the steps of such a process.

• Step a) aims at assigning the parameters representative of an amplitude to each source S _i . In the example shown, a parameter p _{i, k} is assigned by frequency component f _k as described above.
• Step b) aims at the duplication of these parameters and their multiplication by the gains g _i ^m of the encoding channels.
• Step c) relates to the grouping of the products obtained in step b), with in particular the calculation of their sum over all the sources S _i .
• Step d) targets the parametric synthesis with multiplication by a spectral envelope env _k as described above, followed by a grouping of the subchannels by applying, in each channel, a sum over all the frequency components (d index k ranging from 1 to K).
• Step e) aims at decoding the spatialization of the signals ss ^m originating from the respective channels, synthesized, spatialised and represented in the frequency domain, for a reproduction on two loudspeakers, for example in binaural format.

The present invention also provides a device for generating synthetic and spatialized sounds, comprising in particular a processor, and in particular a working memory adapted to store instructions of the computer program product defined above.

Of course, the present invention is not limited to the embodiment described above by way of example; it extends to other variants.

Thus, it has been described above by way of example a spatialisation encoding in ambiophonic format produced by the SPAT module of the figure 2 followed by an adaptation of the surround format to the binaural format. As a variant, it may be provided, for example, to directly apply an encoding to the binaural format.

Moreover, the spectral envelope multiplication of the parametric synthesis is described above by way of example, other models that can be provided alternatively.

Claims (7)

1. Method for jointly synthesizing and spatializing a plurality of sound sources in associated spatial positions, comprising:

   a) a step of assigning to each source at least one parameter (p_i) representing an amplitude of at least one frequency component of the source,

   b) a spatialization step implementing an encoding into a plurality of channels, wherein each amplitude parameter (p_i) is duplicated to be multiplied with a spatialization gain (g_i ^m), each spatialization gain being determined, on the one hand, for an encoding channel (p_g ^m) and, on the other hand, for a source to be spatialized (S_i),

   c) a step of grouping together (R) the parameters (p_i ^m) multiplied by the gains, in respective channels (p_g ¹, ... p_g ^M), by applying a sum of said multiplied parameters (p_i ^m) to all the sources (S_i) for each channel (p_g ^m), and

   d) a parametric synthesis step (SYNTH (1), ..., SYNTH (M)) applied to each of the channels (pg^m).
2. Method according to Claim 1, wherein:

   a) each source (S_i) is assigned a plurality of parameters (p_i, _k), each representing an amplitude of a frequency component (f_k),

   b) each amplitude parameter (p_i, _k) representing a frequency component (f_k) is duplicated to be multiplied with a spatialization gain (g_i ^m), each spatialization gain being determined, on the one hand, for an encoding channel (p_g ^m) and, on the other hand, for a source to be spatialized (S_i),

   c) in each channel, there are grouped together, frequency component by frequency component, the products of the parameters (p_{i, k}) by the gains (g_i ^m), into sub-channels (p_g, k^m) each associated with a frequency component (f_k).
3. Method according to Claim 2, wherein the synthesis is conducted, in each channel, by:

   d1) multiplying the output of each sub-channel associated with a frequency component (f_k) by a spectral envelope (env_k) centered on a frequency corresponding to said frequency component (f_k),

   d2) and grouping together, by a sum over the frequency components (f_k), the products resulting from the operation d1),

   to obtain, following the operation d2), a signal (ss^m) derived from each channel, spatially encoded and synthesized.
4. Method according to one of the preceding claims, wherein the spatialization is conducted by ambiophonic encoding and the parameters representing an amplitude that are assigned to the sources correspond to spherical harmonic amplitudes (Y_m).
5. Method according to Claim 4, taken in combination with Claim 3, wherein, to switch from an ambiophonic encoding to a decoding with a view to playback in binaural spatialization mode, a processing is applied in the frequency domain directly to the results of the products derived from the respective channels after the operation d2).
6. Computer program product, stored in a memory of a central unit or of a terminal, and/or on a removable medium specifically for cooperating with a drive of said central unit, and/or downloadable via a telecommunication network, characterized in that it comprises instructions for the implementation of the method according to one of Claims 1 to 5.
7. Module for generating spatialized synthetic sounds, notably comprising a processor, characterized in that it also comprises a working memory storing instructions of the computer program product according to Claim 6.

Images