EP1600042B1 - Method for the treatment of compressed sound data for spatialization - Google Patents

Abstract

The invention relates to the treatment of sound data for spatialized restitution of acoustic signals. At least one first and one second series of weighting terms are obtained for each acoustic signal, said terms representing a direction of perception of said acoustic signal by a listener. The acoustic signals are then applied to at least two sets of filtering units, which are disposed in parallel, in order to provide at least one first and one second output signal (L,R), corresponding to a linear combination of signals provided by said filtering units, which are respectively weighted by the weighting terms of the first and second series. According to the invention, each acoustic signal to be treated is at least partially compression coded and is expressed in the form of a vector of sub-signals associated with respective frequency sub-bands. Matrix filtering applied to each vector is carried out by each filtering unit in the space of the frequential sub-bands.

Description

The invention relates to a sound data processing for spatialized reproduction of acoustic signals.

The advent of new data encoding formats over telecommunications networks allows the transmission of complex and structured sound scenes that include multiple sound sources. In general, these sound sources are spatialized, that is, they are processed so as to provide a realistic final rendering in terms of source position and room effect (reverberation). This is the case, for example, of coding according to the MPEG-4 standard which makes it possible to transmit complex sound scenes comprising compressed or uncompressed sounds, and synthetic sounds, which are associated with spatialization parameters (position, effect of the room surrounding). This transmission is done on networks with constraints, and the sound reproduction depends on the type of terminal used. For example, on a PDA-type mobile terminal (for "Personal Digital Assistant"), a headset will preferably be used. The constraints of this type of terminals (computing power, memory size) make it difficult to implement sound spatialization techniques.

Sound spatialization covers two different types of treatment. From a monophonic audio signal, we try to give the illusion to a listener that the sound source (s) are in good positions. accurate space (that we want to be able to change in real time), and immersed in a space with particular acoustic properties (reverberation, or other acoustic phenomena such as occlusion). For example, on mobile type telecommunication terminals, it is natural to consider a sound reproduction with a stereo headset. The most effective sound source positioning technique is then binaural synthesis.

It consists, for each sound source, in filtering the monophonic signal by acoustic transfer functions, called HRTFs (of the "Head Related Transfer Functions"), which model the transformations generated by the torso, the head and the horn. the listener's ear on a signal from a sound source. For each position of space, one can measure a pair of these functions (one for the right ear, one for the left ear). The HRTFs are thus functions of a spatial position, more particularly of an azimuth angle θ and of an elevation angle (φ, and of the sound frequency f, which gives a given subject a database of acoustic transfer functions of N space positions for each ear, in which a sound can be "placed" (or "spatialized" according to the terminology used hereinafter).

It is indicated that a similar spatialization treatment consists of a so-called "transaural" synthesis , in which more than two loudspeakers are simply provided in one playback device (which is then in a different form of a helmet with two left and right atria).

In a conventional manner, the implementation of this technique is in the so-called "two-channel" form (treatment shown schematically in FIG. 1 relating to the prior art). For each sound source to be positioned according to the pair of azimuthal and elevation angles [θ, φ], the signal of the source is filtered by the HRTF function of the left ear and by the HRTF function of the right ear. The two left and right channels deliver acoustic signals that are then broadcast to the listener's ears with a stereo headset. This two-channel binaural synthesis is of type hereinafter "static", because in this case, the positions of the sound sources do not evolve in time.

If, on the contrary, it is desired to vary the positions of the sound sources in space over time ("dynamic" synthesis), the filters used to model the HRTFs (left ear and right ear) must be modified. However, since these filters are mostly of the finite impulse response (FIR) or infinite impulse response (IIR) type, discontinuity problems in the left and right output signals appear, causing audible clicks . The technical solution conventionally used to overcome this problem is to run two sets of binaural filters in parallel. The first game simulates a position [θ1, φ1] at time t1, the second a position [θ2, φ2] at the instant t2. The signal giving the illusion of a displacement between the positions at the instants t1 and t2 is then obtained by a fade-out of the left and right signals resulting from the filtering processes for the position [θ1, φ1] and for the position [θ2, φ2]. Thus, the complexity of the positioning system of sound sources is then multiplied by two (two positions at two instants) compared to the static case.

In order to overcome this problem, linear decomposition techniques for HRTFs have been proposed (treatment shown schematically in FIG. 2 relating to the prior art). One of the advantages of these techniques is that they allow an implementation whose complexity depends much less on the total number of sources to be positioned in space. Indeed, these techniques make it possible to break down the HRTFs on the basis of functions common to all the positions of the space, and thus only dependent on the frequency, which makes it possible to reduce the number of necessary filters. Thus, this number of filters is fixed, regardless of the number of sources and / or the number of source positions to be provided. The addition of an additional sound source then adds only multiplication operations by a set of weighting coefficients and by a delay τ _i , these coefficients and this delay being dependent only on the position [θ, φ]. No additional filter is needed.

These linear decomposition techniques are also of interest in the case of dynamic binaural synthesis (ie when the position of sound sources varies course of time). In fact, in this configuration, the coefficients of the filters are no longer varied, but the values of the weighting coefficients and delays as a function solely of the position. The principle described above of linear decomposition of the sound rendering filters is generalized to other approaches, as will be seen below.

Moreover, in the various group communication services (teleconferencing, audio conferencing, videoconferencing, or other) or "streaming" communication (of the English "STREAMING"), to adapt a bit rate to the bandwidth provided by a network, the audio and / or speech streams are transmitted in a compressed coded format. Only flows initially compressed by frequency-domain (or frequency-converted) encoders such as those operating according to the MPEG-1 standard (Layer I-II-III), the MPEG-2/4 standard, are considered below. AAC, the MPEG-4 TwinVQ standard, the Dolby AC-2 standard, the Dolby AC-3 standard, or a ITU-T G.722.1 standard in speech coding, or the Applicant's TDAC coding method. The use of such encoders amounts to first performing a time / frequency transformation on blocks of the time signal. The resulting parameters are then quantized and encoded to be transmitted in a frame with other complementary information needed for decoding. This time / frequency transformation can take the form of a bank of filters in frequency sub-bands or a transform of the MDCT type (for "Modified Discrete Cosine Transform"). Hereinafter, we will designate by the same terms "subband domain" means a domain defined in a frequency subband space, a frequency-transformed time domain domain, or a frequency domain.

To perform sound spatialization on such streams, the conventional method consists in first decoding, performing the sound spatialization processing on the time signals, and then recoding the resulting signals, for transmission to a rendering terminal. This sequence of steps, tedious, is often very expensive in terms of computing power, the memory required for processing and algorithmic delay introduced. It is therefore often unsuited to the constraints imposed by the machines where the processing and the communication constraints are carried out.

For example, document US Pat. No. 6,470,087 describes a device for rendering a multichannel acoustic signal compressed on two loudspeakers. All calculations are done in the entire frequency band of the input signal, which therefore must be completely decoded.

The present invention improves the situation.

One of the aims of the present invention is to propose a sound data processing method comprising the coding / decoding operations in compression of the audio streams and the spatialization of said streams.

Another object of the present invention is to propose a sound data processing method, by spatialization, which adapts to a variable number (dynamically) of sound sources to be positioned.

A general object of the present invention is to propose a method for processing sound data, for example spatialization, allowing a wide dissemination of spatialized sound data, in particular a broadcast for the general public, the playback devices being simply equipped with a decoder of received signals and playback speakers.

• a) on obtient, pour chaque signal acoustique, au moins un premier jeu et un second jeu de termes pondérateurs, représentatifs d'une direction de perception dudit signal acoustique par un auditeur ;
• b) et on applique à au moins deux jeux d'unités de filtrage, disposées en parallèle, lesdits signaux acoustiques, pour délivrer au moins un premier signal de sortie et un second signal de sortie correspondant chacun à une combinaison linéaire des signaux acoustiques pondérés par l'ensemble des termes pondérateurs respectivement du premier jeu et du second jeu et filtrés par lesdites unités de filtrage.

To this end, it proposes a method for processing sound data, for a spatialized reproduction of acoustic signals, in which:

• a) there is obtained, for each acoustic signal, at least a first set and a second set of weighting terms, representative of a direction of perception of said acoustic signal by a listener;
• b) and applying to at least two sets of filter units, arranged in parallel, said acoustic signals, for outputting at least a first output signal and a second output signal each corresponding to a linear combination of the weighted acoustic signals; the set of weighting terms respectively of the first set and the second set and filtered by said filtering units.

Each acoustic signal in step a) of the method according to the invention is at least partially coded in compression and is expressed in the form of a subsignal vector associated with respective frequency sub-bands, and each filtering unit is arranged to perform a matrix filtering applied to each vector, in the frequency subband space.

Advantageously, each matrix filtering is obtained by converting, in the space of the frequency sub-bands, an impulse response filter (finite or infinite) defined in the time space. Such an impulse response filter is preferably obtained by determining an acoustic transfer function depending on a direction of perception of a sound and the frequency of this sound.

According to an advantageous characteristic of the invention, these transfer functions are expressed by a linear combination of terms dependent on the frequency and weighted by terms dependent on the direction, which allows, as indicated above, on the one hand , process a variable number of acoustic signals in step a) and, secondly, dynamically vary the position of each source in time. In addition, such an expression of the transfer functions "integrates" the interaural delay which is conventionally applied to one of the output signals, with respect to the other, before the restitution, in binaural processing. For this purpose, gain filter matrices associated with each signal are provided.

Thus, said first and second output signals being preferentially intended to be decoded into first and second reproduction signals, the aforementioned linear combination already takes into account a time shift between these first and second reproduction signals, advantageously.

Finally, between the reception / decoding step of the signals received by a rendering device and the rendering step itself, no additional step of sound spatialization can be envisaged, this spatialization processing being performed completely upstream and directly on coded signals.

According to one of the advantages afforded by the present invention, the combination of linear decomposition techniques of the HRTFs with filtering techniques in the field of the sub-bands makes it possible to take advantage of the advantages of the two techniques to arrive at sound spatialization systems. Low complexity and reduced memory for multiple coded audio signals.

Indeed, in a conventional "dual channel " architecture, the number of filters to use is a function of the number of sources to be positioned. As mentioned above, this problem is not found in an architecture based on the linear decomposition of HRTFs. This technique is therefore preferable in terms of computing power, but also memory space required for binaural filters storage. Finally, this architecture makes it possible to optimally manage the dynamic binaural synthesis, because it makes it possible to "fading" between two instants t1 and t2 on coefficients that depend only on the position, and therefore does not require two sets of filters in parallel.

According to another advantage provided by the present invention, the direct filtering of the signals in the coded domain allows the economy of a complete decoding by audio stream before proceeding to the spatialization of the sources, which implies a considerable gain in complexity.

According to another advantage provided by the present invention, the sound spatialization of audio streams can occur at different points of a transmission chain (servers, network nodes or terminals). The nature of the application and the architecture of the communication used may favor one case or another. Thus, in a teleconference context, the spatialization processing is preferably performed at the terminals in a decentralized architecture and, conversely, at the audio bridge (or MCU for "Multipoint Control Unit") in a centralized architecture. For "streaming" audio applications, especially on mobile terminals, the spatialization can be performed either in the server or in the terminal, or during the creation of content. In these different cases, a reduction in the processing complexity and also the memory required for storing the HRTF filters is still appreciated. For example, for mobile terminals (second- and third-generation mobile phones, PDAs, or hand-held microcomputers) having strong constraints in terms of computing capacity and memory size, preferably spatialization processing is provided directly at the level of the device. a content server.

The present invention can also find applications in the field of the transmission of multiple audio streams included in structured sound scenes, as provided by the MPEG-4 standard.

• la figure 1 illustre schématiquement un traitement correspondant à une synthèse binaurale "bicanale" statique pour des signaux audionumériques temporels S_i, de l'art antérieur ;
• la figure 2 représente schématiquement une mise en oeuvre de la synthèse binaurale basée sur la décomposition linéaire des HRTFs pour des signaux audionumériques temporels non codés, de l'art antérieur ;
• la figure 3 représente schématiquement un système, au sens de l'art antérieur, de spatialisation binaurale de N sources audio initialement codées, puis complètement décodées pour le traitement de spatialisation dans le domaine temporel et ensuite recodées pour une transmission à un ou plusieurs dispositifs de restitution, ici à partir d'un serveur ;
• la figure 4 représente schématiquement un système, au sens de la présente invention, de spatialisation binaurale de N sources audio partiellement décodées pour le traitement de spatialisation dans le domaine des sous-bandes et ensuite recodées complètement pour la transmission à un ou plusieurs dispositifs de restitution, ici à partir d'un serveur ;
• la figure 5 représente schématiquement un traitement de spatialisation sonore dans le domaine des sous-bandes, au sens de l'invention, basé sur la décomposition linéaire des HRTFs dans le contexte binaural ;
• la figure 6 représente schématiquement un traitement d'encodage/décodage pour spatialisation, mené dans le domaine des sous-bandes et basé sur une décomposition linéaire de fonctions de transfert dans le contexte ambisonique, dans une variante de réalisation de l'invention ;
• la figure 7 représente schématiquement un traitement de spatialisation binaurale de N sources audio codées, au sens de la présente invention, effectué auprès d'un terminal de communication, selon une variante du système de la figure 4 ;
• la figure 8 représente schématiquement une architecture d'un système de téléconférence centralisée, avec un pont audio entre une pluralité de terminaux ; et
• la figure 9 représente schématiquement un traitement, au sens de la présente invention, de spatialisation de (N-1) sources audio codées parmi N sources en entrée d'un pont audio d'un système selon la figure 8, effectué auprès de ce pont audio, selon une variante du système de la figure 4.

Other characteristics, advantages and applications of the invention will appear on examining the detailed description below, and the attached drawings in which:

• FIG. 1 diagrammatically illustrates a processing corresponding to a binaural "two- channel " static synthesis for temporal digital audio signals S _i , of the prior art;
• FIG. 2 diagrammatically represents an implementation of the binaural synthesis based on the linear decomposition of the HRTFs for uncoded temporal digital audio signals of the prior art;
• FIG. 3 diagrammatically represents a system, in the sense of the prior art, of binaural spatialisation of N initially coded audio sources, then completely decoded for spatialization processing in the time domain, and then recoded for transmission to one or more transmission devices. restitution, here from a server;
• FIG. 4 diagrammatically represents a system, within the meaning of the present invention, of binaural spatialisation of N partially decoded audio sources for spatialization processing in the subband domain and then completely recoded for transmission to one or more rendering devices. , here from a server;
• FIG. 5 schematically represents a sound spatialization processing in the field of the subbands, at the sense of the invention, based on the linear decomposition of HRTFs in the binaural context;
• FIG. 6 schematically represents an encoding / decoding process for spatialization, carried out in the subband domain and based on a linear decomposition of transfer functions in the ambisonic context, in an alternative embodiment of the invention;
• FIG. 7 diagrammatically represents a binaural spatialization processing of N coded audio sources, within the meaning of the present invention, carried out with a communication terminal, according to a variant of the system of FIG. 4;
• Figure 8 schematically shows an architecture of a centralized teleconferencing system, with an audio bridge between a plurality of terminals; and
• FIG. 9 schematically represents a processing, within the meaning of the present invention, of spatialization of (N-1) audio sources encoded among N input sources of an audio bridge of a system according to FIG. 8, carried out with this bridge according to a variant of the system of FIG. 4.

Firstly, reference is made to FIG. 1 to describe a "binational" binaural synthesis treatment. This treatment consists in filtering the signal of the sources (Si) that one wishes to position at a position chosen in space by the acoustic transfer functions left (HRTF_1) and right (HRTF_r) corresponding to the direction <θi, φi) appropriate. Two signals are obtained which are then added to the left and right signals resulting from the spatialization of the other sources, to give the global signals L and R diffused to the left and right ears of a listener. The number of necessary filters is then 2.N for static binaural synthesis and 4.N for dynamic binaural synthesis, where N is the number of audio streams to spatialize.

Reference is now made to FIG. 2 to describe a conventional binaural synthesis treatment based on the linear decomposition of HRTFs. Here, each HRTF filter is first decomposed into a minimum phase filter, characterized by its modulus, and a pure delay τ _i . The spatial and frequency dependencies of the HRTFs modules are separated by a linear decomposition. These modules of transfer functions HRTFs are then written as a sum of spatial functions C _n (θ, φ) and of reconstruction filters L _n (f), as expressed hereafter: $$\left|\mathit{HRTF}\left(\theta ,\phi ,f\right)\right|={\displaystyle {\Sigma }_{\mathrm{not}=1}^P{\mathrm{VS}}_{\mathrm{not}}\left(\theta ,\phi \right).{\mathrm{The}}_{\mathrm{not}}\left(f\right)}$$

Each signal of a source S _i to be spatialised (i = 1, ..., N) is weighted by coefficients C _ni (θ, φ) (n = 1, ..., P) resulting from the linear decomposition of HRTFs. These coefficients have the particularity of being dependent only on the position [ θ, φ ] where it is desired to place the source, and not on the frequency f. The number of these coefficients depends on the number P of basic vectors that has been preserved for the reconstruction. The N signals of all the sources weighted by the "directional" coefficient C _ni are then summed (for the right channel and the left channel, separately), then filtered by the filter corresponding to the nth base vector. Thus, unlike binaural synthesis "two-channel", the addition of an additional source does not require the addition of two additional filters (often type FIR or IIR). The basic P filters are indeed shared by all the sources present. This implementation is called "multichannel". Moreover, in the case of dynamic binaural synthesis, it is possible to vary the coefficients C _ni (θ, φ) without appearance of clicks at the device output. In this case, only 2.P filters are needed, whereas 4.N filters were needed for two-channel synthesis.

In FIG. 2, the coefficients C _ni correspond to the directional coefficients for the source i at the position (θi, φi) and for the reconstruction filter n. They are noted C for the left channel (L) and D for the right channel (R). It is indicated that the processing principle of the right channel R is the same as that of the left channel L. However, the dotted line arrows for the treatment of the right channel have not been shown for the sake of clarity of the drawing. Between the two vertical lines in dashed line of Figure 2, then defines a system noted I, of the type shown in Figure 3.

However, before referring to FIG. 3, it is indicated that different methods have been proposed for determining the spatial functions and the reconstruction filters. A first method is based on a so-called Karhunen-Loeve decomposition and is described in particular in WO94 / 10816. Another method is based on principal component analysis of HRTFs and is described in WO96 / 13962. The document FR-2782228 more recent also describes such an implementation.

In the case where a spatialization processing of this type is done at the communication terminal, a step of decoding the N signals is necessary before the actual spatialization processing. This step requires considerable computing resources (which is problematic on current communication terminals, particularly of the portable type). Moreover, this step causes a delay on the processed signals, which interferes with the interactivity of the communication. If the transmitted sound scene comprises a large number of sources (N), the decoding step may in fact become more expensive in computing resources than the sound spatialization step itself. Indeed, as indicated above, the calculation cost of binaural synthesis "multichannel" depends very little on the number of sound sources to spatialize.

• décodage (pour N signaux),
• application du retard interaural τ_i,
• multiplication par les gains positionnels C_ni (PxN gains pour l'ensemble des N signaux),
• sommation des N signaux pour chaque filtre de base d'indice n,
• filtrage des P signaux par les filtres de base,
• et sommation des P signaux de sortie des filtres de base.

The cost of calculating the spatialization operation of the N coded audio streams (in the multichannel synthesis of FIG. 2) can thus be deduced from the following steps (for the synthesis of one of the two left or right rendering channels):

• decoding (for N signals),
• application of the interaural delay τ _i ,
• multiplication by positional gains C _ni (PxN gains for all N signals),
• summation of the N signals for each basic filter of index n,
• filtering the P signals by the basic filters,
• and summing the P output signals of the basic filters.

In the case where the spatialization is not done at the level of a terminal but at the level of a server (case of Figure 3), or in a node of a communication network (case of an audio bridge in teleconference), it is necessary to add a complete coding operation of the output signal.

• un décodage complet des N sources audio S₁, ..., S_i, ..., S_N codées en entrée du système représenté (noté "Système I") pour obtenir N flux audio décodés, correspondant par exemple à des signaux PCM (pour "Pulse Code Modulation"),
• un traitement de spatialisation dans le domaine temporel ("Système I") pour obtenir deux signaux spatialisés L et R,
• et ensuite un recodage complet sous forme de canaux gauche et droit L et R, véhiculés dans le réseau de communication pour être reçus par un ou plusieurs dispositifs de.restitution.

Referring to FIG. 3, the spatialization of N sound sources (for example part of a complex sound scene of the MPEG4 type) therefore requires:

• a complete decoding of the N audio sources S ₁ ,..., S _i ,..., S _N encoded at the input of the system represented (denoted "System I") to obtain N decoded audio streams, corresponding, for example, to PCM signals (for " Pulse Code Modulation" ),
• temporal spatialization processing ("System I") to obtain two spatialized signals L and R,
• and then a complete recoding in the form of left and right channels L and R, conveyed in the communication network to be received by one or more restitution devices.

Thus, the decoding of N coded streams is necessary before the step of spatialization of the sound sources, which leads to an increase in the calculation cost and the addition of a delay due to the processing of the decoder. It indicates that the original audio sources are usually stored directly in coded format, in the current content servers.

It is furthermore indicated that for a restitution on more than two loudspeakers (transaural synthesis or in "ambisonic" context that is described hereinafter), the number of signals resulting from the spatialization processing is generally greater than two, which further increases the cost of calculation to completely recode these signals before their transmission through the communication network.

Referring now to Figure 4 to describe an implementation of the method within the meaning of the present invention.

It consists of associating the "multichannel" implementation of binaural synthesis (FIG. 2) with filtering techniques in the transformed domain (so-called "sub-band" domain ) so as not to have to carry out N complete decoding operations before 'stage of spatialization. This reduces the overall calculation cost of the operation. This "integration" of the coding and spatialization operations can be performed in the case of a processing at a communication terminal or a processing at a server as shown in FIG. 4.

The different stages of data processing as well as the architecture of the system are described in detail below.

In the case of a spatialization of multiple coded audio signals, at the server as in the example shown in Figure 4, a partial decoding operation is still necessary. However, this operation is much less expensive than the decoding operation in a conventional system as shown in FIG. 3. Here, this operation consists mainly in recovering the parameters of the subbands from the encoded binary audio stream. This operation depends on the initial encoder used. It may for example consist of entropy decoding followed by inverse quantization as in an MPEG-1 Layer III coder. Once these parameters of the subbands found, the treatment is performed in the field of sub-bands, as will be seen below.

The overall calculation cost of the spatialization operation of the coded audio streams is then considerably reduced. Indeed, the initial decoding operation in a conventional system is replaced by a partial decoding operation of much less complexity. The computing load in a system according to the invention becomes substantially constant as a function of the number of audio streams that it is desired to spatialize. Compared to conventional systems, we obtain a gain in terms of calculation cost which then becomes proportional to the number of audio streams that we wish to spatialize. In addition, the partial decoding operation results in a lower processing delay than the full decoding operation, which is particularly interesting in a context of interactive communication.

The system for implementing the method according to the invention, performing spatialization in the field of sub-bands, is denoted "System II" in Figure 4.

The following describes obtaining the parameters in the subband domain from binaural impulse responses.

Conventionally, the binaural transfer functions or HRTFs are accessible in the form of temporal impulse responses. These functions generally consist of 256 time samples, at a sampling frequency of 44.1 kHz (typical in the audio field). These impulse responses can be derived from measurements or acoustic simulations.

• extraction du retard interaural à partir de réponses impulsionnelles binaurales h _l (n) et h _r (n) (si l'on dispose de D directions de l'espace mesurées, on obtient un vecteur de D valeurs de retard interaural ITD (exprimé en secondes)) ;
• modélisation des réponses impulsionnelles binaurales sous forme de filtres à phase minimale ;
• choix du nombre de vecteurs de base (P) que l'on souhaite conserver pour la décomposition linéaire des HRTFs ;
• décomposition linéaire des réponses à phase minimale selon la relation Eq[1] ci-avant (on obtient ainsi les D coefficients directionnels C_ni et D_ni qui ne dépendent que de la position de la source sonore à spatialiser et les P vecteurs de base qui ne dépendent que de la fréquence) ;
• modélisation des filtres de base L_n et R_n sous forme de filtres IIR ou FIR ;
• calcul de matrices de filtres de gains G _i dans le domaine des sous-bandes à partir des D valeurs d'ITD (ces retards ITD sont alors considérés comme des filtres FIR destinés à être transposés dans le domaine des sous-bandes, comme on le verra ci-après. Dans le cas général, G _i est une matrice de filtres. Les D coefficients directionnels C _ni , D _ni à appliquer dans le domaine des sous-bandes sont des scalaires de mêmes valeurs que les C_ni et D_ni respectivement dans le domaine temporel) ;
• transposition des filtres de base L_n et R_n, initialement sous forme IIR ou FIR, dans le domaine des sous-bandes (cette opération donne des matrices de filtres, notées ci-après L _n et R_n, à appliquer dans le domaine des sous-bandes. La méthode pour effectuer cette transposition est indiquée ci-après).

The pre-treatment steps for obtaining the parameters in the subband domain are preferably the following:

• extraction of the interaural delay from binaural impulse responses h _l (n) and h _r (n) (if we have D measured space directions, we obtain a vector of D interaural delay values ITD (expressed in seconds));
• modeling of binaural impulse responses in the form of minimal phase filters;
• choice of the number of basic vectors (P) that one wishes to keep for the linear decomposition of HRTFs;
• linear decomposition of the minimal phase responses according to the equation Eq [1] above (we thus obtain the D directional coefficients C _ni and D _ni which depend only on the position of the sound source to be spatialised and the P basic vectors which depend only on the frequency);
• modeling of the basic filters L _n and R _n in the form of IIR or FIR filters;
• calculation of gain filter matrices G _i in the subband domain from the D values of ITD (these ITD delays are then considered as FIR filters to be transposed in the subband domain, as it is see below: In the general case, G _i is a matrix of filters The D directional coefficients C _ni , D _ni to be applied in the subband domain are scalars of the same values as the C _ni and D _ni respectively in the time domain);
• transposition of the basic filters L _n and R _n , initially in IIR or FIR form, in the domain of the sub-bands (this operation gives filter matrices, hereinafter denoted L _n and R _n , to be applied in the field of subbands The method for performing this transposition is shown below).

It will be noted that the filter matrices Gi applied independently to each source "integrate" a conventional delay calculation operation for adding the interaural delay between a signal L _i and a signal R _i to return. Indeed, in the time domain, delay lines τ _i are conventionally provided (FIG. 2) to be applied to a "left ear" signal with respect to a "right ear" signal. In the field of sub-bands, rather, such a matrix of filters G _{i is provided} , which in addition makes it possible to adjust gains (for example in energy) of certain sources with respect to others.

In the case of a transmission from a server to playback terminals, all these steps are performed advantageously offline. The filter matrices above are thus calculated once and then permanently stored in the server memory. In particular, it will be noted that the set of weighting coefficients C _n , D _{ni and} advantageously remains unchanged from the time domain to the domain of the subbands.

For spatialization techniques based on filtering by HRTFs filters and adding the ITD delay (for "Interaural Time Delay") such as binaural and transaural synthesis, or else transfer function filters in the ambisonic context, a difficulty is presented to find equivalent filters to be applied to samples in the subband field. Indeed, these filters from the analysis filter bank must preferably be constructed in such a way that the left and right temporal signals reproduced by the synthesis filter bank have the same sound reproduction, and without any artifact, as that obtained by direct spatialization on a time signal. The design of filters to achieve such a result is not immediate. Indeed, the modification of the spectrum of the signal provided by a filtering in the time domain can not be carried out directly on the signals of the subbands without taking into account the phenomenon of spectrum overlap ("aliasing") introduced by the filter bank d 'analysis. The dependency relationship between the aliasing components of the different sub-bands is preferentially retained during the filtering operation so that their deletion is ensured by the synthesis filter bank.

A method for transposing a rational-type filter S (z) of type FIR or IIR is described below (its z-transform being a quotient of two polynomials) in the case of linear decomposition of HRTFs or transfer functions. of this type, in the area of sub-bands, for a filterbank M subband and critical sampling, respectively defined by its filter analysis and synthesis of H _k (z) and F _k (z), where 0≤k≤M-1. The term "critical sampling" means that the number of all the output samples of the subbands corresponds to the number of samples in inputs. This filter bank is also supposed to satisfy the condition of perfect reconstruction.

où S _k (z) (0≤k≤M-1) sont les composantes polyphasées du filtre S(z).We first consider a transfer matrix S (z) corresponding to the scalar filter S (z), which is expressed as follows: $$\mathrm{S}\left(z\right)=\left[\begin{array}{cccccc}{\mathrm{S}}_0\left(z\right)& {\mathrm{S}}_1\left(z\right)& ...& & & {\mathrm{S}}_{M-1}\left(z\right)\\ z^{-1}{\mathrm{S}}_{M-1}\left(z\right)& {\mathrm{S}}_0\left(z\right)& {\mathrm{S}}_1\left(z\right)& ...& & {\mathrm{S}}_{M-\mathrm{two}}\left(z\right)\\ z^{-1}{\mathrm{S}}_{M-\mathrm{two}}\left(z\right)& z^{-1}{\mathrm{S}}_{M-1}\left(z\right)& {\mathrm{S}}_0\left(z\right)& {\mathrm{S}}_1\left(z\right)& ...& {\mathrm{S}}_{M-3}\left(z\right)\\ ⋮& \ddots & \ddots & \ddots & & ⋮\\ & & & & & {\mathrm{S}}_1\left(z\right)\\ z^{-1}{\mathrm{S}}_1\left(z\right)& ...& & & z^{-1}{\mathrm{S}}_{M-1}\left(z\right)& {\mathrm{S}}_0\left(z\right)\end{array}\right],$$

where S _k (z) (0≤k≤M-1) are the polyphase components of the filter S (z).

• [1] A. Benjelloun Touimi, "Traitement du signal audio dans le domaine codé : techniques et applications" thèse de doctorat de l'Ecole Nationale Supérieure des Télécommunications de Paris, (Annexe A, p.141), Mai 2001.

These components are obtained directly for a FIR filter. For IIR filters, a calculation method is indicated in:

• [1] A. Benjelloun Touimi, "Processing of the audio signal in the coded domain: techniques and applications" Doctoral thesis of the National School of Telecommunications of Paris, (Annex A, p.141), May 2001.

Polyphase matrices E (z) and R (z) corresponding to the analysis and synthesis filter banks are then determined. These matrices are definitively determined for the filter bank under consideration.

The complete filter matrix in sub-bands is then calculated by the following formula:
S _sb (z) = z ^k E (z) S (z) R (z), where z ^K corresponds to an advance with K = (L / M) -1 (characterizing the bank of filters used), L being the length of filters for analysis and synthesis of filter banks used.

• i est l'indice de la (i+1)ième ligne et compris entre 0 et M-1,
• 1 = i-δ mod[M], où δ correspond à un nombre choisi de sous-diagonales adjacentes, la notation mod[M] correspondant à une opération de soustraction modulo M,
• n = i+δ mod[M], la notation mod[M] correspondant à une opération d'addition modulo M.

The matrix S _sb (z) whose lines are obtained from those of S _sb (z) are then constructed as follows:
[0 ... S ^sb _il (z) ... S ^sb _ii (z) ... S ^sb _in (z) ... 0] (0≤n≤M-1), where:

• i is the index of the (i + 1) th line and between 0 and M-1,
• 1 = i-δ mod [M], where δ corresponds to a chosen number of adjacent subdiagals, the mod notation [M] corresponding to a modulo M subtraction operation,
• n = i + δ mod [M], the mod notation [M] corresponding to a modulo M addition operation.

It is indicated that the chosen number δ corresponds to the number of bands which overlap sufficiently on one side with the bandwidth of a filter of the filterbank. It therefore depends on the type of filter banks used in the chosen coding. By way of example, for the MDCT filter bank, δ can be taken as 2 or 3. For the Pseudo-QMF filter bank of the MPEG-1 coding, δ is taken as 1.

It will be noted that the result of this transposition of a finite or infinite impulse response filter to the domain of the sub-bands is a matrix of filters of size MxM. However, not all filters in this matrix are considered during subband filtering. Advantageously, only the filters of the main diagonal and some adjacent sub-diagonals can be used to obtain a result similar to that obtained by filtering in the time domain (without altering the quality of the restitution).

The matrix S _sb (z) resulting from this transposition, then reduced, is that used for the subband filtering.

By way of example, the expression of the polyphase matrices E (z) and R (z) for an MDCT filter bank, widely used in current transform coders such as those operating according to the MPEG-2 standards, is indicated below. / 4 AAC, or Dolby AC-2 & AC-3, or TDAC of the Applicant. The following treatment can also be adapted to a Pseudo-QMF filter bank of the MPEG-1/2 Layer I-II coder.

où h[l] correspond à la fenêtre de pondération dont un choix possible est la fenêtre sinusoïdale qui s'exprime sous la forme suivante : $$h\left[l\right]=\sin \left[\left(l+\frac{1}{2}\right)\frac{\pi }{2M}\right],\hspace{0.17em}0\le l\le 2M-1.$$

Les matrices polyphasées d'analyse et de synthèse sont alors données respectivement par les formules suivantes : $$\mathrm{E}\left(z\right)={\mathrm{T}}_1{\mathrm{J}}_{\mathrm{M}}+{\mathrm{T}}_0{\mathrm{J}}_{\mathrm{M}}{z}^{-1},$$

$$\mathrm{R}\left(z\right)={\mathrm{J}}_{\mathrm{M}}{\mathrm{T}}_0^T+{\mathrm{J}}_{\mathrm{M}}{\mathrm{T}}_1^T{z}^{-1},$$

où ${\mathrm{J}}_{\mathrm{M}}=\left(\begin{array}{ccc}0& \dots & 1\\ ⋮& ⋰& ⋮\\ 1& \dots & 0\end{array}\right)$

correspond à la matrice anti-identité de taille MxM et T₀ et T₁ sont des matrices de taille MxM résultant de la partition suivante: $$\mathrm{T}=\left[\begin{array}{cc}{\mathrm{T}}_0& {\mathrm{T}}_1\end{array}\right].$$

On indique que pour ce banc de filtres L=2M et K =1.An MDCT filter bank is generally defined by a matrix T = [t _{k, l} ], of size M × 2M, whose elements are expressed as follows: $$t_{k,l}=\sqrt{\frac{\mathrm{two}}{M}}h\left[l\right]\cos \left[\frac{\pi }{M}\left(k+\frac{1}{\mathrm{two}}\right)\left(l+\frac{M+1}{\mathrm{two}}\right)\right],0\le k\le M-1\mathrm{and}0\le l\le \mathrm{two}M-1,$$

where h [ l ] corresponds to the weighting window whose possible choice is the sinusoidal window which is expressed in the following form: $$h\left[l\right]=\sin \left[\left(l+\frac{1}{\mathrm{two}}\right)\frac{\pi }{\mathrm{two}M}\right],0\le l\le \mathrm{two}M-1.$$

The polyphase analysis and synthesis matrices are then respectively given by the following formulas: $$\mathrm{E}\left(z\right)={\mathrm{T}}_1{\mathrm{J}}_{\mathrm{M}}+{\mathrm{T}}_0{\mathrm{J}}_{\mathrm{M}}{z}^{-1},$$

$$\mathrm{R}\left(z\right)={\mathrm{J}}_{\mathrm{M}}{\mathrm{T}}_0^T+{\mathrm{J}}_{\mathrm{M}}{\mathrm{T}}_1^T{z}^{-1},$$

or ${\mathrm{J}}_{\mathrm{M}}=\left(\begin{array}{ccc}0& ...& 1\\ ⋮& ⋰& ⋮\\ 1& ...& 0\end{array}\right)$

corresponds to the anti-identity matrix of size MxM and T ₀ and T ₁ are matrices of size MxM resulting from the following partition: $$\mathrm{T}=\left[\begin{array}{cc}{\mathrm{T}}_0& {\mathrm{T}}_1\end{array}\right].$$

It is indicated that for this bank of filters L = 2M and K = 1.

avec les relations suivantes : L=2mM et K=2m-1 où m est un entier. Plus particulièrement dans le cas du codeur MPEG-1/2 Layer I-II, ces paramètres prennent les valeurs suivantes : M=32, L=512, m=8 et K=15. For Pseudo-QMF filter banks of MPEG-1/2 Layer I-II, we define a weighting window h [i], i = 0 ... L-1, and a cosine modulation matrix Ĉ = [c _kl ], of size M × 2M, whose coefficients are given by: $${\mathrm{vs}}_{kl}=\cos \left[\frac{\pi }{M}\left(k+\frac{1}{\mathrm{two}}\right)\left(l-\frac{M}{\mathrm{two}}\right)\right],0\le l\le \mathrm{two}M-1\mathrm{and}0\le k\le M-1,$$

with the following relations: L = 2mM and K = 2m- 1 where m is an integer. More particularly in the case of the MPEG-1/2 Layer I-II encoder, these parameters take the following values: M = 32, L = 512, m = 8 and K = 15.

où g₀ (z) et g₁ (z) sont des matrices diagonales définies par : $$\{\begin{array}{l}{\mathrm{g}}_0\left(z\right)=\mathrm{diag}\left[\begin{array}{cccc}{\mathrm{G}}_0\left(z\right)& {\mathrm{G}}_1\left(z\right)& \cdots & {\mathrm{G}}_{\mathrm{M}-1}\left(z\right)\end{array}\right],\\ {\mathrm{g}}_1\left(z\right)=\mathrm{diag}\left[\begin{array}{cccc}{\mathrm{G}}_{\mathrm{M}}\left(z\right)& {\mathrm{G}}_{\mathrm{M}+1}\left(z\right)& \cdots & {\mathrm{G}}_{2\mathrm{M}-1}\left(z\right)\end{array}\right],\end{array}$$

avec $${\mathrm{G}}_k\left(-z^2\right)={\displaystyle \sum _{l=0}^{m-1}{\left(-1\right)}^{l}h\left(2lM+k\right)z^{-2l}},\hspace{0.17em}0\le k\le 2M-1.$$

The polyphase analysis matrix is then expressed as follows: $$\mathrm{E}\left(z\right)=\widehat{\mathrm{VS}}\left[\begin{array}{c}{\mathrm{boy\; Wut}}_0\left(-z^{\mathrm{two}}\right)\\ z^{-1}{\mathrm{boy\; Wut}}_1\left(-z^{\mathrm{two}}\right)\end{array}\right],$$

where g ₀ (z) and g ₁ (z) are diagonal matrices defined by: $$\{\begin{array}{l}{\mathrm{boy\; Wut}}_0\left(z\right)=\mathrm{diag}\left[\begin{array}{cccc}{\mathrm{BOY\; WUT}}_0\left(z\right)& {\mathrm{BOY\; WUT}}_1\left(z\right)& \cdots & {\mathrm{BOY\; WUT}}_{\mathrm{M}-1}\left(z\right)\end{array}\right],\\ {\mathrm{boy\; Wut}}_1\left(z\right)=\mathrm{diag}\left[\begin{array}{cccc}{\mathrm{BOY\; WUT}}_{\mathrm{M}}\left(z\right)& {\mathrm{BOY\; WUT}}_{\mathrm{M}+1}\left(z\right)& \cdots & {\mathrm{BOY\; WUT}}_{\mathrm{two}\mathrm{M}-1}\left(z\right)\end{array}\right],\end{array}$$

with $${\mathrm{BOY\; WUT}}_k\left(-z^{\mathrm{two}}\right)={\displaystyle \underset{l=0}{\overset{m-1}{\Sigma }}{\left(-1\right)}^{l}h\left(\mathrm{two}lM+k\right)z^{-\mathrm{two}l}},0\le k\le \mathrm{two}M-1.$$

In the MPEG-1 Audio Layer I-II standard, the values of the window (-1) ¹ h (2 lM + k ) are typically provided, with 0≤k≤2M-1, 0≤l≤m-1.

The polyphase synthesis matrix can then be deduced simply by the following formula: $$\mathrm{R}\left(z\right)=z^{-\left(\mathrm{two}m-1\right)}{\mathrm{E}}^T\left(z^{-1}\right)$$

Thus, with reference now to FIG. 4 in the sense of the present invention, partial _N- coded audio sources S ₁ ,..., S _i ,..., S _{N are} decoded in compression to obtain S ₁ , ..., S _i , ..., S _N signals preferably corresponding to signal vectors whose coefficients are values each assigned to a sub-band. "Partial decoding" is understood to mean a processing that makes it possible to obtain, from the compression-coded signals, such signal vectors in the field of the sub-bands. Position information from which respective values of gains G ₁ ,..., G _i ,..., G _N (for binaural synthesis) and coefficients C _ni (for the left ear) can be obtained. ) and D _ni (for the right ear) for the spatialization processing according to equation Eq [1] given above, as shown in Figure 5. However, the spatialization processing is conducted directly in the field of sub-bands and apply the 2P matrices L _n and R _n of basic filters, obtained as indicated above, to the signal vectors S 1 weighted by the scalar coefficients C _ni and D _ni , respectively.

$$\mathrm{R}\left(z\right)={\displaystyle \sum _{n=1}^P{\mathrm{R}}_n\left(z\right).\left[{\displaystyle \sum _{i=1}^N{D}_{ni}.{\mathrm{S}}_i\left(z\right)}\right]}$$

Referring to FIG. 5, the signal vectors L and R, resulting from the spatialization processing in the subband domain (for example in a processing system denoted "System II" in FIG. 4) are then expressed by the following relations, in a representation by their transformed into z: $$\mathrm{The}\left(z\right)={\displaystyle \underset{\mathrm{not}=1}{\overset{P}{\Sigma }}{\mathrm{The}}_{\mathrm{not}}\left(z\right).\left[{\displaystyle \underset{i=1}{\overset{\mathrm{NOT}}{\Sigma }}{\mathrm{VS}}_{\mathrm{not}i}.{\mathrm{S}}_i\left(z\right)}\right]}$$

$$\mathrm{R}\left(z\right)={\displaystyle \underset{\mathrm{not}=1}{\overset{P}{\Sigma }}{\mathrm{R}}_{\mathrm{not}}\left(z\right).\left[{\displaystyle \underset{i=1}{\overset{\mathrm{NOT}}{\Sigma }}{D}_{\mathrm{not}i}.{\mathrm{S}}_i\left(z\right)}\right]}$$

In the example shown in FIG. 4, the spatialization processing is carried out in a server connected to a communication network. Thus, these L and R signal vectors can be recoded completely in compression to broadcast the compressed signals L and R (left and right channels) in the communication network and to the playback terminals.

Thus, an initial step of partially decoding the coded signals S _i is provided before the spatialization processing. However, this step is much less expensive and faster than the complete decoding operation that was necessary in the prior art (FIG. 3). In addition, the signal vectors L and R are already expressed in the subband domain and the partial recoding of FIG. 4 to obtain the coded signals in L and R compression is faster and less expensive than a complete coding such as shown in Figure 3.

It is indicated that the two vertical discontinuous lines of FIG. 5 delimit the spatialization processing carried out in "System II" of FIG. 4. As such, the present invention also aims at such a system comprising partially coded signal processing means. If, for the implementation of the method according to the invention.

• [2] "A Generic Framework for Filtering in Subband Domain" A. Benjelloun Touimi, IEEE 9^th Workshop on Digital Signal Processing, Hunt, Texas, USA, Octobre 2000,

ainsi que le document [1] cité ci-avant, concernent une méthode générale de calcul d'une transposition dans le domaine des sous-bandes d'un filtre de réponse impulsionnelle finie ou infinie.It indicates that the document:

• [2] "A Generic Framework for Filtering in Subband Domain" A. Benjelloun Touimi, IEEE 9 ^th Workshop on Digital Signal Processing, Hunt, Texas, USA, October 2000,

as well as the document [1] cited above, concern a general method for calculating a transposition in the subband domain of a finite or infinite impulse response filter.

• [3] "Subband-Domain Filtering of MPEG Audio Signals", C.A. Lanciani and R. W. Schafer, IEEE Int. Conf. on Acoust., Speech, Signal Proc., 1999.

It is further indicated that sound spatialization techniques in the field of subbands have been proposed recently, in particular in another document:

• [3] "Subband-Domain Filtering of MPEG Audio Signals", CA Lanciani and RW Schafer, IEEE Int. Conf. Acoust., Speech, Signal Proc., 1999.

The latter document presents a method for transposing a finite impulse response (FIR) filter in the sub-band domain of the pseudo-QMF filterbanks of the MPEG-1 Layer I-II and MDCT encoder of the MPEG-2/4 encoder AAC. The equivalent filtering operation in the subband field is represented by an array of FIR filters. In particular, this proposal is in the context of a transposition of HRTFs filters, directly in their classical form and not in the form of a linear decomposition as expressed by equation Eq [1] above and on a filter basis in the sense of the invention. Thus, a disadvantage of the method in the sense of the latter document is that the spatialization processing can not adapt to any number of sources or audio streams encoded spatialize.

It is indicated that, for a given position, each HRTF filter (of order 200 for a FIR and order 12 for an IIR) gives rise to a matrix of filters (square) of dimension equal to the number of subbands of the bench of filters used. In document [3] cited above, there must be a sufficient number of HRTFs to represent the different positions in space, which poses a problem of memory size if it is desired to spatialize a source at any position in space.

On the other hand, an adaptation of a linear decomposition of the HRTFs in the field of the subbands, within the meaning of the present invention, does not present this problem since the number (P) of basic filter matrices L _n and R _n is much smaller. These matrices are then permanently stored in a memory (of the content server or of the rendering terminal) and allow simultaneous spatialization processing of any number of sources, as shown in FIG.

Hereinafter, a generalization of the spatialization processing in the sense of FIG. 5 will be described with other sound rendering processes, such as a so-called " ambisonic encoding " process. Indeed, a sound rendering system can be generally in the form of a real or virtual sound recording system (for a simulation) consisting of an encoding of the sound field. This phase consists in recording p sound signals in a real way or in simulating such signals (virtual encoding) corresponding to the whole of a sound scene including all the sounds, as well as a room effect.

The aforementioned system may also be in the form of a sound rendering system of decoding the sound output signals to suit the sound rendering translators (such as a plurality of loudspeakers or a loudspeaker). stereophonic headphones). The p signals are transformed into n signals which supply the n loudspeakers.

By way of example, binaural synthesis consists of making a real sound recording, using a pair of microphones introduced into the ears of a human head (artificial or real). The recording can also be simulated by convolving a monophonic sound with the pair of HRTFs corresponding to a desired direction of the virtual sound source. From one or more monophonic signals coming from predetermined sources, two signals (left ear and right ear) are obtained corresponding to a so-called " binaural encoding" phase, these two signals being then simply applied to a two-ear headset (such as a stereo headset).

However, other encodings and decodings are possible from the filter decomposition corresponding to transfer functions on a filter basis. As indicated above, the spatial and frequency dependencies of the transfer functions, of the HRTFs type, are separated by a linear decomposition and are written as a sum of spatial functions C _i (θ, φ) and reconstruction filters L _i ( f ) which depend on the frequency: $$\mathit{HRTF}\left(\theta ,\phi ,f\right)={\displaystyle \underset{i=1}{\overset{P}{\Sigma }}{\mathrm{VS}}_i\left(\theta ,\phi \right).{\mathrm{The}}_i\left(f\right)}$$

où, par exemple dans le cas d'une synthèse binaurale, X_ij peut s'exprimer sous la forme d'un produit des filtres de gains G _j et des coefficients C_ij,D_ij.However, it is indicated that this expression can be generalized to any type of encoding, for n sound sources S _j ( f ) and an encoding format comprising p output signals, to: $$E_i\left(f\right)={\displaystyle \underset{j=1}{\overset{\mathrm{not}}{\Sigma }}{X}_{ij}\left(\theta ,\phi \right).S_j\left(f\right)},1\le i\le p$$

where, for example in the case of a binaural synthesis, X _ij can be expressed in the form of a product of the gain filters G _j and coefficients C _ij , D _ij .

Referring to FIG. 6, in which N audio streams S _j represented in the field of the sub-bands after partial decoding, undergo a spatialization processing, for example an ambisonic encoding, to deliver p signals E _i encoded in the sub domain. -bands. Such Spatialization processing therefore respects the general case governed by equation Eq [2] above. It will also be noted in FIG. 6 that the application to the signals S _j of the filter matrix G _j (to define the interaural delay ITD) is no longer necessary here, in the ambisonic context.

Similarly, a general relationship for a decoding format comprising p signals E _i (f) and a sound rendering format comprising m signals is given by: $$D_j\left(f\right)={\displaystyle \underset{i=1}{\overset{p}{\Sigma }}{K}_{ji}\left(f\right)E_i\left(f\right)},1\le j\le m$$

For a given sound rendering system, the filters K _ji (f) are fixed and depend, at constant frequency, only on the sound rendering system and its arrangement with respect to a listener. This situation is shown in Figure 6 (to the right of the dashed vertical line), in the example of the ambisonic context. For example, the signals E _i spatially encoded in the field of the subbands are recoded completely in compression, transmitted in a communication network, recovered in a rendering terminal, partially decoded in compression to obtain a representation in the sub domain. -bands. Finally, after these steps, we find substantially the same signals E _i described above, in the terminal. A processing in the field of the subbands of the type expressed by the equation Eq [3] then makes it possible to recover m signals D _j , spatially decoded and ready to be restored after decoding in compression.

Of course, several decoding systems can be arranged in series, depending on the intended application.

$$E_2=X={\displaystyle {\sum }_{j=1}^n\cos \left({\theta }_j\right)S_j}$$

$$E_3=Y={\displaystyle {\sum }_{j=1}^n\sin \left({\theta }_j\right)S_j}$$

For example, in the two-dimensional ambisonic context of order 1, an encoding format with three signals W, X, Y for p sound sources is expressed, for encoding, by: $${\mathrm{<figure-callout\; id="1"\; label="E"\; filenames="imgf0004.png,imgf0006.png"\; state="\{\{state\}\}">E</figure-callout>}}_1=W={\displaystyle {\Sigma }_{j=1}^{\mathrm{not}}{S}_j}$$

$$E_{\mathrm{two}}=X={\displaystyle {\Sigma }_{j=1}^{\mathrm{not}}\cos \left({\theta }_j\right)S_j}$$

$$E_3=Y={\displaystyle {\Sigma }_{j=1}^{\mathrm{not}}\sin \left({\theta }_j\right)S_j}$$

For "ambisonic" decoding with a five-speaker reproduction device in two frequency bands [0, f ₁ ] and [ f ₁ , f ₂ ] with f ₁ = 400 Hz and f ₂ corresponding to a bandwidth of the considered signals, the filters K _ji (f) take the constant numerical values on these two frequency bands, given in Tables I and II below. <b> Table I: </ b> values of coefficients defining filters <i> K </ i>_{<i> ji </ i>}<i> (f) </ i> for 0 <<i> f </ i> ≤ <i> f </ i><sub> 1 </ sub> W X Y 0342 0233 0.000 0268 0382 0505 0268 0382 -0505 0561 -0499 0457 0561 -0499 -0457 W X Y 0383 0372 0.000 0440 0234 0541 0440 0234 -0541 0782 -0553 0424 0782 -0553 -0424

Of course, different spatialization methods (ambisonic context and binaural and / or transaural synthesis) can be combined with a server and / or with a restitution terminal, such spatialization methods respecting the general expression of a linear decomposition of transfer functions in the frequency space, as indicated above.

An implementation of the method in the sense of the invention is described below in an application related to a teleconference between remote terminals.

Referring again to FIG. 4, coded signals ( S _i ) emanate from N remote terminals. They are spatialised at the teleconference server (for example at an audio bridge for a star architecture as shown in FIG. 8) for each participant. This step, performed in the field of sub-bands after a partial decoding phase, is followed by partial recoding. The signals thus coded in compression are then transmitted via the network and, upon receipt by a rendering terminal, are decoded completely in compression and applied to the two left and right lanes 1 and r, respectively, of the rendering terminal, in the case of a binaural spatialization. At the terminals, the compression decoding process thus makes it possible to deliver two left and right temporal signals which contain the information of positions of N distant speakers and which feed two respective loudspeakers (two-ear headphones). Of course, for a general spatialization, for example in the ambisonic context, m channels can be retrieved at the output of the communication server, if the encoding / decoding in spatialization are performed by the server. However, it is advantageous, in a variant, to provide the spatialization encoding with the server and the spatialization decoding with the terminal from the p coded signals in compression, on the one hand, to limit the number of signals to be conveyed via the network (in general p <m) and, secondly, to adapt the spatial decoding to the sound reproduction characteristics of each terminal (for example the number of speakers it contains, or other).

This spatialization can be static or dynamic and, in addition, interactive. Thus, the position of the speakers is fixed or may vary over time. If the spatialization is not interactive, the position of the different speakers is fixed: the listener can not modify it. On the other hand, if the spatialization is interactive, each listener can configure his terminal to position the voice of the other N speakers where it wishes, substantially in real time.

Referring now to FIG. 7, the rendering terminal receives N audio streams (Si) encoded in compression (MPEG, AAC, or others) of a communication network. After partial decoding to obtain the signal vectors (Si), the terminal (" System II") processes these signal vectors to spatialize the audio sources, here in binaural synthesis, in two signal vectors L and R which are then applied to banks synthesis filters for compression decoding. The left and right PCM signals, respectively 1 and r, resulting from this decoding are then intended to directly supply loudspeakers. This type of processing advantageously adapts to a decentralized teleconferencing system (several terminals connected in point-to-point mode).

The following is the case of a " streaming " or a download of a sound scene, particularly in the context of compression coding according to the MPEG-4 standard.

This scene can be simple, or complex as often in the context of MPEG-4 transmissions where the sound scene is transmitted in a structured format. In the MPEG-4 context, the client terminal receives, from a multimedia server, a multiplexed bitstream corresponding to each of the coded primitive audio objects, as well as instructions as to their composition for reconstructing the sound scene. " Audio object " an elementary bit stream obtained by an MPEG-4 Audio encoder. The MPEG-4 System standard provides a special format, called "AudioBIFS" (for "BInary Format for Scene Description"), to convey these instructions. The role of this format is to describe the spatio-temporal composition of audio objects. To build the sound stage and ensure a certain rendering, these different decoded streams can undergo further processing. In particular, a sound spatialization processing step can be performed.

In the "AudioBIFS" format , the operations to be performed are represented by a graph. The decoded audio signals are provided at the input of the graph. Each node of the graph represents a type of processing to be performed on an audio signal. At the output of the graph are provided the different sound signals to be restored or associated with other media objects (images or other).

The algorithms used are updated dynamically and are transmitted with the graph of the scene. They are described as routines written in a specific language such as "SAOL" (for " Structured Audio Score Language"). This language has predefined functions which notably include, and particularly advantageously, FIR and IIR filters (which can then correspond to HRTFs, as indicated above).

In addition, in the audio compression tools provided by the MPEG-4 standard, there are transform coders used mainly for high quality audio transmission. (monophonic and multichannel). This is the case of AAC and TwinVQ encoders based on the MDCT transform.

Thus, in the MPEG-4 context, the tools making it possible to implement the method within the meaning of the invention are already present.

In an MPEG-4 receiver terminal, it is then sufficient to integrate the low decoding layer at the nodes of the upper layer which provides particular processing, such as binaural spatialization by HRTFs filters. Thus, after partial decoding of the demultiplexed elementary audio bit streams and coming from the same type of coder (MPEG-4 AAC for example), the nodes of the "AudioBIFS" graph which involve a binaural spatialization can be processed directly in the field of subbands (MDCT for example). The filter bank synthesis operation is performed only after this step.

In a centralized multipoint teleconferencing architecture as shown in FIG. 8, between four terminals in the example shown, signal processing for spatialization can only be performed at the audio bridge level. Indeed, terminals TER1, TER2, TER3 and TER4 receive streams already mixed and therefore no treatment can be achieved at their level for spatialization.

It is understood that a reduction of the processing complexity is particularly desired in this case. In Indeed, for a conference with N terminals ( N ≥3), the audio bridge must realize a spatialization of the speakers from the terminals for each of the N subsets consisting of (N- 1) speakers among the N participating in the conference. A treatment in the coded domain of course brings more benefit.

Figure 9 shows schematically the processing system provided in the audio bridge. This processing is thus performed on a subset of ( N -1) coded audio signals among the N at the input of the bridge. The left and right coded audio frames in the case of a binaural spatialization, or the m coded audio frames in the case of a general spatialization (for example in ambisonic encoding) as represented in FIG. 9, which result from this processing are thus transmitted to the remaining terminal which participates in the teleconference but which is not included in this subset (corresponding to a "listener terminal ") . In total, N treatments of the type described above are performed in the audio bridge ( N subsets of ( N -1) coded signals). It is indicated that the partial coding of FIG. 9 designates the operation of constructing the coded audio frame after the spatialization processing and to transmit on a channel (left or right). By way of example, it may be a quantization of the L and R signal vectors that result from the spatialization processing, based on a number of bits allocated and calculated according to a selected psychoacoustic criterion. Conventional compression coding treatments after the application of the Analysis filters can therefore be maintained and performed with spatialization in the subband field.

Moreover, as indicated above, the position of the sound source to be spatialised may vary over time, which amounts to varying over time the directional coefficients of the domain of the subbands C _ni and D _ni . The variation of the value of these coefficients is preferentially done in a discrete manner.

Of course, the present invention is not limited to the embodiments described above by way of examples but extends to other variants defined in the context of the claims below.

Claims (26)

1. Method of processing sound data, for spatialized restitution of acoustic signals, in which:

   a) at least one first set C_ni and one second set D_ni of weighting terms, representative of a direction of perception of said acoustic signal by a listener, are obtained for each acoustic signal (Si); and

   b) said acoustic signals are applied to at least two sets of filtering units, disposed in parallel, so as to deliver at least a first output signal (L) and a second output signal (R) each corresponding to a linear combination of the acoustic signals weighted by the collection of weighting terms respectively of the first set (C_ni) and of the second set (D_ni) and filtered by said filtering units,

   characterized in that each acoustic signal in step a) is at least partially compression-coded and is expressed in the form of a vector of subsignals associated with respective frequency subbands,
   and in that each filtering unit is devised so as to perform a matrix filtering applied to each vector, in the frequency subband space.
2. Method according to Claim 1, characterized in that each matrix filtering is obtained by conversion, in the frequency subbands space, of a filter represented by an impulse response in the time space.
3. Method according to Claim 2, characterized in that each impulse response filter is obtained by determination of an acoustic transfer function dependent on a direction of perception of a sound and the frequency of this sound.
4. Method according to Claim 3, characterized in that said transfer functions are expressed by a linear combination of frequency dependent terms weighted by direction dependent terms (Eq[1]).
5. Method according to one of the preceding claims, characterized in that said weighting terms of the first and of the second set depend on the direction of the sound.
6. Method according to Claim 5, characterized in that the direction is defined by an azimuth angle (θ) and an elevation angle (ϕ).
7. Method according to one of Claims 2 and 3, characterized in that the matrix filtering is expressed on the basis of a product matrix involving polyphase matrices (E(z), R(z)) corresponding to analysis and synthesis filter banks and a transfer matrix (S(z)) whose elements are dependent on the impulse response filter.
8. Method according to one of the preceding claims, characterized in that the matrix of the matrix filtering is of reduced form and comprises a diagonal and a predetermined number (δ) of adjacent subdiagonals below and above, whose elements are not all zero.
9. Method according to Claim 8, taken in combination with claim 7, characterized in that the rows of the matrix of the matrix filtering are expressed by:
   [0...S^sb _il (z) ... S^sb _ii (z) ... S^sb _in (z) ... 0] , where:

   - i is the index of the (i+1)th row and lies between 0 and M-1, M corresponding to a total number of subbands,

   - 1 = i-δ mod[M], where δ corresponds to said number of adjacent subdiagonals, the notation mod[M] corresponding to an operation of subtraction modulo M,

   - n = i+δ mod[M], the notation mod[M] corresponding to an operation of addition modulo M,

   - and S^sb _ij(z) are the coefficients of said product matrix involving the polyphase matrices of the analysis and synthesis filter banks and said transfer matrix.
10. Method according to one of Claims 7 to 9, characterized in that said product matrix is expressed by S ^sb(z) = Z^k E (z) S (z) R (z), where

    - Z^k is an advance defined by the term K= (L/M) -1 where L is the length of the impulse response of the analysis and synthesis filters of the filter banks and M the total number of subbands,

    - E(z) is the polyphase matrix corresponding to the analysis filter bank,

    - R(z) is the polyphase matrix corresponding to the synthesis filter bank, and

    - S(z) corresponds to said transfer matrix.
11. Method according to one of claims 7 to 10, characterized in that said transfer matrix is expressed by: $$\mathrm{S}\left(\mathrm{z}\right)=\left[\begin{array}{cccccc}{S}_0\left(z\right)& S_1\left(z\right)& \dots & & & S_{M-1}\left(z\right)\\ z^{-1}{S}_{M-1}\left(z\right)& S_0\left(z\right)& S_1\left(z\right)& \dots & & S_{M-2}\left(z\right)\\ z^{-1}{S}_{M-2}\left(z\right)& z^{-1}{S}_{M-1}\left(z\right)& S_0\left(z\right)& S_1\left(z\right)& \dots & S_{M-3}\left(z\right)\\ ⋮& \ddots & \ddots & \ddots & & ⋮\\ & & & & & S_1\left(z\right)\\ z^{-1}{S}_1\left(z\right)& \dots & & & z^{-1}{S}_{M-1}\left(z\right)& S_0\left(z\right)\end{array}\right],$$

    

    
    where S_k(z) are the polyphase components of the impulse response filter S(z), with k lying between 0 and M-1 and M corresponding to a total number of subbands.
12. Method according to one of Claims 7 to 11, characterized in that said filter banks operate in critical sampling.
13. Method according to one of Claims 7 to 12, characterized in that said filter banks satisfy a perfect reconstruction property.
14. Method according to one of Claims 2 to 13, characterized in that the filter with impulse response is a rational filter, expressed in the form of a fraction of two polynomials.
15. Method according to Claim 14, characterized in that said impulse response is infinite.
16. Method according to one of Claims 8 to 15, characterized in that said predetermined number (δ) of adjacent subdiagonals is dependent on a type of filter bank used in the compression coding chosen.
17. Method according to Claim 16, characterized in that said predetermined number (δ) is between 1 and 5.
18. Method according to one of Claims 7 to 17, characterized in that the matrix elements ( L _n, R _n) resulting from said product matrix are stored in a memory and reused for all partially coded acoustic signals to be spatialized.
19. Method according to one of the preceding claims, characterized in that it furthermore comprises a step d) consisting in applying a synthesis filter bank to said first (L) and second output signals (R), before their rendering.
20. Method according to Claim 19, characterized in that it furthermore comprises a step c) prior to step d) consisting in conveying the first and second signals into a communication network, from a remote server and to a rendering device, in coded and spatialized form, and in that step b) is performed at said remote server.
21. Method according to Claim 19, characterized in that it furthermore comprises a step c) prior to step d) consisting in conveying the first and second signals into a communication network, from an audio bridge of a multipoint teleconferencing system, of centralized architecture, and to a rendering device of said teleconferencing system, in coded and spatialized form, and in that step b) is performed at said audio bridge.
22. Method according to Claim 19, characterized in that it furthermore comprises a step subsequent to step a) consisting in conveying said acoustic signals in compression-coded form into a communication network, from a remote server and to a rendering terminal, and in that steps b) and d) are performed at said rendering terminal.
23. Method according to one of the preceding claims, characterized in that a sound spatialization by binaural synthesis based on a linear decomposition of acoustic transfer functions is applied in step b).
24. Method according to Claim 23, characterized in that a matrix of filters of gains (G _i) is furthermore applied, in step b), to each partially coded acoustic signal (Si) ,
    in that said first and second output signals are intended to be decoded into first and second rendered signals (1, r),
    and in that the application of said matrix of gain filters amounts to applying a chosen time delay (ITD) between said first and second restitution signals.
25. Method according to one of Claims 1 to 22, characterized in that, in step a), more than two sets of weighting terms are obtained, and in that, in step b), more than two sets of filtering units are applied to the acoustic signals so as to deliver more than two output signals comprising encoded ambisonic signals.
26. System for processing sound data, characterized in that it comprises means for the implementation of the method according to one of the preceding claims.

Images