EP3559947B1 - Processing in sub-bands of an actual ambisonic content for improved decoding - Google Patents

Description

The present invention relates to the field of audio or acoustic signal processing, and more particularly to the processing of real multi-channel sound content in surround sound (or “ambisonic” format hereinafter). In the state of the prior art, ambisonic treatment methods are known: M. Baqué, A. Guérin and M. Melon "Source separation applied to ambisonic content: localization and extraction of direct fields", French Congress of Acoustics and the 20th conference Vibrations, SHocks and NOise, CFA / VISHNO 2016, 1 April 2016 (2016-04-01), pages 1-6 , Le Mans, describes the use of a different mixing matrix according to the frequency sub-bands. The patent application EP2866475A1 describes a similar technique.

• La séparation de sources sonores :
  • ∘ Pour le divertissement (karaoké : suppression de la voix),
  • ∘ Pour la musique (mixage des sources séparées dans un contenu multicanal),
  • ∘ Pour les télécommunications (rehaussement de la voix, débruitage),
  • ∘ Pour la domotique (commande vocale),
  • ∘ Le codage audio multicanal.
• Le décodage pour une diffusion multicanal :
  • ∘ Pour le cinéma,
  • ∘ Pour la musique,
  • ∘ Pour la réalité virtuelle.

The ambisonic technique consists in using a subset of channels in each frequency band which have the desired directivity characteristics. By way of example of application, one can quote:

• Separation of sound sources:
  • ∘ For entertainment (karaoke: voice suppression),
  • ∘ For music (mixing separate sources into multi-channel content),
  • ∘ For telecommunications (voice enhancement, noise reduction),
  • ∘ For home automation (voice control),
  • ∘ Multichannel audio coding.
• Decoding for multichannel broadcasting:
  • ∘ For cinema,
  • ∘ For music,
  • ∘ For virtual reality.

est l'harmonique sphérique d'ordre m et d'indice nσ, dépendant des coordonnées sphériques (θ, φ), définie avec la formule suivante : $$Y_{\mathit{mn}}^{\sigma }\left(\theta ,\varphi \right)={\tilde{P}}_{\mathit{mn}}\left(\cos \varphi \right).\{\begin{array}{ll}\cos \mathit{n\theta }& \mathrm{si}\sigma =1\\ \sin \mathit{n\theta }& \mathrm{si}\sigma =-1\mathrm{et}n\ge 1\end{array}$$

où P̃_mn (cos φ) est une fonction pôlaire impliquant le polynome de Legendre : $${\tilde{P}}_{\mathit{mn}}\left(\mathrm{x}\right)=\sqrt{{\epsilon }_n\frac{\left(m-n\right)!}{\left(m+n\right)!}}{\left(-1\right)}^n{\left(1-{\cos }^{2}x\right)}^{\frac{n}{2}}\frac{d^n}{{\mathit{dx}}^n}{P}_m\left(x\right)\mathrm{avec}{\epsilon }_0=1\mathrm{et}{\epsilon }_0=0\mathrm{pour}n\ge 1$$

et $$P_m\left(x\right)=\frac{1}{2^m.m!}\frac{d^n}{{\mathit{dx}}^n}{\left(x^2-1\right)}^m$$

Ambisonia consists of a projection of the acoustic field on a basis of spherical harmonic functions (basis illustrated on figure 1 ), to obtain a spatial representation of the sound scene. Function $Y_{\mathit{mn}}^{\sigma }\left(\theta ,\varphi \right)$

is the spherical harmonic of order m and index nσ, depending on the spherical coordinates ( θ , φ ), defined with the following formula: $$Y_{\mathit{mn}}^{\sigma }\left(\theta ,\varphi \right)={\tilde{P}}_{\mathit{mn}}\left(\cos \varphi \right).\{\begin{array}{ll}\cos \mathit{n\theta }& \mathrm{if}\sigma =1\\ \sin \mathit{n\theta }& \mathrm{if}\sigma =-1\mathrm{and}\mathrm{not}\ge 1\end{array}$$

where P̃ _mn (cos φ ) is a polar function involving the Legendre polynomial: $${\tilde{P}}_{\mathit{mn}}\left(\mathrm{x}\right)=\sqrt{{\epsilon }_{\mathrm{not}}\frac{\left(m-\mathrm{not}\right)!}{\left(m+\mathrm{not}\right)!}}{\left(-1\right)}^{\mathrm{not}}{\left(1-{\cos }^{2}x\right)}^{\frac{\mathrm{not}}{2}}\frac{d^{\mathrm{not}}}{{\mathit{dx}}^{\mathrm{not}}}{P}_m\left(x\right)\mathrm{with}{\mathrm{<figure-callout\; id="0"\; label="\epsilon "\; filenames="imgf0001.png,imgf0002.png"\; state="\{\{state\}\}">\epsilon </figure-callout>}}_0=1\mathrm{and}{\mathrm{<figure-callout\; id="0"\; label="\epsilon "\; filenames="imgf0001.png,imgf0002.png"\; state="\{\{state\}\}">\epsilon </figure-callout>}}_0=0\mathrm{for}\mathrm{not}\ge 1$$

and $$P_m\left(x\right)=\frac{1}{2^m.m!}\frac{d^{\mathrm{not}}}{{\mathit{dx}}^{\mathrm{not}}}{\left(x^2-1\right)}^m$$

In the representation of the figure 1 , the first "vector" of the base of spherical harmonics (at the top of the figure 1 ) corresponds to the order m = 0, the three "vectors" in the following line correspond to the order m = 1 (oriented in the three directions of space), and so on.

In practice, a real ambisonic encoding is done from a network of sensors, generally distributed on a sphere, which are combined to synthesize an ambisonic content whose channels best respect the directivities of the spherical harmonics (as illustrated on figure 2 ). With reference to the figure 2 , a MIC microphone comprises a plurality of piezoelectric capsules C1, C2, ... which receive sound waves in different directions of arrival in space. A processing unit UT receiving the signals coming from these capsules performs ambisonic encoding using a matrix of filters presented below, and delivers ambisonic signals (formalized in a base of spherical harmonics of the type illustrated on figure 1 ).

The basic principles of ambisonic encoding are described below.

The ambisonic formalism, initially limited to the representation of spherical harmonic functions of order 1, was subsequently extended to higher orders. The ambisonic formalism with a larger number of components is commonly called “ Higher Order Ambisonics ” (or “HOA” hereafter).

To each m order correspond 2m + 1 spherical harmonic functions, as illustrated on figure 1 . Thus, an M-order content contains a total of (M + 1) ² channels (4 channels in order 1, 9 channels in order 2, 16 channels in order 3, and so on).

• une composante ambisonique pour l'ordre m=0,
• trois composantes ambisoniques pour l'ordre m=1,
• cinq composantes ambisoniques pour l'ordre m=2,
• sept composantes ambisoniques pour l'ordre m=3, etc.

The term “ambisonic components” is understood hereinafter to mean the ambisonic signal in each ambisonic channel, with reference to the “vector components” in a vector base which would be formed by each spherical harmonic function. So for example, we can count:

• an ambisonic component for the order m = 0,
• three ambisonic components for the order m = 1,
• five ambisonic components for the order m = 2,
• seven ambisonic components for the order m = 3, etc.

The ambisonic signals picked up for these different components are then distributed over a number N of channels which is deduced from the maximum order m that it is intended to pick up in the sound scene. For example, if a soundstage is picked up with a 20 piezoelectric capsule ambisonic microphone, then the maximum ambisonic order picked up is M = 3, so that there is no more than 20 channels N = (M + 1) ² , the number of ambisonic components considered is 7 + 5 + 3 + 1 = 16 and the number N of channels is N = 16, given elsewhere by the relation N = (M + 1) ² , with M = 3.

The ambisonic capture x (t) of order M and composed of N sound sources s _i of incidence ( θ _i , φ _i ) propagating in a free field can then be written mathematically in the following matrix form: $$x\left(t\right)=\mathit{Ace}\left(t\right)=\left[\begin{array}{ccc}1& \cdots & 1\\ ⋮& \ddots & ⋮\\ Y_{\mathit{<figure-callout\; id="1"\; label="Mn\; \sigma \; \theta "\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">Mn\; </figure-callout>}}^{\sigma }\left({\theta }_{}\right)\left({}_1,{\mathrm{<figure-callout\; id="1"\; label="\phi "\; filenames="imgf0002.png,imgf0003.png"\; state="\{\{state\}\}">\varphi </figure-callout>}}_1\right)& \cdots & Y_{\mathit{Mn}}^{\sigma }\left({\theta }_{\mathrm{NOT}},{\varphi }_{\mathrm{NOT}}\right)\end{array}\right]\mathrm{s}\left(\mathrm{t}\right)$$

Where A is a so-called “mixing matrix”, of dimensions (M + 1) ² x N and of which each column A _i contains the mixing coefficients of the source i .

Physically, this matrix A corresponds to the encoding coefficients of each source i, associated with each direction of each source i . To extract the sources of such content, it is necessary to estimate a matrix B called the “separation matrix”, the inverse of matrix A. To obtain matrix B, a step of blind separation of sources can be performed. implementation, for example using an independent component analysis algorithm (or “ACI” hereinafter), or else a principal component analysis algorithm. The matrix B = A ^-1 allows the extraction of the sources by the following operation:
s ( t ) = Bx ( t )

This step amounts to forming channels (or “beamforming” hereafter), that is to say to combining different channels having distinct directivities, in order to create a new component having the desired directivity. An example of beamforming to extract three components, for HOA content of order 2, 4 or 6, is shown figure 3 . The higher the order, the more directional the beamforming and the higher the number of components that can be extracted.

In practice, the generation of the ambisonic signals x (t) = As (t) goes through an intermediate stage of microphone capture as illustrated on figure 2 , where the sources s (t) are picked up by the capsules of the microphone MIC to form the signals p1, p2, p3 ... We then formalize the microphone encoding matrix E such that x (t) = Ep (t), to obtain the ambisonic components x1, x2, ..., xN (in N ambisonic channels as shown in figure 4 ). Referring now to the figure 4 , we estimate, as presented above, the inverse decoding matrix B of the matrix A, to determine the source signals s1, s2, s3:
s ( t ) = Bx ( t )

To decode HOA content on a speaker system, the process is similar. We acquire ambisonic signals in N channels x1, x2, ..., xN, but, here, instead of considering s (t) as the sum of the contributions of sources, we consider s (t) as the sum of the transmitted signals by a set of loudspeakers (which then makes it possible to supply these loudspeakers effectively with the signals s1, s2, s3, etc.). The decoding matrix B is therefore formulated here from the positions of the loudspeakers of a sound reproduction system and the signals intended for the loudspeakers are extracted according to the same process as that used for the separation of sources.

In reality, the sensors used have physical limitations which lead to a degradation of the microphone encoding, and therefore a degradation of the directivity of the ambisonic components. For example, the encoding of high frequencies deteriorates when the inter-sensor spacing becomes approximately greater than half a wavelength: this is due to the phenomenon of spatial aliasing. At low frequencies, the microphone capsules tend to become omnidirectional and it becomes impossible to obtain the desired directivities. More precisely, the degradations at low frequencies are more marked when it comes to synthesizing high order ambisonic components. In general, the associated directivities are more complex and therefore more sensitive to variations in the properties of the sensors. The figure 5 illustrates the degree of correlation between theoretical encoding and actual encoding from a 32 capsule spherical microphone, as a function of frequency and ambisonic order. The figure 5 shows that the highest degree of correlation is generally achieved for frequencies between 1 kHz and 10 kHz. Nevertheless, for the other frequency ranges (except for the ambisonic orders 0 and 1), the extraction of sources would not always lead to the same result for a theoretical encoding and for a real encoding of these same sources. More precisely, for frequencies outside the interval [1 kHz-10 kHz], the extracted components are potentially degraded.

The figure 6 shows the real directivity in the horizontal plane of the first components of orders 0, 1, 2 and 3 as a function of the sound frequency. It appears, on the figure 6 , that the real components are not properly encoded. Indeed, if we take the example of the component of order 0 at the frequency of 10 kHz, we see that it is not circular, unlike the theoretical component and the same component calculated at frequencies between 300 and 1000Hz. Thus, the directivity of this component at the frequency of 10 kHz is no longer respected, which could induce a degraded spatial rendering. Furthermore, the components at order 1, 2 and 3 also have biased directivities for frequencies lower than 10 kHz.

More generally, once the theoretical directivity is no longer respected, the beamforming carried out no longer makes it possible to properly extract the desired components. For example, this results in the appearance of interference during source separation. This can also result in a degradation of the spatial rendering in frequency bands concerned by multichannel broadcasting. More particularly, there is a loss of energy at low frequencies in high orders during encoding. This implies that the sources extracted using high order channels can lose part of their energy in the frequencies concerned.

The use of beamforming for the separation of sources or the restitution of an ideal ambisonic content or of a multichannel capture is already used in particular for the separation, or for the multichannel decoding. For source separation, an inversion of the mixing matrix estimated by independent component analysis is used to extract the sources. For multichannel decoding, the matrix of ambisonic coefficients relating to the loudspeakers can be inverted. In contrast, the processing of actual ambisonic content, affected by the physical limitations of the recording system, is not addressed in the prior art. The only solution currently proposed is to limit the total bandwidth of the extracted sources, which is not satisfactory.

The present invention improves this situation.

• un filtrage fréquentiel des composantes ambisoniques dans une pluralité de bandes de fréquences,
• une élaboration d'une matrice de décodage ambisonique,
• un traitement de la matrice de décodage ambisonique pour extraire, par réduction de dimension de matrice, une pluralité de sous-matrices de décodage ambisonique chacune associée à un ordre ambisonique et à une bande de fréquences choisie pour cet ordre ambisonique,
• des applications respectives des sous-matrices de décodage aux composantes ambisoniques dans chaque bande de fréquences choisie, et une reconstruction bandes-à-bandes des résultats desdites applications respectives, pour délivrer une pluralité de signaux décodés, associés chacun à une source sonore.

For this purpose, it proposes a method, implemented by computer means, for processing an ambisonic content comprising a plurality of ambisonic components of a plurality of orders defining a succession of ambisonic channels in each of which an ambisonic component is represented. , the process comprising:

• frequency filtering of the ambisonic components in a plurality of frequency bands,
• development of an ambisonic decoding matrix,
• processing of the ambisonic decoding matrix to extract, by reduction of the matrix dimension, a plurality of ambisonic decoding sub-matrices each associated with an ambisonic order and with a frequency band chosen for this ambisonic order,
• respective applications of the decoding sub-matrices to the ambisonic components in each chosen frequency band, and a band-to-band reconstruction of the results of said respective applications, to deliver a plurality of decoded signals, each associated with a sound source.

• une source sonore effectivement identifiée et localisée dans l'espace tridimensionnel (en technique d'extraction de source), auquel cas la matrice de décodage est une matrice de séparation de sources, ou
• un haut-parleur parmi plusieurs haut-parleurs, de position bien identifiée dans l'espace, et alimenté en particulier par l'un des signaux décodés précités.

By "sound source" is meant here as well:

• a sound source effectively identified and located in three-dimensional space (in source extraction technique), in which case the decoding matrix is a source separation matrix, or
• a loudspeaker among several loudspeakers, with a clearly identified position in space, and supplied in particular by one of the aforementioned decoded signals.

A frequency band can be defined by several frequency bands or frequency subbands.

The development of ambisonic decoding sub-matrices for each frequency band, and for each ambisonic order, makes it possible to take advantage in each frequency band of a maximum number of ambisonic channels which are really valid in each sub-matrix, in order to to restore a decoded signal with little or no degradation.

According to one embodiment, each ambisonic decoding sub-matrix is associated with a frequency band chosen as a function of a criterion of validity of the ambisonic components of the order with which said sub-matrix is associated, in said chosen frequency band.

Such an embodiment makes it possible to isolate the ambisonic components constituting each order, in order to process them in the frequency range in which they are valid. By “valid” is meant compliance with the theoretical ambisonic representation, such as the order m = 4 in the frequency band 4000 to 6000 Hz in the example of figure 5 , or the order m = 3 in the frequency band 2000 to 9000 Hz.

Thus, in one embodiment, the component validity criterion can be defined by conditions for capturing said ambisonic components, by at least one ambisonic microphone.

• une réception de données d'au moins un microphone ambisonique utilisé pour capter lesdites composantes ambisoniques ;
• une détermination des bandes de fréquences choisies pour construire lesdites sous-matrices, en fonction desdites données de microphone ambisonique.

In this embodiment, for example, the method may further comprise:

• receiving data from at least one ambisonic microphone used to pick up said ambisonic components;
• determining the frequency bands chosen to construct said sub-matrices, as a function of said ambisonic microphone data.

Knowledge of the data of the ambisonic microphone used for ambisonic capture makes it possible to refine the determination of the frequency bands chosen for the development of the sub-matrices. In fact, the ambisonic processing is carried out on sub-matrices whose ambisonic components strictly meet the validity criterion in the associated frequency bands. However, the data from the ambisonic microphone used for recording is not always accessible. As a variant, it is therefore possible to provide for the determination of the frequency bands with the aid of an abacus previously established from measurements carried out on a plurality of ambisonic microphones, in order to establish “average” frequency ranges associated with a ambisonic order, in which the ambisonic components of each ambisonic order generally meet the aforementioned validity criterion.

• une bande de fréquences peut être choisie dans une plage de 100Hz à 10kHz pour l'ordre ambisonique m=1,
• une bande de fréquences peut être choisie dans une plage de 500Hz à 10kHz pour l'ordre ambisonique m=2,
• une bande de fréquences peut être choisie dans une plage de 2000Hz à 9000Hz pour l'ordre ambisonique m=3,
• une bande de fréquences peut être choisie dans une plage de 3000Hz à 7000Hz pour l'ordre ambisonique m=4.

Thus, according to one embodiment, each ambisonic decoding sub-matrix being associated with an ambisonic order and with a frequency band chosen for this ambisonic order,

• a frequency band can be chosen in a range of 100Hz to 10kHz for the ambisonic order m = 1,
• a frequency band can be chosen in a range of 500Hz to 10kHz for the ambisonic order m = 2,
• a frequency band can be chosen in a range of 2000Hz to 9000Hz for the ambisonic order m = 3,
• a frequency band can be chosen in a range of 3000Hz to 7000Hz for the ambisonic order m = 4.

In an embodiment where the frequency bands are obtained by short-term Fourier transform (FFT), a frequency band associated with an ambisonic order can include several FFT frequency bands. Thus, several frequency bands can be associated with an ambisonic order.

In an example of this embodiment where an FFT is used, for a signal sampled at 48kHz and for an FFT size of 4096 points (2 ¹² ), the bands No. 10 to 910 correspond to the frequency band 100 to 10kHz and are associated with the ambisonic order m = 1.

Thus, it turns out that it is possible to define a validity criterion on the basis of average values of the frequency bands for each ambisonic order, even if the data of the ambisonic microphone used for the capture of ambisonic components are inaccessible.

• une inversion de la matrice élaborée de décodage ambisonique, pour obtenir une matrice de mélange dont :
  • * les lignes correspondent à des canaux ambisoniques respectifs, et
  • * les colonnes correspondant à des sources sonores,
• un traitement de la matrice de mélange pour extraire, par réduction de dimension de matrice, une pluralité de sous-matrices de mélange chacune associée à un ordre ambisonique et à une bande de fréquences choisie, et
• une inversion des sous-matrices de mélange pour obtenir respectivement lesdites sous-matrices de décodage ambisonique.

According to a particular embodiment, the processing of the ambisonic decoding matrix comprises:

• an inversion of the elaborate ambisonic decoding matrix, to obtain a mixing matrix including:
  • * the lines correspond to respective ambisonic channels, and
  • * the columns corresponding to sound sources,
• a processing of the mixing matrix to extract, by reduction in matrix size, a plurality of mixing sub-matrices each associated with an ambisonic order and with a chosen frequency band, and
• an inversion of the mixing sub-matrices to respectively obtain said ambisonic decoding sub-matrices.

It is thus understood that a frequency filtering of the components of order m = 4 between 4000 to 6000 Hz, in the example of figure 5 , makes it possible to construct a submatrix, in particular of mixing (matrix denoted A above), at N = (m + 1) ² = 25 lines, by retaining the first 25 ambisonic channels. Nevertheless, for this purpose, it is preferable that the ambisonic signal is sufficiently represented in this 4-6 kHz frequency band, as will be seen below. Moreover, if the ambisonic signal is also well represented in the low frequencies, for example between 100 and 200 Hz, it is also possible to construct a sub-matrix for the order m = 1 for example, at N = 4 lines. It is therefore finally possible to obtain a plurality of mixing sub-matrices, each associated with an ambisonic order m, and each comprising a number of lines corresponding to a number of ambisonic channels valid for this order m and in the frequency band at which this sub-matrix is associated.

In one embodiment, the processing of the ambisonic content is carried out for source separation and said decoding matrix is a source blind separation matrix constructed from the ambisonic components.

For example, the separation matrix can be produced from ambisonic components filtered at a chosen frequency band and preferably in which the number of ambisonic channels valid according to the aforementioned criterion is maximum.

Thus, the channels are retained for an accuracy of representation at such a highest ambisonic order, but also to keep a maximum of channels represented correctly in this frequency band, at lower ambisonic orders.

In this embodiment, it is possible to simplify the mixing sub-matrices before their inversion, by reducing a number of columns of each sub-matrix, the remaining columns of the sub-matrices being chosen so as to conserve signals of greater energies. after application of the decoding sub-matrices.

In fact, keeping the signals of greater energy makes it possible to better represent, and therefore to better restore, the sound field.

In addition or as a variant, it is possible to choose to favor the most decorrelated extracted signals, or the most independent according to a chosen independence criterion.

Thus, in this embodiment, mixing sub-matrices are simplified before their inversion, by reducing a number of columns of each sub-matrix, the remaining columns of the sub-matrices being chosen so as to keep the least correlated signals. after application of the decoding sub-matrices.

Furthermore, in a reverberant environment, the signal is made up of direct fields resulting from the equivalent “free field” propagation of each source and reflections on the walls of the acoustic environment. Thus, in an alternative or complementary embodiment, mixing sub-matrices are simplified before their inversion, by reducing a number of columns of each sub-matrix, the remaining columns of the sub-matrices being chosen so as to keep signals. corresponding to direct sound fields after application of the decoding sub-matrices.

Of course, in an embodiment where the processing of the ambisonic content is carried out for ambisonic reproduction on a plurality of loudspeakers, the aforementioned decoding matrix can be an inverse matrix of the relative spatial positions of the loudspeakers.

• Pour chaque ordre ambisonique du contenu, une détermination d'une bande de fréquences sur laquelle ledit ordre respecte un critère de validité prédéterminé d'encodage ambisonique,
• Sur la base desdites bandes de fréquences, une application d'un banc de filtres au contenu ambisonique pour produire une pluralité de signaux en sous-bandes, de dimensions variables correspondant à des canaux ambisoniques valides dans cette sous-bande,
• Une détermination d'une matrice de décodage de taille maximale dans la bande de fréquence de l'ordre ambisonique maximal et d'une matrice de mélange associée, inverse ou pseudo-inverse de ladite matrice de décodage,
• Pour chaque autre bande de fréquences, une détermination d'une matrice de mélange de taille réduite, sous-matrice de ladite matrice de mélange, et d'une sous-matrice de séparation, inverse ou pseudo-inverse de ladite sous-matrice de mélange,
• Une reconstruction des signaux séparés pleine-bande par application d'un banc de filtre de synthèse aux signaux séparés issus de la multiplication desdits signaux par lesdites matrices.

In an embodiment illustrated below with reference to figure 9 , the method comprises in particular, for an ambisonic content broken down into frequency sub-bands, an application of decoding sub-matrices, obtained by:

• For each ambisonic order of the content, a determination of a frequency band on which said order meets a predetermined validity criterion of ambisonic encoding,
• On the basis of said frequency bands, an application of a bank of filters with ambisonic content to produce a plurality of signals in sub-bands, of variable dimensions corresponding to ambisonic channels valid in this sub-band,
• A determination of a decoding matrix of maximum size in the frequency band of the maximum ambisonic order and of an associated mixing matrix, inverse or pseudo-inverse of said decoding matrix,
• For each other frequency band, a determination of a mixing matrix of reduced size, sub-matrix of said mixing matrix, and of a separation, inverse or pseudo-inverse sub-matrix of said mixing sub-matrix ,
• A reconstruction of the full-band separated signals by applying a synthesis filter bank to the separated signals resulting from the multiplication of said signals by said matrices.

The present invention is also aimed at a computer program comprising instructions for implementing the method when this program is executed by a processor. An example of a flowchart of the general algorithm of such a program is shown in Figure figure 7 commented on below, which is specified in the figures 8 and 9 .

• une interface d'entrée pour recevoir des signaux de composantes ambisoniques,
• une interface de sortie pour délivrer des signaux décodés, associés chacun à une source sonore,
• et un programme informatique pour la mise en œuvre du procédé.

The present invention also relates to a computer device comprising:

• an input interface to receive signals of ambisonic components,
• an output interface for delivering decoded signals, each associated with a sound source,
• and a computer program for implementing the method.

An example of such a device is illustrated on figure 10 discussed below.

The present invention thus proposes to use the formation of channels from a real ambisonic encoding by taking advantage, in each frequency band, of all the channels whose directivity respects the ambisonic formalism. An embodiment presented above then makes it possible to determine one or more mixing matrices Ak, corresponding to sub-matrices obtained from the theoretical matrix A, and each formulated in a frequency band, then inverted to give matrices of Bk decoding.

Thus, the invention offers a generic processing of any ambisonic content, and in particular real, possibly affected by the physical limitations of a recording system, and this without any constraint aiming to limit the total bandwidth of the extracted sources. .

• la figure 1 illustre une base de fonctions harmoniques sphériques d'ordre 0 (première ligne) à 3 (dernière ligne), avec en gris clair les valeurs positives, et en gris foncé les valeurs négatives,
• la figure 2 illustre un système d'encodage ambisonique à partir d'un microphone sphérique,
• la figure 3 illustre la formation de voies pour l'extraction de trois composantes, pour différents ordres ambisoniques,
• la figure 4 illustre très schématiquement un système de décodage ambisonique à partir de composantes ambisoniques,
• la figure 5 illustre la corrélation entre un encodage ambisonique idéal et un encodage réel,
• la figure 6 illustre la directivité dans le plan horizontal, mesurée pour un encodage ambisonique réel (avec de gauche à droite successivement les composantes des ordres 0, 1, 2 et 3),
• la figure 7 illustre les principales étapes d'un exemple de procédé au sens de l'invention,
• la figure 8 illustre les étapes d'un mode de réalisation particulier du procédé selon l'invention,
• la figure 9 est un schéma-bloc d'un algorithme de traitement correspondant au mode de réalisation illustré sur la figure 7, et
• la figure 10 illustre schématiquement un dispositif possible pour la mise en œuvre de l'invention.

Other advantages and characteristics of the invention will become apparent on reading the detailed description below of exemplary embodiments of the invention, and on examining the appended drawings in which:

• the figure 1 illustrates a base of spherical harmonic functions of order 0 (first line) to 3 (last line), with positive values in light gray, and negative values in dark gray,
• the figure 2 illustrates an ambisonic encoding system from a spherical microphone,
• the figure 3 illustrates the formation of pathways for the extraction of three components, for different ambisonic orders,
• the figure 4 very schematically illustrates an ambisonic decoding system from ambisonic components,
• the figure 5 illustrates the correlation between an ideal ambisonic encoding and a real encoding,
• the figure 6 illustrates the directivity in the horizontal plane, measured for a real ambisonic encoding (with successively from left to right the components of orders 0, 1, 2 and 3),
• the figure 7 illustrates the main steps of an example of a method within the meaning of the invention,
• the figure 8 illustrates the steps of a particular embodiment of the method according to the invention,
• the figure 9 is a block diagram of a processing algorithm corresponding to the embodiment illustrated in figure 7 , and
• the figure 10 schematically illustrates a possible device for implementing the invention.

The overall diagram of a global ambisonic treatment method within the meaning of the invention is presented figure 7 . This is for example an ambisonic decoding process. The terms “ambisonic decoding” are understood to mean both the supply of decoded signals, for example intended to supply respective loudspeakers for surround sound reproduction, and more generally the supply of signals each associated with a sound source, especially in the source separation technique.

In step S1, there is an ambisonic content x (t) comprising a plurality of ambisonic components CA, of successive orders m = 0, 1, ..., M (with for example M = 4) and, from a recording, or a “capture”, by at least one ambisonic microphone MIC. An ambisonic microphone is a microphone made up of a plurality of microphone capsules generally distributed in a spherical manner and as regularly as possible. These capsules act as sound signal sensors. The microphone capsules are arranged on the ambisonic microphone so as to pick up sound signals according to their directionality in space. As shown on the figure 5 , the set of capsules forming such an ambisonic microphone can acquire different ambisonic components at ambisonic orders up to M, but the accuracy of the ambisonic representation for these different orders is not really respected for all the frequencies of the audio spectrum between 0 and 20kHz. Nevertheless, the invention proposes here to isolate certain frequencies of the spectrum for which the components ambisonics, for given orders, are exact (as for example in the frequency range between 4000 and 6000Hz for the order m = 4 on the figure 5 , or more broadly the range between 2000Hz and 9000Hz for the order m = 3, etc.).

Nevertheless, the frequency variations in the accuracy of ambisonic representation of each order of the figure 5 are obtained for a particular microphone having dimensions and a given number of capsules. Thus, for another microphone, other spectral variations can be expected.

Step S2 therefore aims to recover the data characterizing the ambisonic microphone MIC (and possibly the conditions for capturing the ambisonic content c (t), and / or the reverberation conditions during capturing, or the like).

More generally, a data characterizing the ambisonic microphone MIC can be the inter-capsule spacing. In fact, the encoding of the high frequencies is degraded when the inter-sensor spacing becomes greater than half a wavelength. This is due to the phenomenon of spatial aliasing (or “aliasing”). Conversely, for a low frequency signal, microphone capsules that are too close together cannot generate the desired directivity.

In step S3, a BFA analysis filter bank can be applied to the ambisonic content x (t) in order to then select, in step S31, ambisonic component signals filtered in frequency ranges in which the representation ambisonic for a given order m is the most exact (thus respecting a “validity criterion” of the ambisonic representation), and this according to the microphone data defined above.

Depending on the type of processing applied to the ambisonic content x (t), between a processing for separation of SAS sources or a processing with a view to restitution on RES loudspeakers, step S4 aims to obtain a matrix decoding B, depending on the type of processing chosen. In the case of ambisonic reproduction on loudspeakers, the decoding matrix B is the inverse of a matrix A containing coefficients specific to the spatial positions of the loudspeakers used for the reproduction.

In the case of source separation, the decoding matrix B is initially produced in step S4 with a view to blind source separation processing from the filtered and selected ambisonic components. More particularly, this decoding matrix B is produced for the frequency band containing the greatest number of valid ambisonic channels (and the greatest order likely to be obtained M).

The determination of the frequency bands of validity of the various ambisonic orders can be adapted to the ambisonic microphone used to capture the ambisonic components to be decoded. To do this, it is possible for example to base oneself on the frequency variations of the accuracy of the ambisonic representation for different orders m, of the type illustrated on the figure 5 .

More generally, it can still be determined an "average" rate of the frequency variations of the accuracy of the ambisonic representation for the different orders m for different models of ambisonic microphones, and use these average rates if these data are not available. , when decoding.

In step S7, at least two matrices B1, B2, resulting from a matrix reduction of the decoding matrix B for each frequency sub-band (in the example illustrated, the frequency sub-bands f1 and f2 ). A more precise embodiment of this matrix reduction will be described later with reference to the figure 8 . Then, in step S8, the product of each matrix B1 and B2 obtained in the previous step is carried out by the ambisonic signals filtered in the corresponding sub-bands f1, f2. Thus, in each sub-band k (k = 1.2), a set of extracted signals sk is obtained.

In step S9, the vectors of extracted signals s1 (1 for k = 1) and s2 (2 for k = 2) are combined in order to obtain the reconstructed full-band signals (by application for example of a filter bank of synthesis).

The figure 8 illustrates the steps of a particular embodiment of the method according to the invention. More precisely, the figure 8 presents process steps which can be implemented between steps S4 and S7 of the figure 7 .

In step S4, as described above, the decoding matrix B defined above is obtained. In step S5, it is possible to perform an inversion of this decoding matrix B (or in an equivalent manner, a determination of its pseudo-inverse) in order to obtain the corresponding mixing matrix A (step S51). In the case of source separation, the mixing matrix A can thus contain coefficients relating to respective positions of sound sources to be extracted. In the case of reproduction on loudspeakers, the mixing matrix A can contain coefficients relating to the position of the loudspeakers on which it is desired to reproduce the decoded signals. More precisely, the rows of the mixing matrix A correspond to the successive ambisonic channels (successively defining the orders m = 0 to m = M, where M is the maximum ambisonic order available) and its columns correspond to the sources or to the loudspeakers .

• C11, C12, C13,
• C21, C22, C23,
• C31, C32, C33, et
• C41, C42, C43.

In step S6, it is possible to reduce the dimensions of the mixing matrix A, to obtain sub-matrices A1, A2. This is a matrix reduction whose number of lines corresponds to the number of ambisonic channels for each order. Typically, if the ambisonic signals are well encoded in the 100 to 1000Hz band, where the order m = 1 is well respected (at least for the ambisonic microphone of the figure 5 ), a sub-matrix A1 with N = 4 lines associated with the order m = 1 and with the frequency band 100-1000Hz is already extracted from the matrix A. Then, if the ambisonic signals are well represented in the band from 1000 to 10 000Hz, where the order m = 2 is well respected, it is then extracted from the matrix A a matrix A2 with N = 9 lines and associated with the order m = 2 and at the frequency band 1000-10 000Hz, and so on. The number of sub-matrices thus depends on the order of the ambisonic content x (t) whose components are retained as valid in step S31. Each sub-matrix then corresponds to a frequency band, and can thus contain a number of lines corresponding to the number of valid channels for this frequency band. Specifically, as illustrated on figure 8 , for each sub-band, the corresponding number of valid channels is identified. For example, for a sub-band f1 chosen for the order m = 1 of the ambisonic content x (t), one extracts a matrix A1 comprising four lines (N1 = (m + 1) ² ) corresponding to the four ambisonic channels at l order 1, and the number of “sources” (sources to be extracted or loudspeakers) in columns. As shown on the figure 8 , the four lines retained for the construction of the sub-matrix A1 are the coefficients of the total initial matrix A:

• C11, C12, C13,
• C21, C22, C23,
• C31, C32, C33, and
• C41, C42, C43.

• C91, C92, C93.

Concerning the sub-matrix A2, these rows of the global matrix A can be taken, as well as the following ones, up to the row:

• C91, C92, C93.

For the mixing matrix A2, corresponding to order 2 of the ambisonic content x (t), and therefore to the sub-band f2, we therefore keep nine lines, corresponding to the nine channels of order 2, and the number of sources to extract in columns.

Each mixing sub-matrix thus obtained is of dimension N x Ntarget, with Ntarget the number of sources resulting from the separation of blind sources or the number of loudspeakers provided for a reproduction.

In the case of playback on loudspeakers, the number of loudspeakers is preferably equal to or greater than the number of lines. For example, for the mixture matrix A1 of four rows, it is possible to keep only a set of four columns. In the case of source separation, the number of columns may be less than or equal to the number of rows. For example, for the mixing matrix A1 of four rows, we can delete columns and keep for example sources whose signals are of greater energies and / or those which are the least correlated (sources less "mixed" possible) and / or the signals correspond to the direct field of the sources, or the like.

In step S71, an inversion of each mixing sub-matrix A1, A2 is carried out in order to obtain respectively the decoding sub-matrices B1, B2 presented above (step S7). Passing through the mixing matrix A makes it possible in particular to conserve satisfactory levels of energy of the ambisonic components linked to each order, despite the matrix reductions. In other words, steps S5 to S71 make it possible to “refine” the decoding of the ambisonic content x (t).

The figure 9 is a block diagram of a processing algorithm corresponding to the embodiment illustrated in the figures 7 and 8 . The same step references S1, S2, etc. have been used to denote identical or similar steps presented above with reference to figures 7 and 8 .

We call “channels” the ambisonic microphone signals and “sources” the signals to be extracted (sources actually to be extracted or the loudspeaker supply signals). In step S1, there is an ambisonic content x (t) of order M, comprising a plurality of N ambisonic channels recorded to be processed. Generally speaking, the number of recorded Ambisonic channels is equal to N = (M + 1) ² . In step S2, the data relating to the ambisonic capture of the content x (t) (data relating to the ambisonic microphone MIC used, etc.) is available.

Knowing the validity limits of microphone encoding, a frequency band is determined for each ambisonic order. A filter bank allowing reconstruction is applied to the N ambisonic channels in step S3, to give K sub-bands denoted xk. The sub-bands are chosen to correspond to the different ranges of validity of the microphone encoding.

In a particular embodiment in step S4A illustrated in solid lines, a source separation matrix B is used, developed as a function of the frequency-filtered ambisonic components. (arrow from above coming on rectangle S4A). More particularly, a blind source separation method is applied in the sub-band containing the most valid channels, to obtain a separation matrix B of dimensions Ntarget x N, Ntarget being the number of sources obtained by the blind separation method in the selected frequency sub-band.

The valid channels are determined on the basis of a validity criterion relating to each order of the ambisonic content x (t) as a function of each frequency band of the filter bank. More generally, in order to maximize the quality of the source separation, a frequency band comprising the most valid ambisonic components is chosen. The term “valid” is understood to mean components whose energy criteria or directivity have not been biased during ambisonic capture, as presented above with reference to the. figure 5 . The validity of each order in frequency bands of the audio domain can be established by knowing the limits of the ambisonic microphone used during the capture of the ambisonic content x (t), or with the help of an abacus established on the basis of measurements carried out on a plurality of ambisonic microphones, making it possible to obtain an average of the validity of each ambisonic order in each frequency band.

For example, 1st order ambisonics channels tend to be valid in a frequency band ranging from 100Hz to about 10kHz. The frequency band in which 2nd order ambisonics can be more generally valid can for example range from 1kHz to 9kHz, etc.

In an alternative embodiment with a view to reproducing a sound scene on several loudspeakers (more than two in general), in step S4B (illustrated by the dotted lines on the figure 9 , to denote this variant), the decoding matrix is constructed as a function of the position of the loudspeakers on which the content must be reproduced. More exactly, this decoding matrix B corresponds to the inverse of a mixing matrix A which is defined by the respective spatial positions of the loudspeakers.

Returning to the general processing (for a restitution or for a separation of sources), in step S5, the “theoretical” mixing matrix A (for the two aforementioned variants) is constructed by inversion of B. For the separation of sources , the mixing matrix is made up of N rows and Ntarget columns, the ith column containing the spherical harmonic coefficients, relating to the coordinates ( θ _i , φ _i ) of the source s _i . Below is an example of a mixing matrix A in the case of source separation for ambisonic content of order 2 composed of five sources:

For the broadcast on loudspeakers, A is composed of N lines and a minimum of N columns, the ith column containing the spherical harmonic coefficients, relating to the coordinates ( θ _i , φ _i ) of the loudspeaker i.

In step S6, and for each sub-band k, a mixing sub-matrix Ak is constructed, such that Ak is a truncated version of the matrix A, by keeping only the Nk rows corresponding to the channels effectively valid in this subband k.

For the separation of sources, if Nk is less than the number of sources Ntarget sought in the sub-band, only one set of Ntarget, k, columns (with Ntarget, k less than or equal to Nk), chosen according to energy criteria (for example by separating the sources with the greatest contribution) or according to other criteria of interest as defined previously. The matrix Ak thus has the dimensions Nk x Ntarget, k, with Ntarget, k = min (Nk, Ntarget) for example. Below is an example of an Ak (4x4) matrix truncated to order 1 ambisonic:

For reproduction on loudspeakers, a set of Nk loudspeakers is selected for reproduction, and Ak therefore has the dimensions Nk x Nk.

In step S7, the matrix Ak is inverted to give Bk. When the sub-matrix Ak is not a square matrix, an infinite number of possibilities exist for the inversion. A pseudo-inversion can be applied, or else an inversion by applying additional constraints (for example choice of the solution giving the most directional beamforming, or minimizing the secondary lobes).

In general, the term “matrix inversion” is understood to mean both a conventional matrix inversion and a pseudo-inversion as presented above.

Then, in step S8, Bk is applied to the subband xk to obtain the signals sk such as
sk = Bk. xk

Once sources have been extracted in each sub-band, the corresponding full-band signals are reconstructed by a synthesis filter from the signals of sub-bands of the same direction, in step S9.

Below, an example of implementation of the method according to a particular embodiment of the invention is described by way of example.

We have an ambisonic content of order 2 (9 channels) sampled at 16kHz, noted x (t) made up of 3 sources that we want to extract. Ambisonic encoding at orders 0 and 1 is valid between 200Hz and 8000Hz. Order 2 encoding is valid between 900Hz and 8000Hz.

A filter bank is implemented, consisting of two frequency bands, 200Hz-900Hz (up to order 1) and 900Hz-8000Hz (use of order 2)

The filterbank is applied to x (t), to form x1 (t) and x2 (t). x1 (t) consists of 4 channels (order 1 ambisonics) and x2 (t) contains 9 channels (order 2 ambisonics).

A separation matrix B of dimensions 3x9 is estimated by independent component analysis carried out in the 900Hz-8000Hz sub-band, that is to say x2 (t).

A theoretical mixing matrix A, of dimensions 9x3, is deduced by inversion of B, each column i containing the spherical harmonic coefficients of the source i.

• A1 contient uniquement les coefficients jusqu'à l'ordre 1 pour les trois sources, soit : A1= A (les quatre premières lignes, les trois premières colonnes),
• A2 contient les coefficients relatifs aux neufs canaux pour les trois sources, on a donc : A2=A

At the same time, the A1 and A2 matrices are calculated from A to extract the sources in each subband:

• A1 only contains the coefficients up to order 1 for the three sources, i.e.: A1 = A (the first four rows, the first three columns),
• A2 contains the coefficients relating to the nine channels for the three sources, so we have: A2 = A

A1 and A2 are inverted to form the separation matrices B1 and B2.

The three sources are extracted in each sub-band of respective indices 1 and 2:
s1 = B1.x1 and s2 = B2.x2

Then, the full-band sources are reconstituted by applying the synthesis filter to the signals in sub-bands s1 and s2, for example by band-to-band summation (if the analysis filter bank operated in base band): $$\mathrm{s}=\mathrm{s}1+\mathrm{<figure-callout\; id="2"\; label="s"\; filenames="imgf0001.png,imgf0002.png"\; state="\{\{state\}\}">s</figure-callout>}2$$

With reference to the figure 10 , the present invention further relates to a DIS device for implementing the invention. This DIS device can include an input interface IN to receive ambisonic signals x (t). The device DIS can comprise a memory MEM for storing instructions of a computer program within the meaning of the invention. The instructions of the computer program are instructions for processing the ambisonic signals x (t). They are implemented by a processor PROC, in order to deliver, via an output interface OUT, decoded signals s (t).

Of course, the present invention is not limited to the embodiments described above by way of example; it extends to other variants.

Typically, the frequency ranges for which the ambisonic representation is valid are given above by way of example and may differ depending on the nature of the ambisonic microphone (s) used for pickup, or even the pickup conditions themselves.

Claims (15)

1. Method, implemented by computing means, for processing an ambisonic content comprising a plurality of ambisonic components of a plurality of orders defining a succession of ambisonic channels in each of which there is represented an ambisonic component, the method comprising:

   - a frequency-domain filtering of the ambisonic components in a plurality of frequency bands,

   - a creation of an ambisonic decoding matrix (B), the method being characterized by:

   - a processing of the ambisonic decoding matrix (B) to extract by matrix size reduction, a plurality of ambisonic decoding sub-matrices (B1, B2) each associated with an ambisonic order and with a frequency band chosen for this ambisonic order,

   - respective applications of the decoding sub-matrices to the ambisonic components in each chosen frequency band, and a band-by-band reconstruction of the results of said respective applications, to deliver a plurality of decoded signals, each associated with a sound source.
2. Method according to Claim 1, wherein each sub-matrix is associated with a frequency band chosen as a function of a validity criterion of the ambisonic components of the order with which said sub-matrix is associated, in said chosen frequency band.
3. Method according to Claim 2, wherein the validity criterion of the components is defined by conditions of pickup of said ambisonic components, by at least one ambisonic microphone.
4. Method according to Claim 3, comprising:

   - a reception of data from at least one ambisonic microphone used to capture up said ambisonic components;

   - a determination of the frequency bands chosen to construct said sub-matrices (B1, B2), as a function of said ambisonic microphone data.
5. Method according to one of the preceding claims, wherein, each ambisonic decoding sub-matrix (B1, B2) being associated with an ambisonic order and with a frequency band chosen for this ambisonic order,

   - a frequency band is chosen within a range from 100 Hz to 10 kHz for the ambisonic order m = 1,

   - a frequency band is chosen in a range from 500 Hz to 10 kHz for the ambisonic order m = 2,

   - a frequency band is chosen in a range from 2000 Hz to 9000 Hz for the ambisonic order m = 3,

   - a frequency band is chosen in a range from 3000 Hz to 7000 Hz for the ambisonic order m = 4.
6. Method according to one of the preceding claims, wherein the processing of the ambisonic decoding matrix (B) comprises:

   - an inversion of the created ambisonic decoding matrix (B), to obtain a mix matrix (A) in which:

   * the rows correspond to respective ambisonic channels, and

   * the columns correspond to sound sources,

   - a processing of the mix matrix (A) to extract, by matrix size reduction, a plurality of mix sub-matrices (A1, A2) each associated with an ambisonic order and with a chosen frequency band, and

   - an inversion of the mix sub-matrices (A1, A2) to obtain, respectively, said ambisonic decoding sub-matrices (B1, B2).
7. Method according to one of the preceding claims, wherein the processing of the ambisonic content is conducted for a source separation and said decoding matrix (B) is a matrix of blind separation of sources created from the ambisonic components (S4A).
8. Method according to Claim 7, taken in combination with Claim 2, wherein the separation matrix (B) is created from the ambisonic components filtered at a chosen frequency band and wherein the number of valid ambisonic channels according to said criterion is the maximum.
9. Method according to one of Claims 7 and 8, taken in combination with Claim 6, further comprising a simplification of the mix sub-matrices (A1, A2) before the inversion thereof, by reduction of a number of columns of each sub-matrix, the remaining columns of the sub-matrices being chosen so as to conserve signals of greater energies after application of the decoding sub-matrices.
10. Method according to one of Claims 7 to 9, taken in combination with Claim 6, further comprising a simplification of the mix sub-matrices (A1, A2) before the inversion thereof, by reduction of a number of columns of each sub-matrix, the remaining columns of the sub-matrices being chosen so as to conserve the least correlated signals after application of the decoding sub-matrices.
11. Method according to one of Claims 7 to 10, taken in combination with Claim 6, further comprising a simplification of the mix sub-matrices (A1, A2) before the inversion thereof, by reduction of a number of columns of each sub-matrix, the remaining columns of the sub-matrices being chosen so as to conserve signals corresponding to direct sound fields after application of the decoding sub-matrices.
12. Method according to one of Claims 1 to 6, wherein the processing of the ambisonic content is conducted for an ambisonic rendering on a plurality of loudspeakers and said decoding matrix (B) is an inverse matrix of relative spatial positions of the loudspeakers (S4B).
13. Method according to one of the preceding claims, comprising, for an ambisonic content (x) broken down into frequency sub-bands (k), an application of decoding sub-matrices (Bk), obtained by:

    - for each ambisonic order of the content, a determination of a frequency band over which said order observes a predetermined ambisonic encoding validity criterion,

    - on the basis of said frequency bands, an application of a bank of filters to the ambisonic content (x) to produce a plurality of signals in sub-bands (xk), of variable dimensions corresponding to valid ambisonic channels in this sub-band (k),

    - a determination of a decoding matrix (B) of maximum size in the frequency band of the maximum ambisonic order and of an associated mix matrix (A), the inverse or pseudo-inverse of said decoding matrix (B),

    - for each other frequency band (k), a determination of a mix matrix (Ak) of reduced size, a sub-matrix of said mix matrix (A), and of a decoding sub-matrix (Bk), the inverse or pseudo-inverse of said mix sub-matrix (Ak),

    - a reconstruction of the full-band separated signals (s) by application of a synthesis filter bank to the separated signals (sk) derived from the multiplication of said signals (xk) by said matrices (Bk) .
14. Computer program, characterized in that it comprises instructions for the implementation of the method according to one of Claims 1 to 13, when this program is run by a processor.
15. Computing device comprising:

    - an input interface for receiving ambisonic component signals,

    - an output interface for delivering decoded signals, each associated with a sound source,

    - and a processing circuit configured to implement the method according to one of Claims 1 to 13.

Images