EP3040989A1 - Improved method of separation and computer program product - Google Patents

Abstract

A method of separating, in a mixing signal (w(t)), a pure specific contribution x(t) and a background contribution z(t) using a spectrogram modeling the mixing signal V corresponding to the sum a spectrogram of a reverberant specific contribution V rev,y and a spectrogram of the background sound contribution V z , the spectrogram of the reverberant specific contribution depending on the spectrogram of the pure contribution V x according to the model: V ^ f, t rev. y = Ä = 1 T V ^ f , t − Ä + 1 x R f , Ä where R is a reverb matrix, f is a frequency step, t is a time step, and Ä an integer between 1 and T; and minimizing a cost function (C) between the mixing signal spectrogram and the modeling mixing signal spectrogram.

Description

The present invention relates to methods of separating a plurality of contributions into a mixing acoustic signal, and in particular to separating a voice contribution from a background musical contribution into an acoustic signal. mixture.

A soundtrack of a song includes a vocal contribution (the lyrics sung by one or more singers) and a musical contribution (accompanying music played by one or more instruments).

A soundtrack of a film includes a vocal contribution (dialogues between actors) superimposed on a musical contribution (special sound effects and / or background music).

Separation algorithms are known to separate the vocal contribution from the musical contribution into an original soundtrack.

For example the article of Jean-Louis Durrieu et al. "An iterative approach to monaural musical mixture of soloing," in International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, Taiwan, April 2009, pp. 105 - 108 discloses a separation algorithm of the underdetermined source separation algorithm based on non-negative matrix decomposition, to separate the voice contribution from the background contribution.

However, the known separation algorithms do not make it possible to correctly take into account the reverberation phenomenon affecting the components of the mixture.

In the particular case of a vocal component, this one results from the superposition of the dry voice, or pure in what follows, corresponding to the recording of the sound emitted by the singer and which propagated directly towards the microphone recording, and reverberation, corresponding to the recording of the sound emitted by the singer but which has propagated indirectly to the recording microphone, that is to say by reflection, possibly multiple, on the walls from the recording room. Reverb, consisting of the echoes of the pure voice at a given moment, spreads over a time interval that can be significant (for example three seconds). In other words, at a given moment, the vocal contribution results from the superposition of the pure voice at this moment and the different echoes of the pure voice at previous moments.

However, the known separation algorithms do not take into account the long-term effects of the reverberation affecting a component of the mixture. The document Ngoc QK Duong, Emmanuel Vincent, and Remi Gribonval, "Underdetermined reverberant audio source separation using a full-rank spatial covariance model," IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 7, pp. 1830 - 1840, Sept 2010 is concerned with the instantaneous effects of spatial reverberation, but does not model the effects of memory, ie, taking into account the latency between the recording of a sound and the recording of echoes associated with this sound. Thus, the type of algorithm proposed by this document applies only to multichannel signals and does not allow a correct extraction of reverb effects, which can be found in music. In the case of a voice component, the reverberation that affects this component is distributed in the different components obtained after the separation. The separate vocal component loses its richness and the accompanying music component is not of good quality.

It should be noted that reverb can be caused by the conditions under which sound is taken, but can also be artificially added during the post-production of the soundtrack, mainly for aesthetic reasons.

There is therefore a need for a method for separating contributions in a mixture, these contributions integrating a reverberation of the corresponding pure sound signal. More particularly, there is a need to separate a pure vocal contribution affected by reverberation, from a background musical contribution, into a sound signal.

The invention therefore aims to overcome this problem.

The invention therefore relates to a separation method and a program product according to the claims.

• la figure 1 est une représentation sous forme de blocs des différentes étapes du procédé de séparation selon l'invention ; et,
• les figures 2 et 3 correspondent à des graphes qui résultent de tests permettant de comparer, selon des critères normatifs connus, les résultats de la mise en oeuvre du procédé de la figure 1.

The invention will be better understood on reading the following description of a particular embodiment, given solely by way of illustrative and nonlimiting example, and with reference to the appended drawings in which:

• the figure 1 is a block representation of the different steps of the separation process according to the invention; and,
• the Figures 2 and 3 correspond to graphs which result from tests making it possible to compare, according to known normative criteria, the results of the implementation of the method of the figure 1 .

Referring to the figure 1 , the separation method 100 uses a mixing temporal acoustic signal w ( t ), to deliver a vocal acoustic signal y ( t ) and a musical acoustic signal z ( t ).

The signals are all acoustic signals, so that the qualifier of acoustics will be omitted in what follows.

These signals are time signals. They depend on time t .

The acoustic mix signal is a source soundtrack, or at least an extract from a soundtrack.

The acoustic mixing signal w ( t ) comprises a first so-called specific contribution and a second so-called accompanying contribution.

In the present description, the first contribution is a vocal contribution and corresponds to words sung by a singer.

The second contribution is a musical contribution and corresponds to the musical accompaniment of the singer.

The vocal acoustic signal y ( t ) corresponds to the only vocal contribution, isolated from the rest of the mixing signal w ( t ), and the musical acoustic signal z ( t ) corresponds to the only musical contribution, isolated from the rest of the mixing signal w (t).

In the present embodiment, it is considered that only the voice contribution is reverberated.

où x(t) est le signal vocal pur, c'est-à-dire le signal sonore généré par le chanteur est qui s'est propagé directement vers le microphone d'enregistrement ; et où r(t) est une réponse impulsionnelle, qui est une distribution donnant l'amplitude des échos pour chaque instant d'arrivé de l'écho correspondant sur le microphone d'enregistrement, et où * correspond au produit de convolution.The reverb is modeled as follows: $$\mathrm{there}\left(t\right)=r\left(t\right)*x\left(t\right)$$

where x (t) is the pure speech signal, that is, the sound signal generated by the singer is that propagated directly to the recording microphone; and where r ( t ) is an impulse response, which is a distribution giving the amplitude of the echoes for each arrival time of the corresponding echo on the recording microphone, and where * is the convolution product.

The pure speech signal x (t) is the free-field signal and the impulse response r ( t ) is characteristic of the acoustic environment of the recording.

où V ^rev,y est le spectrogramme du signal y(t), considéré comme affecté par de la réverbération, V^x est le spectrogramme du signal x(t), R est une matrice FxT de réverbération correspondant au spectrogramme de la réponse impulsionnelle r(t), avec F la dimension fréquentiel et T la dimension temporelle de R. In the time-frequency domain, for non-negative spectrograms, this reverberation model can be approximated, as proposed in the Rita Singh, Bhiksha Raj, and Paris Smaragdis, "Latent-variable decomposition based dereverberation of monaural and multi-channel signals," in IEEE International Conference on Audio and Speech Signal Processing, Dallas, Texas, USA, March 2010 , by : $$V_{f,t}^{\mathit{rev},\mathrm{there}}={\displaystyle \underset{\tau =1}{\overset{T}{\Sigma }}}{V}_{f,t-\tau +1}^x{R}_{f,\tau }$$

where V ^{rev , y} is the spectrogram of the signal y (t), considered to be affected by reverberation, V ^x is the spectrogram of the signal x (t), R is a matrix FxT of reverb corresponding to the spectrogram of the impulse response r ( t ), with F the frequency dimension and T the time dimension of R.

The first step 110 of the method 100 consists of sampling the mixing signal w ( t ) and calculating a spectrogram V of the mixing signal w (t). In general, a spectrogram is defined as the absolute value (or the square of the absolute value) of the short-term Fourier transform of a sampled signal. Other time-frequency transformations are possible, such as a constant Q transform, or a short-term Fourier transform followed by frequency filtering (using a Mel or Bark scale filter bank, for example).

For each time sampling step, the spectrogram comprises a frequency frame, indicating for each frequency sampling step, the instantaneous power of the signal.

The spectrogram V is therefore a matrix F x U, positive real numbers

U represents the total number of frames that subdivided the signal duration of the mixture w (t). F is the total number of frequency sampling steps, which is generally between 200 and 2000.

The method 100 then comprises a first part in which the voice signal is considered as a pure vocal signal, without reverberation.

In this first part, the mixing signal modeling spectrogram is the sum of the spectrogram of the speech signal V ^y , and the spectrogram of the musical signal V ^z . V ^y is the spectrogram of the signal y (t), considered unaffected by reverberation. This modeling is finally the usual modeling in the context of the methods of decomposition by factorization in non-negative matrices.

It should be noted that â refers to a quantity which is an estimate of the quantity a.

Thus, in the steps of the first departure of the method 100, it is sought to estimate the two output spectrograms whose sum approximates (sign ≈ in the following expression) at best the spectrogram of the mixture: $$V\approx \hat{V}={\hat{V}}^{\mathrm{there}}+{\hat{V}}^z$$

The voice signal modeling is based on a voice / source type production model, as proposed in the Jean-Louis Durrieu et al. "An iterative approach to monaural musical mixture of soloing," in International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, Taiwan, April 2009, pp. 105 - 108 : $${\hat{V}}^{\mathrm{there}}=\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(W_K{H}_K\right)$$

The first term of this modeling is the source of the voice, which corresponds to the excitation of the vocal chords: W _{F 0} is a matrix of harmonic atoms, which is predefined and specific to speech signals; H _{F 0} is an activation matrix indicating at each moment the harmonic atoms of the matrix W _{F 0} which are activated.

The second term of this modeling is the filter of the voice, which corresponds to the filtering performed by the vocal tract: W _K is a matrix of filtering atoms; H _K is an activation matrix indicating at each instant the filtering atoms of the matrix W _K that are activated.

The operator ⊙ corresponds to the term-to-term matrix multiplication of two matrices (also called Hadamard product).

The modeling of the musical signal is based on a generic model of factorization by non-negative matrices: $${\hat{V}}^z=\left(W_R{H}_R\right)$$

Columns of W _R can be seen as elemental spectral models and H _R as an activation matrix of these elementary models over time.

The first part of the process then consists in estimating the matrices H _{F 0} , W _K , H _K , W _R and H _R.

In order to estimate the parameters of these matrices, a cost function C, based on a divergence d by element, is used: $$\mathrm{VS}=D\left(V|{\hat{V}}^{\mathrm{there}}+{\hat{V}}^z\right)={\mathrm{\Sigma }}_{f,t}d\left(V_{\mathit{ft}}|{\hat{V}}_{\mathit{ft}}^{\mathrm{there}}+{\hat{V}}_{\mathit{ft}}^z\right)$$

In the presently contemplated embodiment, the Itakura-Saito divergence, well known to those skilled in the art, is used. This one is obtained by fixing the value of the parameter of the beta-divergence with β = 0 and is thus expressed: $$d\left(\mathrm{at}|b\right)=\frac{\mathrm{at}}{b}-\log \frac{\mathrm{at}}{b}-1$$

où a et b sont deux scalaires réels positifs.For the record, beta-divergence is defined by: $$d_{\beta }\left(\mathrm{at}|b\right)=\{\begin{array}{c}\frac{1}{\beta \left(\beta -1\right)}\left({\mathrm{at}}^{\beta }+\left(\beta -1\right)b^{\beta }-{\mathit{\beta ab}}^{\beta -1}\right),\beta \in \mathbb{R}\setminus \left\{0,1\right\}\\ \begin{array}{cc}\mathrm{at}\log \frac{\mathrm{at}}{b}-\mathrm{at}+b,& \beta =1\\ \frac{\mathrm{at}}{b}-\log \frac{\mathrm{at}}{b}-1,& \beta =0\end{array}\end{array}$$

where a and b are two real positive scalars.

In step 120, the cost function C is thus minimized so as to determine the optimum value of each parameter of each matrix. This minimization is performed by iterations, with multiplicative updating rules which are successively applied to each of the parameters of the matrices H _{F 0} , W _K , H _K , W _R and H _R.

These update rules are for example elaborated by considering the gradient (that is to say the partial derivative) of the cost function C with respect to each parameter. More precisely, the gradient of the cost function with respect to the parameter considered is written in the form of a difference between two positive terms, and the corresponding updating rule is a multiplication of the parameter considered by the ratio of these two terms. .

This allows in particular that the parameters remain non-negative at each update and become constant when the gradient of the cost function with respect to the parameter considered tends to zero.

In this way, the parameters evolve towards a local minimum.

$$H_K\leftarrow H_K\odot \frac{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)}{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)}$$

$$W_K\leftarrow W_K\odot \frac{\left(\left(W_{F0}{H}_{F0}\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)H_K^T}{\left(\left(W_{F0}{H}_{F0}\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)H_K^T}$$

$$H_R\leftarrow H_R\odot \frac{W_R^T\left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)}{W_R^T\left({\hat{V}}^{\odot \left(\beta -1\right)}\right)}$$

$$W_R\leftarrow W_R\odot \frac{\left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)H_R^T}{\left({\hat{V}}^{\odot \left(\beta -1\right)}\right)H_R^T}$$

où ⊙ est un opérateur correspondant au produit terme à terme entre matrices (ou vecteur) ; .⊙(.) est un opérateur correspondant à l'exponentiation terme à terme d'une matrice par un scalaire ; (.)^T est la transposée d'une matrice.The update rules are as follows: $${\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F<figure-callout\; id="0"\; label="\leftarrow \; H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">\leftarrow \; </figure-callout>H_F{0}_{}\odot \frac{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)}{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)}$$

$$H_K\leftarrow H_K\odot \frac{W_K^T\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)}{W_K^T\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)}$$

$$W_K\leftarrow W_K\odot \frac{\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)H_K^T}{\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)H_K^T}$$

$$H_R\leftarrow H_R\odot \frac{W_R^T\left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)}{W_R^T\left({\hat{V}}^{\odot \left(\beta -1\right)}\right)}$$

$$W_R\leftarrow W_R\odot \frac{\left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)H_R^T}{\left({\hat{V}}^{\odot \left(\beta -1\right)}\right)H_R^T}$$

where ⊙ is an operator corresponding to the term term product between matrices (or vector); .⊙ (.) Is an operator corresponding to the term-exponentiation of a matrix by a scalar; (.) ^T is the transpose of a matrix.

For this first part, all parameters are initialized with non-negative values chosen randomly.

Then, in step 130, the matrix H _{F 0} is constrained by using a tracking algorithm such as the Viterbi "tracking" algorithm in order to select, for each time step, the frequency step in which we find a maximum power, without being too far in frequency power maxima selected for previous time steps.

Then, in step 140, the coefficients of the matrix H _{F 0} which are at a frequency distance greater than a reference distance are set to 0.

A matrix H ' _{F 0} is obtained.

In the second part of the method 100, the speech signal is considered to be affected by reverberation. It should be noted that the first part of the process allows to obtain initial values for the parameters which will be estimated by successive iterations during the implementation of the second part of the process. Other ways of defining the initial values of these parameters are conceivable.

où * _t dénote un opérateur de convolution ligne par ligne tel qu'explicité dans le membre de droite de l'équation ci-dessus.In this second part, the modeling of the vocal signal considered as reverberated, V ^{rev, y} , as a function of the pure vocal signal V ^x is then written: $${\left[{\hat{V}}^{\mathit{rev},\mathrm{there}}\right]}_{f,t}={\left[{\hat{V}}^x*{}_{t}R\right]}_{f,t}={\displaystyle \underset{\tau =1}{\overset{T}{\Sigma }}}{V}_{f,t-\tau +1}^x{R}_{f,\tau }$$

where * _t denotes a line-by-line convolution operator as explained in the right-hand side of the equation above.

The reverberation matrix R has T time step (of the same length a step of sampling mixed signal), and F no sampling frequency. T is predetermined by the user and is generally between 20 and 200, for example 100.

Moreover, as above, the spectrogram V ^x of the pure signal is modeled by: $${\hat{V}}^x=\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(W_K{H}_K\right)$$

The second part of the process then consists in estimating the matrices H _{F 0} , W _K , H _K , W _R , H _R and R which make it possible to approximate the spectrogram of the mixture V : $$V\approx {\hat{V}}^{\mathit{rev}}={\hat{V}}^{\mathit{rev},\mathrm{there}}+{\hat{V}}^z$$

In order to estimate the parameters of these matrices, a cost function C, based on a divergence d by element, is used: $$\mathrm{VS}=D\left(V|{\hat{V}}^{\mathit{rev},\mathrm{there}}+{\hat{V}}^z\right)={\mathrm{\Sigma }}_{f,t}d\left(V_{\mathit{ft}}|{\hat{V}}_{\mathit{ft}}^{\mathit{rev},\mathrm{there}}+{\hat{V}}_{\mathit{ft}}^z\right)$$

In the presently contemplated embodiment, the Itakura-Saito divergence, well known to those skilled in the art, is used. This one is obtained by fixing the value of the parameter of the beta-divergence with β = 0 and is thus expressed: $$d\left(\mathrm{at}|b\right)=\frac{\mathrm{at}}{b}-\log \frac{\mathrm{at}}{b}-1$$

Advantageously, the cost function of the second part is similar to that used in the first part.

In step 220, the cost function C is then minimized so as to determine the optimum value of each parameter of each matrix, in particular the parameters of the reverberation matrix.

$$H_{F0}\leftarrow H_{F0}\odot \frac{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)}{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)}$$

$$H_K\leftarrow H_K\odot \frac{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)}{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)}$$

$$W_K\leftarrow W_K\odot \frac{\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)H_K^T}{\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)H_K^T}$$

où * _t désigne l'opérateur de convolution ligne par ligne tel que défini ci-dessus.This minimization is performed by iterations with multiplicative updating rules, which are successively applied to each of the parameters of the matrices. For the matrices of the voice component integrating a reverberation, we have: $$R\leftarrow R\odot \frac{\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)*{}_t{\hat{V}}^x}{{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}*{}_{\mathrm{<figure-callout\; id="0"\; label="t\; H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">t\; </figure-callout>}}{}^{}{\hat{V}}^x}$$

$$H_F$$$$0_{}<figure-callout\; id="0"\; label="\leftarrow \; H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">\leftarrow \; </figure-callout>H_F{0}_{}\odot \frac{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)}{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)}$$

$$H_K\leftarrow H_K\odot \frac{W_K^T\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)}{W_K^T\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)}$$

$$W_K\leftarrow W_K\odot \frac{\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)H_K^T}{\left(\left(W_{F0}{\mathrm{<figure-callout\; id="0"\; label="H\; F"\; filenames="imgf0002.png"\; state="\{\{state\}\}">H\; </figure-callout>}}_F\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)H_K^T}$$

where * _t denotes the line-by-line convolution operator as defined above.

$$W_R\leftarrow W_R\odot \frac{\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)H_R^T}{\left({\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)H_R^T}$$

For the musical background component, we have, as in the first part of the process: $$H_R\leftarrow H_R\odot \frac{W_R^T\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)}{W_R^T\left({\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)}$$

$$W_R\leftarrow W_R\odot \frac{\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)H_R^T}{\left({\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)H_R^T}$$

As indicated above, the update rules are developed from the partial derivative of the cost function C with respect to each relevant parameter. They therefore depend on the form chosen for the cost function, in particular the divergence used in this cost function. The rules above are therefore specific to the use of beta-divergence.

It should be noted that, since these rules each result from a partial derivation according to a specific parameter, the update rule of the reverberation matrix R is general in the sense that it does not depend on the modeling chosen for the spectrogram V ^x of the pure signal or that of the background sound spectrogram V ^z .

As regards the matrix H _{F 0} , the iterations start from the matrix H ' _{F 0} determined in the first part of the method. It should be noted that, since the update rules are multiplicative, the coefficients of the matrix H _{F 0} initially set to 0 will remain at 0 during the minimization of the cost function in the second part of the method.

The other parameters of the modeling and, in particular those of the spectrogram of the specific reverberant contribution V ^rev, are initialized with random values.

When the distance between the mixing spectrogram V and the estimated spectrogram V = V ^{rev, y} + V ^z is less than a predetermined threshold or when a number iterative limit iterations is reached, the process goes out of the iteration loop and the values of the matrices obtained, R, H _{F 0} , W _K , H _K , W _R and H _R , are the final values .

In step 230, conventional adapted processes (in particular a Wiener filtering type treatment) are applied to the above spectrograms to obtain in particular the spectrograms of interest V ^x , V ^z . Then, in step 240, an inverse transformation of that of step 110 is performed on these spectrograms to obtain the output signals, pure speech signal x (t) and musical signal z (t).

In the embodiments described here in detail, these acoustic signals are monophonic signals. In a variant, these signals are stereophonic. More generally, they are multichannel. The person skilled in the art knows how to adapt to stereophonic or multichannel signals the treatments presented for the case of monophonic signals.

The preferred embodiment relates to a specific component or interest component that is a voice component. However, the modeling of the reverberation of a component is general and applies to any type of component. In particular, the background sound component can also be affected by reverberation.

In addition, any type of non-negative non-reverberated sound spectrograms may also be used instead of those used above.

Moreover, in the embodiment presented above, the mixture comprises two components. Generalization to any number of components is straightforward.

• le premier procédé est une séparation, fondée sur une méthode de type NMF, sans inclure de modélisation sur la réverbération ;
• le second procédé est une séparation selon le procédé décrit ci-dessus, c'est-à-dire incluant une modélisation de la réverbération du signal vocal ; et,
• le troisième procédé est une limité mathématique théorique.

Comparative tests were conducted to compare the results of the implementation of the present process:

• the first method is a separation, based on an NMF type method, without including modeling on the reverberation;
• the second method is a separation according to the method described above, that is to say including a modeling of the reverberation of the voice signal; and,
• the third method is a theoretical mathematical limit.

In order to quantify the results obtained for the various processes, standard indicators of the field of source separation have been calculated. These indicators are the signal-to-distortion ratio (SDR), which corresponds to a quantitative test; the Signal to Artefact SAR (Signal to Artefact Ratio) ratio, which corresponds to the artifacts in the separate components; and the signal-to-interference ratio SIR (according to the acronym English "Signal to Interference Ratio"), which corresponds to the residual interferences between the separate components.

The results are presented on figures 2 for the voice signal and the figure 3 for the musical signal.

The method according to the invention therefore improves the results obtained, whatever the way of analyzing them.

Claims (12)

A method of separating (100), in a mixing acoustic signal w (t), a pure specific contribution, affected by reverberation, and a background noise contribution,
characterized by separating the pure specific contribution x (t) and the background noise contribution z ( t ),
by using a spectral sound mixing modeling spectrogram V ^rev corresponding to the sum of a spectrogram of a specific reverberant contribution V ^{rev, y} and a spectrogram of the background noise contribution V ^z , the spectrogram of the contribution specific reverberated dependent spectrogram of pure specific contribution V ^x according to the model: $${\hat{V}}_{f,t}^{\mathit{rev},\mathrm{there}}={\displaystyle \underset{\tau =1}{\overset{T}{\Sigma }}}{\hat{V}}_{f,t-\tau +1}^x{R}_{f,\tau }$$

where R is a reverberation matrix FxT , f is a frequency index, t is a time index, and τ is an integer between 1 and T ; and
by iteratively calculating an estimate of the spectrogram of the background noise contribution V ^z , the spectrogram of the pure specific contribution V ^x and the reverberation matrix R so as to minimize a cost function ( C ) between a signal spectrogram of mixture V and the spectrogram of modeling of the mixing signal V ^rev . Method according to Claim 1, characterized in that the cost function ( C ) uses a divergence ( d ) between the spectrogram of the mixing signal and the mixing signal modeling spectrogram, in particular the so-called beta-divergence divergence defined by: $$d_{\beta }\left(\mathrm{at}|b\right)=\{\begin{array}{c}\frac{1}{\beta \left(\beta -1\right)}\left({\mathrm{at}}^{\beta }+\left(\beta -1\right)b^{\beta }-{\mathit{\beta ab}}^{\beta -1}\right),\beta \in \mathbb{R}\setminus \left\{0,1\right\}\\ \begin{array}{cc}\mathrm{at}\log \frac{\mathrm{at}}{b}-\mathrm{at}+b,& \beta =1\\ \frac{\mathrm{at}}{b}-\log \frac{\mathrm{at}}{b}-1,& \beta =0\end{array}\end{array}$$

where a and b are two real positive scalars. Method according to claim 2, characterized in that the minimization of the cost function uses, to obtain an estimate of the reverberation matrix, multiplicative update rules of the type: $$R\leftarrow R\odot \frac{\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)*{}_t{\hat{V}}^x}{{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}*{}_t{\hat{V}}^x}$$

With V ^rev = V ^{rev, y} + V ^z ; and where ⊙ is an operator corresponding to the term term product between matrices (or vector); .⊙ (.) Is an operator corresponding to the term-exponentiation of a matrix by a scalar; * _t is a time convolution operator between two matrices defined by ${\left[\mathrm{AT}{*}_{t}B\right]}_{f,\tau }=\underset{\tau =t}{\overset{T}{\Sigma }}{\mathrm{AT}}_{f,\tau }{B}_{f,\tau -t+1}\mathrm{.}$

Method according to one of the preceding claims, characterized in that , the pure specific contribution being a vocal contribution, the spectrogram of the pure specific contribution V ^x is modeled by: $${\hat{V}}^x=\left(W_{F0}{H}_{F0}\right)\odot \left(W_K{H}_K\right)$$

where W _{F 0} is a predefined harmonic matrix of atoms, H _{F 0} is an activation matrix of the harmonic atoms of the matrix W _{F 0} , W _K is a matrix of filtering atoms, H _K is a matrix of activation of the filter atoms of the matrix W _K , and where ⊙ is an operator corresponding to the term term product between matrices. Method according to Claim 3 and Claim 4, characterized in that the minimization of the cost function uses multiplicative updating rules of the type: $$H_{F0}\leftarrow H_{F0}\odot \frac{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)}{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)}$$

$$H_K\leftarrow H_K\odot \frac{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)}{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)}$$

$$W_K\leftarrow W_K\odot \frac{\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)\right)\right)H_K^T}{\left(\left(W_{F0}{H}_{F0}\right)\odot \left(R*{}_t{\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)\right)H_K^T}$$

With V ^rev = V ^rev'y + V ^z ; and where ⊙ is an operator corresponding to the term term product between matrices (or vector); .⊙ (.) Is an operator corresponding to the term-exponentiation of a matrix by a scalar; (.) ^T is the transpose of a matrix; _{* t} is a time convolution operator between two matrices defined by ${\left[\mathrm{AT}{*}_{t}B\right]}_{f,\tau }=$

$\underset{\tau =t}{\overset{T}{\Sigma }}{\mathrm{AT}}_{f,\tau }{B}_{f,\tau -t+1}\mathrm{.}$

Method according to any one of the preceding claims, characterized in that the spectrogram of the background noise contribution V ^z is modeled by a factorization in non-negative matrices: $${\hat{V}}^z=\left(W_R{H}_R\right)$$

where W _R is a matrix of elementary spectral models and H _R is an activation matrix of the elementary spectral models of the matrix W _R. Method according to Claim 2 and Claim 6, characterized in that the minimization of the cost function implements multiplicative updating rules of the type: $$H_R\leftarrow H_R\odot \frac{W_R^T\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)}{W_R^T\left({\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)}$$

$$W_R\leftarrow W_R\odot \frac{\left(V\odot {\hat{V}}^{\mathit{rev}\odot \left(\beta -2\right)}\right)H_R^T}{\left({\hat{V}}^{\mathit{rev}\odot \left(\beta -1\right)}\right)H_R^T}$$

With V ^rev = V ^{rev, y} + V ^Z ; and where ⊙ is an operator corresponding to the term term product between matrices (or vector); .⊙ (.) Is an operator corresponding to the term-exponentiation of a matrix by a scalar; (.) ^T is the transpose of a matrix. Method according to one of the preceding claims, characterized in that , the separation of the pure specific contribution x (t) and the background noise contribution z ( t ) by using a spectroscopy of modeling the acoustic mixing signal V ^rev constituting a second part of the method, the latter comprises a first part consisting in separating, in the acoustic mixing signal w (t), a specific contribution and a contribution from the background, without taking into account the reverberation, of the initialization parameters among the parameters obtained at the end of the first part of the process being used as the initial value of the corresponding parameters in the spectrogram of the specific reverberant contribution V ^{rev, y} of the second part of the process. Method according to claim 8, characterized in that the first part comprises the minimization of a cost function implementing an algorithm similar to that implemented in the second part. Method according to Claim 2 and Claim 9, characterized in that , for the purpose of minimizing the cost function, the first part of the method implements multiplicative updating rules of the type: $$H_{F0}\leftarrow H_{F0}\odot \frac{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)}{W_{F0}^T\left(\left(W_K{H}_K\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)}$$

$$H_K\leftarrow H_K\odot \frac{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)}{W_K^T\left(\left(W_{F0}{H}_{F0}\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)}$$

$$W_K\leftarrow W_K\odot \frac{\left(\left(W_{F0}{H}_{F0}\right)\odot \left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)\right)H_K^T}{\left(\left(W_{F0}{H}_{F0}\right)\odot \left({\hat{V}}^{\odot \left(\beta -1\right)}\right)\right)H_K^T}$$

$$H_R\leftarrow H_R\odot \frac{W_R^T\left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)}{W_R^T\left({\hat{V}}^{\odot \left(\beta -1\right)}\right)}$$

$$W_R\leftarrow W_R\odot \frac{\left(V\odot {\hat{V}}^{\odot \left(\beta -2\right)}\right)H_R^T}{\left({\hat{V}}^{\odot \left(\beta -1\right)}\right)H_R^T}$$

with: V = V ^x + V ^Z , V ^z = ( W _R H _R ), and V ^x = ( W _{F 0} H _{F 0} ) ⊙ ( W _K H _K ); where W _R is a matrix of elementary spectral models and H _R is an activation matrix of the elementary spectral models of the matrix W _R ; where W _{F 0} is a predefined harmonic matrix of atoms, H _{F 0} is an activation matrix of the harmonic atoms of the matrix W _{F 0} , W _K is a matrix of filtering atoms, H _K is a matrix of activation of the filter atoms of the matrix W _K ; and where ⊙ is an operator corresponding to the term term product between matrices (or vector); .⊙ (.) Is an operator corresponding to the term-exponentiation of a matrix by a scalar; (.) ^T is the transpose of a matrix. Process according to any one of Claims 8 to 10, characterized in that it comprises, in the first part of the process, following the minimization of the cost function, the application of a maximum tracking algorithm of power in the activation matrix of the specific contribution H _{F 0} , said algorithm preferably being of the Viterbi algorithm type, then the zeroing of all the terms of the activation matrix of the specific contribution H _{F 0} which are too far from the maximum power found, the terms of the activation matrix of the specific contribution H _{F 0} constituting the initialization parameters which are used as the initial value of the corresponding parameters in the spectrogram of the specific reverberated contribution V ^{rev, y} of the second part of the process, the other parameters of the spectrogram of the specific reverberant contribution V ^rev, being initialized with random values. Computer program product, characterized in that it comprises instructions adapted to be stored in the memory of a computer and executed by the processor of said computer to implement a separation method according to any one of the preceding claims. .

Images