EP3828886A1 - Method and system for separating the voice component and the noise component in an audio flow - Google Patents

Abstract

The invention relates to a method and a system for separating in real time in an audio stream the voice component and the noise component.

Description

The invention relates to a method and a system for separating, in real time in an audio stream, the part of the stream associated with a voice or speech, from another part of the stream containing the noise.

The invention finds its application in a context where one or more people are talking in a noisy environment (hubbub, engine noise, ventilation, etc.). The speech signal superimposed on the noisy signals is digitized into an audio stream by a sound sensor.

The invention also relates to a method and a system for enhancing a real-time voice signal in an audio stream from a method of separating audio sources in delayed time.

The state of the art known to the applicant is divided into two categories, the so-called conventional approaches and the approaches possible by artificial intelligence known under the Anglo-Saxon name of “deep learning”.

In the “deep learning” approach, some approaches directly deal with the problem of voice / background noise separation, others relate to signal / signal, voice / voice separation.

The patent application US 20190066713 discloses a method of obtaining, by a device, a combined sound signal for combined signals from multiple sound sources in an area in which a person is located. The processing implemented uses deep neural networks.

An example of a method for separating several voices in an audio signal according to the prior art comprises the steps described below and not shown for reasons of simplification. The incoming audio signal is denoted by X , it has the length L. The signal is transmitted to an encoder M ₁ which transforms X into a tensor X ⁽¹⁾ of dimensions F × T where T is a divisor of L and F a number of filters given by the designer. The encoder M ₁ consists of a 1D Convolution with F filters. The coefficients of the convolution kernels are adjusted during a learning phase. The tensor is transmitted on the one hand to a multiplier for future use and on the other hand to a separation module. The separation module is divided into two submodules M ₂ and M ₄ . The first submodule M ₂ transforms the tensor X ⁽¹⁾ into a tensor X ⁽²⁾ of dimensions F × T. The first submodule M ₂ consists of a normalization layer, a 1x1 convolution and a stack of 1D-Conv modules known from the prior art and whose parameters are set during a learning phase.

The second submodule M ₄ transforms X ⁽²⁾ into an X ⁽⁴⁾ tensor of dimensions 2F × T. For this, the second submodule M ₄ connects a non-linearity, a 1x1 convolution and a sigmoid function. The coefficients of the 1x1 convolution are set during a learning phase.

X ⁽¹⁾ is concatenated to itself to form a tensor of dimensions 2F x T which is multiplied by X ⁽⁴⁾ to form X ⁽⁵⁾ .

The module M ₅ takes as input X ⁽⁵⁾ and outputs two signals of length L by means of a 1D deconvolution, the parameters of which are adjusted during a learning phase.

The digital parameters defining the processing of the different modules are obtained in a prior learning phase on a database.

By replacing one of the voices with noise, it is immediate to use the methods described in the state of the art to separate the voice from the background noise in an audio signal and, by keeping only the output containing the voice signal , to enhance the voice with a noisy signal.

The figure 1 illustrates an application to the separation of signals of different types, by separating the voice channel and the noise channel.

As described, the state of the art does not directly allow real-time processing of an audio stream.

The document Mimilakis Stylianos loannis et al, entitled “A recurrent encoder-decoder approach with skip-filtering connections for monaural singing voice separation”, September 25, 2017, pages 1-6, XP 033263882 , discloses a method for separating the voice from a musical background.

In the technical field of "deep learning", data is represented in the form of tensors. The data is modified by a succession of modules. At the output of each module, the data is projected into an abstract space generally defined by its dimensions.

To do this, the present invention implements the following treatments:

The signal (input stream X ) is split into N frames of length L, with X _N the nth frame. The method carries out the following treatments:

de dimensions F x T avec F le nombre de filtres donné par le concepteur,The frame X _N is encoded by a network of 1D convolutions. The result is a tensor $X_{\mathrm{NOT}}^{}$${}_{\left(1\right)}^{}$

of dimensions F x T with F the number of filters given by the designer,

est ensuite transformé par un module M_2, 101. Le résultat $X_N^{\left(2\right)}$

est un tenseur de dimensions F x T. Le module M₄ estime, 103, à partir de $X_N^{\left(2\right)},$

un tenseur $X_N^{\left(4\right)}$

de dimensions 2F x T. $X_N^{\left(1\right)}$

est concaténé à lui-même, 104, pour former un tenseur de dimensions 2F x T qui est multiplié à $X_N^{\left(4\right)}$

pour former $X_N^{\left(5\right)}\mathrm{.}$

Le module M₅ à partir de $X_N^{\left(5\right)}$

produit un tenseur de dimension 2 x T, 105, à partir duquel on obtient deux sorties de dimensions 1 x T X _N,0 et X _N,1 qui sont respectivement le canal voix et le canal bruit.T a divisor of L depending on the size of the filters F, 100. The result $X_{\mathrm{NOT}}^{}$${}_{\left(1\right)}^{}$

is then transformed by a module M _2, 101. The result $X_{\mathrm{NOT}}^{\left(2\right)}$

is a tensor of dimensions F x T. The modulus M ₄ estimates, 103, from ${\mathrm{<figure-callout\; id="2"\; label="X\; NOT"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_{\mathrm{NOT}}^{},$

a tensor $X_{\mathrm{NOT}}^{\left(4\right)}$

of dimensions 2F x T. $X_{\mathrm{NOT}}^{}$${}_{\left(1\right)}^{}$

is concatenated to itself, 104, to form a tensor of dimensions 2F x T which is multiplied by $X_{\mathrm{NOT}}^{\left(4\right)}$

to train $X_{\mathrm{NOT}}^{\left(5\right)}\mathrm{.}$

The M ₅ module from $X_{\mathrm{NOT}}^{\left(5\right)}$

produces a tensor of dimension 2 x T, 105, from which we obtain two outputs of dimensions 1 x T X _{N , 0} and X _{N , 1} which are respectively the voice channel and the noise channel.

These steps are reiterated on each new frame. The parameters are learned from a sound database. The disadvantage of this method is that it does not use the information from the previous frames to process the current frame. This leads in particular to degraded quality and high latency in processing, due to the duration of the frames.

One of the objectives of the present invention is to provide a method and a device making it possible to separate, in real time, voices from the background noise in an audio stream, or denoising of the voice in an audio stream, in particular by taking counts information from previous frames. This improves performance and processing latency. The method thus enables the propagation of “global information” on the signal, its updating and its use from frame to frame.

• On sépare le flux audio reçu en N trames X_N ,
• Pour chaque trame X_N on associe un tenseur contenant des informations sur l'ensemble du flux audio,
• On transmet la trame X_N à un premier module M₁ qui génère un signal $X_N^{}$${}_{\left(1\right)}^{},$
  
• Le tenseur I _N-1 obtenu lors de l'étape précédente pour le traitement de la trame X_N - 1 est transmis à un module M₃,
• Le module M₃ prend en entrée un signal $X_N^{}$${}_{\left(2\right)}^{},$
  
  résultat de la transformation du signal $X_N^{}$${}_{\left(1\right)}^{}$
  
  par un module M₂ et réalise la concaténation de ${\mathrm{<figure-callout\; id="2"\; label="X\; N"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_N^{}$
  
  et I_N afin de générer un signal $X_N^{\left(3\right)}$
  
  de dimension 2F x T,
• Le signal $X_N^{\left(3\right)}$
  
  est transmis à un module M₄ afin de générer un signal $X_N^{\left(4\right)}$
  
  qui est combiné avec le signal $X_N^{}$${}_{\left(1\right)}^{},$
  
• Le signal résultant de la combinaison est décodé par un décodeur M₅ afin de générer un premier signal de voix X _N,0 et un deuxième signal X _N,1.

The invention relates to a method for separating voice in real time from noise in an audio signal received on a receiver equipped with an audio sensor, characterized in that it comprises at least the following steps:

• The audio stream received is separated into N frames X _N ,
• For each frame X _N we associate a tensor containing information on the whole audio stream,
• The frame X _N is transmitted to a first module M ₁ which generates a signal $X_{\mathrm{NOT}}^{\left(1\right)},$
  
• The tensor I _{N -1} obtained during the previous step for the processing of the frame X _N - 1 is transmitted to a module M ₃ ,
• The M ₃ module takes a signal as input $X_{\mathrm{NOT}}^{}$${}_{\left(2\right)}^{},$
  
  signal transformation result $X_{\mathrm{NOT}}^{}$${}_{\left(1\right)}^{}$
  
  by a module M ₂ and performs the concatenation of ${\mathrm{<figure-callout\; id="2"\; label="X\; NOT"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_{\mathrm{NOT}}^{}$
  
  and I _{N in} order to generate a signal $X_{\mathrm{NOT}}^{\left(3\right)}$
  
  of dimension 2F x T,
• The signal $X_{\mathrm{NOT}}^{\left(3\right)}$
  
  is transmitted to a module M _{4 in} order to generate a signal $X_{\mathrm{NOT}}^{\left(4\right)}$
  
  which is combined with the signal $X_{\mathrm{NOT}}^{\left(1\right)},$
  
• The signal resulting from the combination is decoded by a decoder M _{5 in} order to generate a first voice signal X _{N , 0} and a second signal X _{N, 1} .

To process an N frame, it is assumed that the N - 1 frame has been processed previously and that the quantities resulting from this processing have been stored. For frame 0, I ₀ is set arbitrarily, for example it is identically zero.

• Un premier module M₁ recevant des trames d'un signal contenant de la voix et du bruit,
• Le premier module à une sortie reliée à un deuxième module M₂ configuré pour générer un signal transmis à un troisième module M₃ qui reçoit une valeur de tenseur associée à une trame précédente X_N - 1 pour générer un tenseur I_N associé à la trame courante et un signal $X_N^{\left(3\right)}$
  
  de dimension 2F x T,

The invention also relates to a device for separating voice from noise in an audio signal received on a receiver equipped with an audio sensor characterized in that it comprises at least the following elements:

• A first module M ₁ receiving frames of a signal containing voice and noise,
• The first module has an output connected to a second module M ₂ configured to generate a signal transmitted to a third module M ₃ which receives a tensor value associated with a previous frame X _N - 1 to generate a tensor I _N associated with the frame current and a signal $X_{\mathrm{NOT}}^{\left(3\right)}$
  
  of dimension 2F x T,

• Un module M₄ qui combine le signal $X_N^{\left(3\right)}$
  
  et le signal ${\mathrm{<figure-callout\; id="1"\; label="X\; N"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_N^{}$
  
  afin de générer un signal $X_N^{\left(4\right)},$
  
• Un décodeur M₅ configuré pour générer un premier signal de voix X _N,0 et un deuxième signal de bruit X _N,1 à partir du signal $X_N^{\left(4\right)}\mathrm{.}$
  

The module M ₃ inserted between the module M ₂ and the module M ₄ takes at input a tensor which is homogeneous in dimensions to that supplied at the output of the module M ₂ and provides at the output a homogeneous tensor in dimensions to that which the module M takes at the input ₄ . An additional input I _N - 1 is supplied at the input of the module M ₃ for the processing of the frame number N and the module M ₃ provides the tensor I _{N as an} additional output.

• An M ₄ module which combines the signal $X_{\mathrm{NOT}}^{\left(3\right)}$
  
  and the signal $X_{\mathrm{NOT}}^{\left(1\right)}$
  
  in order to generate a signal $X_{\mathrm{NOT}}^{\left(4\right)},$
  
• A decoder M ₅ configured to generate a first voice signal X _{N , 0} and a second noise signal X _{N, 1} from the signal $X_{\mathrm{NOT}}^{\left(4\right)}\mathrm{.}$
  

• [Fig.1], une illustration de l'art antérieur,
• [Fig.2], un exemple de système permettant la mise en œuvre du procédé selon l'invention,
• [Fig.3] une illustration des étapes mises en œuvre par le procédé selon l'invention.

Other characteristics, details and advantages of the invention will emerge on reading the description given with reference to the appended drawings given by way of non-limiting example and which represent, respectively:

• [ Fig. 1 ], an illustration of the prior art,
• [ Fig. 2 ], an example of a system allowing the implementation of the method according to the invention,
• [ Fig. 3 ] an illustration of the steps implemented by the method according to the invention.

The figure 2 illustrates an example of a device allowing the implementation of the method according to the invention.

The signal from which it is necessary to extract (separate) the voice (s) from the noise contained in the audio stream is received on an audio sensor 10. The audio sensor is connected to a set of equipment or Hardware modules 20 configured to separate the voice from the noise which will be detailed in figure 3 .

The figure 3 illustrates a first variant embodiment for separating a voice from the noise in an audio signal, the processing being carried out at the level of the assembly 20. This separation is carried out in real time. Modules similar to the diagram of the figure 1 bear the same references. The assembly also includes a module M ₃ , the function of which is detailed below.

The audio signal received on the sensor is during a first step separated into N frames X ₁ . .... X _N. Each frame X _N is associated with a tensor I _N which is of constant dimension, independent of the index of the frame. The method will update the value of the tensor I _N from frame to frame and the joint use of X _N and I _N to estimate X _{N , 0} and X _{N, 1} .

Le tenseur I _N-1 obtenu lors de l'étape précédente pour le traitement de la trame X_N - 1 est transmis dans un module M₃, 201.The frame X _N is transmitted to a first module M ₁ , 100, which generates a signal $X_{\mathrm{NOT}}^{}$${}_{\left(1\right)}^{}\mathrm{.}$

The tensor I _{N -1} obtained during the previous step for the processing of the frame X _N - 1 is transmitted in a module M ₃ , 201.

M ₃ generates a tensor I _N , 202, which will be used during the processing of the frame X _{N +1} .

203, résultat de la transformation du signal $X_N^{\left(1\right)}$

par un module M₂ et réalise la concaténation de $X_N^{\left(2\right)}$

et I_N , afin de générer un signal $X_N^{\left(3\right)}$

de dimension 2F x T, $$M_{3:}\left(X_N^{\left(2\right)},I_{N-1}\right)->\left(X_N^{\left(3\right)},I_N\right)\mathrm{.}$$

Encoder M ₃ takes a signal as input $X_{\mathrm{NOT}}^{\left(2\right)},$

203, result of signal transformation $X_{\mathrm{NOT}}^{\left(1\right)}$

by a module M ₂ and performs the concatenation of $X_{\mathrm{NOT}}^{\left(2\right)}$

and I _N , in order to generate a signal $X_{\mathrm{NOT}}^{\left(3\right)}$

of dimension 2F x T, $$M_{3:}\left(X_{\mathrm{NOT}}^{\left(2\right)},I_{\mathrm{NOT}-1}\right)->\left(X_{\mathrm{NOT}}^{\left(3\right)},I_{\mathrm{NOT}}\right)\mathrm{.}$$

204, est transmis à un module M₄ afin de générer un signal $X_N^{\left(4\right)}$

qui est combiné, 104, avec le signal $X_N^{\left(1\right)},$

le signal résultant de la combinaison est décodé par un décodeur M₅, 105, afin de générer un premier signal de voix X _N,0 et un deuxième signal de bruit X _N,1.The signal $X_{\mathrm{NOT}}^{\left(3\right)},$

204, is transmitted to a module M _{4 in} order to generate a signal $X_{\mathrm{NOT}}^{\left(4\right)}$

which is combined, 104, with the signal $X_{\mathrm{NOT}}^{\left(1\right)},$

the signal resulting from the combination is decoded by a decoder M ₅ , 105, in order to generate a first voice signal X _{N , 0} and a second noise signal X _{N, 1} .

In one embodiment, the steps implemented by the method according to the invention are as follows:

For all N, I _N has dimension F x F

• a. ${\mathrm{<figure-callout\; id="2"\; label="X\; N"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_N^{}\cdot {\left(X_N^{\left(2\right)}\right)}^t$
  
  est le produit matriciel de ${\mathrm{<figure-callout\; id="2"\; label="X\; N"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_N^{}$
  
  et de sa transposée
  I_N = I _N-1 + λ(A_N - I _N-1) avec λ un facteur de gain 0 et 1 donné par l'utilisateur
  B_N = Softmax(I _N-1)
• a. La fonction softmax est classique en machine learning ; à un vecteur de K nombres, (v ₁ ...v _K ) elle associe un vecteur de K nombre (w ₁ ... w_K ) avec pour tout $w_k=\frac{\exp \left(v_k\right)}{\sum _{l=1}^K\exp \left(v_l\right)},$
  
• b. Pour calculer B_N , la fonction softmax est appliquée indépendamment à toutes les lignes de I_N ,

$C_N=B_N\cdot {\mathrm{<figure-callout\; id="2"\; label="X\; N"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_N^{}$

est le produit matriciel entre B_N et $X_N^{}$${}_{\left(2\right)}^{};$

ses dimensions sont F×T, $X_N^{\left(3\right)}$

est de dimension 2F x T, c'est la concaténation de ${\mathrm{<figure-callout\; id="2"\; label="X\; N"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_N^{}$

et C_N . A _N is a tensor F x F defined by ${\mathrm{AT}}_{\mathrm{NOT}}=\left(\frac{X_{\mathrm{NOT}}^{\left(2\right)}\cdot {\left(X_{\mathrm{NOT}}^{\left(2\right)}\right)}^t}{\sqrt{T}}\right)$

• at. $X_{\mathrm{NOT}}^{\left(2\right)}\cdot {\left(X_{\mathrm{NOT}}^{\left(2\right)}\right)}^t$
  
  is the matrix product of $X_{\mathrm{NOT}}^{\left(2\right)}$
  
  and its transpose
  I _N = I _{N -1} + λ ( A _N - I _{N -1} ) with λ a gain factor 0 and 1 given by the user
  B _N = Softmax ( I _{N -1} )
• at. The softmax function is classic in machine learning; to a vector of K numbers, ( v ₁ ... v _K ) it associates a vector of K number ( w ₁ ... w _K ) with for all $w_k=\frac{\exp \left(v_k\right)}{\sum _{l=1}^K\exp \left(v_l\right)},$
  
• b. To calculate B _N , the softmax function is applied independently to all the lines of I _N ,

${\mathrm{VS}}_{\mathrm{NOT}}=B_{\mathrm{NOT}}\cdot {\mathrm{<figure-callout\; id="2"\; label="X\; NOT"\; filenames="imgf0001.png"\; state="\{\{state\}\}">X</figure-callout>}}_{\mathrm{NOT}}^{}$

is the matrix product between B _N and $X_{\mathrm{NOT}}^{\left(2\right)};$

its dimensions are F × T , $X_{\mathrm{NOT}}^{\left(3\right)}$

is of dimension 2F x T, it is the concatenation of $X_{\mathrm{NOT}}^{\left(2\right)}$

and C _N.

The method and the device according to the invention allow real-time separation of the voice from the noise in an audio signal received on a sensor in real time and without degrading the parameters specific to the voice.

The digital parameters defining the processing of the different modules are set in a prior learning phase on a database.

The invention allows real-time operation with a controllable latency / quality compromise, so as not to degrade the audio signal which does not contain noise, and makes it possible to enhance the noise in a signal which does not contain words (voice).

The method makes it possible in particular to preprocess the audio signal of speech to improve the quality of the voice processing / enhancement bricks (compression, analysis).

The addition in the processing chain of a module M ₃ makes it possible to improve the quality of implementation of a frame-by-frame strategy for real-time processing.

Claims (2)

Process for separating, in real time, voice from noise in an audio signal received on a receiver equipped with an audio sensor characterized in that it comprises at least the following steps: - We separate the audio stream received into N frames X _N , - For each frame X _N we associate a tensor containing information on the whole audio stream, - The frame X _N is transmitted to a first module M ₁ , (100), which generates a signal $X_{\mathrm{NOT}}^{\left(1\right)},$

- The tensor I _{N -1} obtained during the previous step for the processing of the frame X _N - 1 is transmitted to a module M ₃ , (201), - The M ₃ module takes a signal as input $X_{\mathrm{NOT}}^{\left(2\right)},$

(203), result of the transformation of the signal $X_{\mathrm{NOT}}^{\left(1\right)}$

by a module M ₂ and performs the concatenation of $X_{\mathrm{NOT}}^{\left(2\right)}$

and I _N , in order to generate a signal $X_{\mathrm{NOT}}^{\left(3\right)}$

of dimension 2F x T, - The signal $X_{\mathrm{NOT}}^{\left(3\right)},$

(204), is transmitted to a module M _{4 in} order to generate a signal $X_{\mathrm{NOT}}^{\left(4\right)}$

which is combined, (104), with the signal $X_{\mathrm{NOT}}^{\left(1\right)},$

- the signal resulting from the combination is decoded by a decoder M ₅ , (105), in order to generate a first voice signal X _{N , 0} and a second signal X _{N, 1} . Device for separating, in real time, voice from noise in an audio signal received on a receiver equipped with an audio sensor, characterized in that it comprises at least the following elements: - A first module M ₁ receiving frames of a signal containing the voice and noise, - The first module has an output connected to a second module M ₂ configured to generate a signal transmitted to a third module M ₃ which receives a tensor value associated with a previous frame X _N - 1 to generate a tensor I _N associated with the current frame and a signal $X_{\mathrm{NOT}}^{\left(3\right)}$

of dimension 2F x T, - An M ₄ module which combines the signal $X_{\mathrm{NOT}}^{\left(3\right)}$

and the signal $X_{\mathrm{NOT}}^{\left(1\right)}$

in order to generate a signal $X_{\mathrm{NOT}}^{\left(4\right)},$

- A module (104) which combines the signal $X_{\mathrm{NOT}}^{\left(4\right)}$

with the signal $X_{\mathrm{NOT}}^{\left(1\right)}$

in order to generate a signal $X_{\mathrm{NOT}}^{\left(5\right)}$

- A decoder M ₅ , (105) configured to generate a first voice signal X _{N , 0} and a second signal X _{N , 1} from the signal $X_{\mathrm{NOT}}^{\left(5\right)}\mathrm{.}$

Images