CN112863535A - Residual echo and noise elimination method and device - Google Patents

Abstract

The embodiment of the application discloses a method and a device for eliminating residual echo and noise, wherein the method comprises the following steps: performing framing, windowing and Fourier transformation on the received voice time domain signal containing the echo and the noise and the far-end reference sound time domain signal to obtain a corresponding frequency domain signal, determining an echo frequency domain signal, and further determining the voice frequency domain signal containing the residual echo and the noise; respectively carrying out energy normalization processing on the amplitude spectrums of the voice frequency domain signal containing the residual echo and the noise, the echo frequency domain signal and the far-end reference audio frequency domain signal to obtain corresponding characteristics; determining a target voice frequency domain signal according to the corresponding characteristics and the trained cascade network; and carrying out inverse Fourier transform on the target voice frequency domain signal to obtain a target voice time domain signal. According to the embodiment of the application, the feature attention model is used for endowing the input features with different importance, and redundant information in the input features is reduced. And the multi-domain loss function is trained in a cascade network, so that the sensitivity of the model to signal energy is reduced.

Description

A method and device for eliminating residual echo and noise

technical field

The present invention relates to the field of echo and noise cancellation. In particular, it relates to a residual echo and noise cancellation method and device.

Background technique

At present, the echo cancellation technology mainly removes the echo signal formed by the remote reference sound signal in the speech signal, and the speech noise reduction technology mainly removes the background noise and directional noise interference in the speech signal. Both echo cancellation technology and speech noise reduction technology are designed to improve the quality and intelligibility of speech. In the echo cancellation technology, combining the adaptive filtering method based on traditional signal processing and the residual echo cancellation method based on deep learning can effectively improve the generalization performance of the system.

However, residual echo and noise cancellation are often performed independently and separately in traditional methods, without considering the correlation of these two tasks. In the residual echo cancellation task, there are multiple signal features that can be utilized, and these features have different physical meanings and importance, and traditional methods do not consider the different importance of these features. When training residual echo and noise cancellation models, most of the existing technologies use the mean square error of the target amplitude spectrum and the estimated amplitude spectrum as the loss function, but the above loss function depends on the energy of the signal, and the scale of the signal with different energies is also different. will be different.

SUMMARY OF THE INVENTION

Due to the above problems in the existing methods, the embodiments of the present application provide a residual echo and noise cancellation method and apparatus.

In a first aspect, an embodiment of the present application proposes a residual echo and noise cancellation method, including:

Receive speech time-domain signals containing echo and noise and far-end reference acoustic time-domain signals;

Perform framing, windowing and Fourier transform on the voice time domain signal containing echo and noise and the far-end reference acoustic time domain signal respectively to obtain the voice and audio domain signal containing echo and noise and the far-end reference audio frequency domain signal;

Determine an echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal;

According to the voice and audio domain signal containing echo and noise and the echo frequency domain signal, determine a voice and audio domain signal containing residual echo and noise;

Perform energy normalization processing on the amplitude spectrum of the voice and audio frequency domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal to obtain residual echo and Noise's voice and audio frequency domain signal characteristics, echo frequency domain signal characteristics and far-end reference audio frequency domain signal characteristics;

The voice and audio domain signal features containing residual echo and noise are spliced with the far-end reference audio frequency domain signal features to obtain a first splicing result, and the voice and audio domain signal features containing residual echo and noise are combined with the above. The echo frequency domain signal features are spliced to obtain a second splicing result;

The first splicing result and the second splicing result are input into the post-training feature attention model in the post-training cascade network to obtain the first attention weight and a second attention weight corresponding to the echo frequency domain signal feature;

Multiplying the remote reference audio frequency domain signal feature and the first attention weight to obtain the first fusion attention mechanism feature, and multiplying the echo frequency domain signal feature and the second attention weight to obtain the second fusion Attention mechanism features;

Splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing result;

Inputting the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain a masking estimate value of the target speech and audio domain signal;

According to the masking estimate value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise, obtain the target voice and audio frequency domain signal;

Inverse Fourier transform is performed on the target speech and audio frequency domain signal to obtain the target speech time domain signal.

In a possible implementation, performing framing, windowing and Fourier transform on the voice time-domain signal containing echo and noise and the far-end reference sound time-domain signal respectively, including:

Taking a preset number of sampling points as a frame of signals for the voice time-domain signal containing echo and noise and the far-end reference acoustic time-domain signal respectively; if the length is insufficient, first fill with zeros to a preset number;

Windowing is performed on each frame of signal; wherein, the windowing function adopts a Hamming window;

Fourier transform is performed on each frame of the windowed signal.

In a possible implementation, determining the echo frequency domain signal according to the voice and audio domain signal containing echo and noise and the far-end reference audio frequency domain signal includes:

Inputting the voice and audio frequency domain signals containing echo and noise and the far-end reference audio frequency domain signals into a Kalman filter to obtain filter coefficients and the echo frequency domain signals;

The echo frequency domain signal is the result of multiplying the filter coefficients and the far-end reference audio frequency domain signal.

In a possible implementation, determining the voice and audio domain signal containing residual echo and noise according to the voice and audio domain signal containing echo and noise and the echo frequency domain signal, including:

The echo frequency domain signal is subtracted from the voice and audio domain signal containing echo and noise to obtain the voice and audio domain signal containing residual echo and noise.

In a possible implementation, performing energy energy analysis on the amplitude spectrum of the voice domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal, and the amplitude spectrum of the far-end reference audio frequency domain signal. Normalization processing is performed to obtain the voice and audio domain signal features containing residual echo and noise, the echo frequency domain signal features and the far-end reference audio frequency domain signal features, including:

Determine the corresponding first function, second function and third function;

According to the first function corresponding to the amplitude spectrum of the voice domain signal containing residual echo and noise, and the mean and variance of the features of the voice domain signal containing residual echo and noise, determine the residual echo and noise containing voice domain signal. Voice domain signal features;

According to the second function corresponding to the amplitude spectrum of the echo frequency domain signal, and the mean value and variance of the echo frequency domain signal feature, determine the echo frequency domain signal feature;

The feature of the far-end reference audio-frequency domain signal is determined according to the third function corresponding to the amplitude spectrum of the far-end reference audio-frequency domain signal, and the mean and variance of the feature of the far-end reference audio-frequency domain signal.

In a possible implementation, the post-training cascade network is obtained by training through the following steps:

receiving the first voice time domain signal containing echo and noise, the first remote reference acoustic time domain signal and the first target voice time domain signal;

Perform framing, windowing and Fourier transform on the first voice time-domain signal containing echo and noise, the first far-end reference acoustic time-domain signal and the first target voice time-domain signal, respectively, to obtain a first voice domain signal containing echo and noise, a first remote reference voice domain signal and a first target voice domain signal;

determining a first echo frequency domain signal according to the first voice frequency domain signal containing echo and noise and the first remote reference audio frequency domain signal;

According to the first voice and audio domain signal containing echo and noise and the first echo frequency domain signal, determine a first voice and audio domain signal containing residual echo and noise;

Perform energy normalization processing on the amplitude spectrum of the first voice-frequency domain signal containing residual echo and noise, the amplitude spectrum of the first echo frequency-domain signal, and the amplitude spectrum of the first far-end reference audio-frequency domain signal , obtain the first voice-frequency domain signal feature containing residual echo and noise, the first echo frequency-domain signal feature and the first remote reference audio-frequency domain signal feature;

Splicing the first voice and audio domain signal features containing residual echo and noise with the first remote reference audio frequency domain signal features to obtain a first splicing feature, and splicing the first voice and audio signal features containing residual echo and noise. The audio domain signal feature is spliced with the first echo frequency domain signal feature to obtain a second splicing feature;

The first splicing feature and the second splicing feature are input to the feature attention model in the cascade network, to jointly train the feature attention model and the residual echo and noise cancellation model in the cascade network, and the result is obtained with the first a first weight corresponding to a remote reference audio frequency domain signal feature and a second weight corresponding to the first echo frequency domain signal feature;

The first remote reference audio frequency domain signal feature is multiplied by a first weight to obtain a first fusion feature, and the first echo frequency domain signal feature is multiplied by a second weight to obtain a second fusion feature;

Splicing the first fusion feature, the second fusion feature, and the first voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing feature;

Inputting the first fusion splicing feature into the residual echo and noise cancellation model in the cascaded network to obtain a masking estimate of the second target voice and audio domain signal;

According to the masking estimate value of the second target voice domain signal and the first voice domain signal containing residual echo and noise, determine the second target voice domain signal;

According to at least two loss functions, a multi-domain loss function is determined; wherein, the at least two loss functions include an energy-independent amplitude spectrum loss function and an objective speech quality assessment score loss function; the energy-independent amplitude spectrum loss function is The amplitude spectrum of the first target speech and audio domain signal is a training target, and is determined according to the second target speech and audio domain signal; the objective speech quality evaluation score loss function is determined by taking improving the audio quality of speech as a training target;

The post-training cascade network is obtained by continuously reducing the multi-domain loss function iteratively with model parameters.

In a second aspect, an embodiment of the present application further provides a residual echo and noise cancellation device, including:

The receiving module is used to receive the voice time domain signal containing echo and noise and the far-end reference sound time domain signal;

The processing module is used to perform framing, windowing and Fourier transform on the voice time-domain signal containing echo and noise and the far-end reference acoustic time-domain signal respectively, so as to obtain the voice and audio domain signal containing echo and noise and the far-end reference audio domain signal;

a determining module, configured to determine an echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal;

The determining module is further configured to determine a voice and audio domain signal containing residual echo and noise according to the voice and audio domain signal containing echo and noise and the echo frequency domain signal;

An energy normalization module, configured to perform energy normalization on the amplitude spectrum of the voice and audio domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal process to obtain the voice and audio domain signal features containing residual echo and noise, the echo frequency domain signal features and the far-end reference audio frequency domain signal features;

The splicing module is used for splicing the voice and audio domain signal features containing residual echo and noise with the remote reference audio frequency domain signal features to obtain a first splicing result, and splicing the voice and audio containing residual echo and noise. The domain signal feature is spliced with the echo frequency domain signal feature to obtain a second splicing result;

A weight obtaining module is used to input the first splicing result and the second splicing result into the post-training feature attention model in the post-training cascade network, and obtain the corresponding feature of the far-end reference audio frequency domain signal. a first attention weight and a second attention weight corresponding to the echo frequency domain signal feature;

The fused attention mechanism feature acquisition module is used to multiply the far-end reference audio frequency domain signal feature with the first attention weight to obtain the first fused attention mechanism feature, and combine the echo frequency domain signal feature with the second The attention weights are multiplied to obtain the second fusion attention mechanism feature;

The splicing module is further configured to splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing result;

a masking estimation value obtaining module, which is used to input the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain the masking estimation value of the target speech and audio domain signal;

a target voice and audio domain signal obtaining module, configured to obtain the target voice and audio domain signal according to the masking estimate value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise;

The inverse Fourier transform module is used for performing inverse Fourier transform on the target speech and audio domain signal to obtain the target speech time domain signal.

In a third aspect, an embodiment of the present application further provides a residual echo and noise cancellation device, including at least one processor, where the processor is configured to execute a program stored in a memory, and when the program is executed, the device is made to execute As in the first aspect and various steps in various possible implementations.

In a fourth aspect, the embodiments of the present application further provide a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements each of the first aspect and various possible implementations. step.

The beneficial effect of the embodiments of the present application is that the residual echo and noise do not need to be eliminated separately, but the residual echo and noise are eliminated at one time through a cascaded network after training. The post-training feature attention model is used to assign different importance to the input features, reduce redundant information in the input features, and improve the performance of the cascade network in eliminating residual echo and noise. Using a multi-domain loss function that combines the energy-independent amplitude spectrum loss function and the objective speech quality assessment score loss function to train the cascade network reduces the sensitivity of the model to signal energy and improves the auditory perception quality of the output speech.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic diagram of a process of training a cascaded network capable of eliminating residual echo and noise according to an embodiment of the present application;

2 is a schematic flowchart of using a post-training cascade network to eliminate residual echo and noise according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a residual echo and noise cancellation apparatus provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

Residual echo and noise cancellation are often performed independently in traditional methods. In the residual echo suppression task, the respective importance of multiple signal features is not considered. When training residual echo and noise cancellation models, the mean square error of the target amplitude spectrum and the estimated amplitude spectrum is mostly used as the loss function. The above loss function depends on the magnitude of the signal energy. Therefore, the present application proposes a residual echo and noise elimination method and device, which can assign different importance to multiple signal features, reduce redundant information in multiple signal features, and at the same time use multi-domain loss functions to cascade The training of the network reduces the sensitivity of the model to signal energy.

In the embodiment of the present application, a schematic diagram of the process of training a cascaded network capable of eliminating residual echo and noise is shown in FIG. 1, including: S101-S113; wherein, the cascade network includes a feature attention model and residual echo and noise cancellation Model.

The feature attention model consists of 1 layer of gated recurrent units connected to 1 layer of fully connected neural network. The above-mentioned 1-layer gated recurrent unit has 200 hidden layer nodes, the above-mentioned fully connected neural network output layer has 257 nodes, and the activation function of each neuron uses the Sigmoid function.

The residual echo and noise cancellation model consists of 2 layers of gated recurrent units connected to 1 layer of fully connected neural network. The above-mentioned 2-layer gated recurrent unit includes 400 hidden layer nodes, the above-mentioned fully connected neural network output layer has 257 nodes, and the activation function of each neuron uses the Sigmoid function.

S101: Receive a first voice time domain signal containing echo and noise, a first remote reference acoustic time domain signal, and a first target voice time domain signal. Wherein, the first remote reference acoustic time domain signal is nonlinearly transformed and then convolved with a corresponding room transfer function to form an echo time domain signal in the first voice time domain signal containing echo and noise.

S102: Framing and windowing the received first voice time domain signal containing echo and noise, the first remote reference acoustic time domain signal and the first target voice time domain signal. Specifically, 512 sampling points are respectively taken as one frame signal for the received first voice time domain signal containing echo and noise, the first remote reference voice time domain signal and the first target voice time domain signal. First fill zero to 512 points, and then add window to each frame of signal, and the windowing function adopts Hamming window. Fourier transform is performed on each frame of signal after windowing to obtain a first voice and audio domain signal containing echo and noise, a first far-end reference audio frequency domain signal and a first target voice and audio domain signal.

S103: Input the first voice and audio domain signal containing echo and noise and the first remote reference audio frequency domain signal into a Kalman filter, and estimate the first filter coefficient and the first echo frequency domain signal in real time. Among them, the first echo frequency domain signal estimated by the Kalman filter is:

C(k,f)=W(k,f)*X(k,f)

Wherein, W(k, f) is the first filter coefficient, X(k, f) is the far-end reference audio frequency domain signal, and k and f represent the kth frame and frequency f, respectively.

S104: Subtract the first echo frequency domain signal from the first voice and audio domain signal containing echo and noise to obtain a first voice and audio domain signal containing residual echo and noise, which is used as an output result of the Kalman filter. The above-mentioned first voice domain signal containing residual echo and noise is:

E(k,f)=Y(k,f)-C(k,f)

Among them, Y(k, f) is the first voice and audio domain signal containing echo and noise, C(k, f) is the first echo frequency domain signal, k and f represent the kth frame and frequency f respectively.

S105, performing energy normalization processing on the amplitude spectrum of the above-mentioned first voice-frequency domain signal containing residual echo and noise, the amplitude spectrum of the above-mentioned first echo frequency-domain signal, and the amplitude spectrum of the above-mentioned first remote reference audio-frequency domain signal, Obtain the first voice domain signal feature g _FD (f(|E(k,f)|)) containing residual echo and noise, and the first echo frequency domain signal feature g _FD (f(|C(k,f)| )) and the first far-end reference audio frequency domain signal feature g _FD (f(|X(k,f)|)). in,

The mean and variance of the first voice domain signal features containing residual echo and noise are defined as:

μ _f(e) (k,f)=c ₁ μ _f(e) (k-1,f)+(1-c ₁ )f(|E(k,f)|)

The mean and variance of the first echo frequency domain signal features are defined as:

μ _f(c) (k,f)=c ₁ μ _f(c) (k-1,f)+(1-c ₁ )f(|C(k,f)|)

The mean and variance of the first remote reference audio frequency domain signal feature are defined as:

μ _f(x) (k,f)=c ₁ μ _f(x) (k-1,f)+(1-c ₁ )f(|X(k,f)|)

|E(k,f)|, |C(k,f)| and |X(k,f)| represent the amplitude spectrum of the first voice-frequency domain signal containing residual echo and noise, and the first echo frequency-domain signal, respectively and the amplitude spectrum of the first remote reference audio frequency domain signal, c ₁ is a preset constant.

S106, splicing the above-mentioned first voice and audio domain signal features containing residual echo and noise with the above-mentioned first remote reference audio frequency domain signal features to obtain a first splicing feature, and splicing the above-mentioned first voice and audio containing residual echo and noise. The domain signal feature is spliced with the above-mentioned first echo frequency domain signal feature to obtain the second splicing feature.

S107, the above-mentioned first splicing feature and the above-mentioned second splicing feature are input into the feature attention model in the cascade network, to jointly train the feature attention model and the residual echo and noise elimination model in the cascade network, to obtain a A first weight α(k, f) corresponding to the feature of the far-end reference audio frequency domain signal and a second weight β(k, f) corresponding to the above-mentioned first echo frequency domain signal feature.

S108, the above-mentioned first remote reference audio frequency domain signal feature is multiplied by the first weight, and the obtained first fusion feature is:

X _att (k,f)=g _FD (f(|X(k,f)|))*α(k,f)

And the above-mentioned first echo frequency domain signal feature is multiplied by the second weight, and the second fusion feature is obtained as:

C _att (k,f)=g _FD (f(|C(k,f)|))*β(k,f).

S109, combine the above-mentioned first fusion feature X _att (k, f), the above-mentioned second fusion characteristic C _att (k, f) and the above-mentioned first voice and audio domain signal feature g _FD (f(|E (k,f)|)) for splicing to obtain the first fusion splicing feature.

S110: Input the first fusion and splicing feature into the residual echo and noise elimination model in the cascaded network, and output the masking estimate value G(k, f) of the second target voice and audio domain signal.

S111, using the masking estimated value G(k, f) of the above-mentioned second target speech and audio domain signal to enhance the above-mentioned first speech and audio domain signal containing residual echo and noise, and obtaining the second target speech and audio frequency domain signal is:

S112: Determine a multi-domain loss function according to at least two loss functions. For example, taking the amplitude spectrum of the first target voice and audio domain signal as the training target, and determining the energy-independent amplitude spectrum loss function according to the second target voice and audio domain signal

其中，S(k,f)为第一目标语音频域信号，

为第二目标语音频域信号。加权相加上述能量无关的幅度谱损失函数和上述客观语音质量评估得分损失函数，确定多域的损失函数为：To improve the audio quality of speech as the training goal, determine the objective speech quality evaluation score loss function

Among them, S(k,f) is the first target speech audio domain signal,

is the second target voice domain signal. Weighted addition of the above energy-independent amplitude spectrum loss function and the above objective speech quality assessment score loss function, the multi-domain loss function is determined as:

Among them, λ is a preset constant.

S113, by continuously reducing the multi-domain loss function by iteratively reducing the model parameters, a post-training cascade network is obtained. Among them, the post-training cascade network includes a post-training feature attention model and a post-training residual echo and noise cancellation model.

In the embodiment of the present application, a schematic flowchart of using the above-mentioned post-training cascade network to eliminate residual echo and noise is shown in FIG. 2 , including: S201-S207;

S201, a speech time domain signal containing echo and noise and a far-end reference acoustic time domain signal are received. Wherein, the remote reference acoustic time domain signal is nonlinearly transformed and then convolved with the corresponding room transfer function to form the echo time domain signal in the speech time domain signal containing echo and noise.

S202: Framing and windowing the above-mentioned voice time-domain signal containing echo and noise and the above-mentioned far-end reference acoustic time-domain signal respectively. Specifically, take 512 sampling points for the received voice time-domain signal containing echo and noise, and the far-end reference sound time-domain signal as a frame signal. The signal is windowed, and the windowing function uses a Hamming window. Fourier transform is performed on each frame of the windowed signal to obtain a voice and audio domain signal containing echo and noise and a far-end reference audio frequency domain signal.

S203: Input the above-mentioned voice and audio frequency domain signal containing echo and noise and the above-mentioned far-end reference audio frequency domain signal into a Kalman filter, and estimate the filter coefficients and the echo frequency domain signal in real time. Among them, the echo frequency domain signal estimated by the Kalman filter is:

C ₃ (k,f)=W ₃ (k,f)*X ₃ (k,f)

Wherein, W ₃ (k, f) is the first filter coefficient, X ₃ (k, f) is the far-end reference audio domain signal, and k and f represent the k-th frame and frequency f, respectively.

S204, the above-mentioned echo frequency domain signal is subtracted from the above-mentioned voice and audio domain signal containing echo and noise, and the obtained voice and audio frequency domain signal containing residual echo and noise is:

E ₃ (k,f)=Y ₃ (k,f)-C ₃ (k,f)

Among them, Y ₃ (k, f) is the voice and audio domain signal containing echo and noise, C ₃ (k, f) is the echo frequency domain signal, k and f represent the k-th frame and frequency f, respectively.

第一回声频域信号特征

和第一远端参考声频域信号特征

其中，S205, performing energy normalization processing on the amplitude spectrum of the voice and audio domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal, and the amplitude spectrum of the remote reference audio frequency domain signal to obtain residual echo and noise containing The features of the speech domain signal

The first echo frequency domain signal characteristics

and the first far-end reference audio domain signal characteristics

in,

The mean and variance of the first voice domain signal features containing residual echo and noise are defined as:

The mean and variance of the first echo frequency domain signal features are defined as:

The mean and variance of the first remote reference audio frequency domain signal feature are defined as:

|E ₃ (k,f)|, |C ₃ (k,f)| and |X ₃ (k,f)| represent the amplitude spectrum of the voice-frequency domain signal containing residual echo and noise, and the echo frequency domain signal’s amplitude spectrum, respectively. The amplitude spectrum and the amplitude spectrum of the far-end reference audio frequency domain signal, c ₂ is a preset constant.

S206, splicing the above-mentioned voice and audio domain signal features containing residual echo and noise with the above-mentioned remote reference audio frequency domain signal features to obtain a first splicing result, and combining the above-mentioned voice and audio domain signal features containing residual echo and noise with the above echo The audio frequency domain signal features are spliced to obtain a second splicing result.

S207, input the above-mentioned first splicing result and the above-mentioned second splicing result into the above-mentioned post-training cascade network, that is, input the post-training feature attention model to obtain the first attention weight α corresponding to the above-mentioned remote reference audio frequency domain signal feature ₃ (k, f) and the second attention weight β ₃ (k, f) corresponding to the above echo frequency domain signal features.

S208, the above-mentioned remote reference audio frequency domain signal feature is multiplied by the first attention weight α ₃ (k, f) to obtain the first fusion attention mechanism feature:

And the above echo frequency domain signal features are multiplied by the second attention weight β ₃ (k, f) to obtain the second fusion attention mechanism features:

S209, splicing the above-mentioned first fusion attention mechanism feature, the above-mentioned second fusion attention mechanism characteristic, and the above-mentioned voice and audio domain signal characteristics containing residual echo and noise, to obtain a first fusion and splicing result.

S210: Input the above-mentioned first fusion and splicing result into a post-training residual echo and noise cancellation model in the post-training cascade network to obtain a masking estimation value G ₃ (k, f) of the target speech and audio domain signal.

S211: Multiply the masking estimate value G ₃ (k, f) of the above-mentioned target voice and audio domain signal and the voice and audio domain signal E ₃ (k, f) containing residual echo and noise, to obtain the target voice and audio domain signal as :

S212 , performing an inverse Fourier transform on the target speech and audio domain signals to obtain a target speech time domain signal.

The residual echo and noise in the embodiments of the present application do not need to be eliminated separately, but the residual echo and noise are eliminated at one time through a cascaded network after training. The post-training feature attention model is used to assign different importance to the input features, reduce redundant information in the input features, and improve the performance of the cascade network in eliminating residual echo and noise. Using a multi-domain loss function that combines the energy-independent amplitude spectrum loss function and the objective speech quality assessment score loss function to train the cascade network reduces the sensitivity of the model to signal energy and improves the auditory perception quality of the output speech.

An embodiment of the present application provides a residual echo and noise cancellation device, the schematic structural diagram of which is shown in FIG. 3 , including:

a receiving module 301, a processing module 302, a determination module 303, an energy normalization module 304 and an inverse Fourier transform module 305;

a receiving module 301, configured to receive a voice time-domain signal containing echo and noise and a far-end reference acoustic time-domain signal;

The processing module 302 is used to perform framing, windowing and Fourier transform on the voice time domain signal containing echo and noise and the far-end reference acoustic time domain signal respectively, to obtain a voice and audio domain containing echo and noise signal and far-end reference audio domain signal;

a determining module 303, configured to determine an echo frequency domain signal according to the voice and audio domain signal containing echo and noise and the far-end reference audio frequency domain signal;

The determining module 303 is further configured to determine a voice and audio domain signal containing residual echo and noise according to the voice and audio domain signal containing echo and noise and the echo frequency domain signal;

The energy normalization module 304 is configured to perform energy normalization on the amplitude spectrum of the speech audio frequency domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal. Normalization processing to obtain the voice and audio domain signal features containing residual echo and noise, the echo frequency domain signal features and the far-end reference audio frequency domain signal features;

The splicing module 305 is used for splicing the voice and audio domain signal features containing residual echo and noise with the remote reference audio frequency domain signal features to obtain a first splicing result, and splicing the voice and audio domain signal features containing residual echo and noise. The audio domain signal feature and the echo frequency domain signal feature are spliced to obtain a second splicing result;

The weight obtaining module 306 is used to input the first splicing result and the second splicing result into the post-training feature attention model in the post-training cascade network, and obtain the feature corresponding to the far-end reference audio frequency domain signal The first attention weight of and the second attention weight corresponding to the echo frequency domain signal feature;

The fusion attention mechanism feature obtaining module 307 is used for multiplying the remote reference audio frequency domain signal feature and the first attention weight to obtain the first fusion attention mechanism feature, and combining the echo frequency domain signal feature with the first attention weight. The two attention weights are multiplied to obtain the second fusion attention mechanism feature;

The splicing module 305 is further configured to splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion. splicing result;

A masking estimate value obtaining module 308, configured to input the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain a masking estimate value of the target speech and audio domain signal;

A target voice and audio domain signal obtaining module 309, configured to obtain the target voice and audio domain signal according to the masking estimated value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise;

The inverse Fourier transform module 310 is configured to perform inverse Fourier transform on the target speech and audio domain signal to obtain the target speech time domain signal.

An embodiment of the present application provides a residual echo and noise cancellation device, including at least one processor, where the processor is configured to execute a program stored in a memory, and when the program is executed, the device is made to perform the following steps:

Receive speech time-domain signals containing echo and noise and far-end reference acoustic time-domain signals;

Perform framing, windowing and Fourier transform on the voice time-domain signal containing echo and noise and the far-end reference acoustic time-domain signal, respectively, to obtain the voice and audio domain signal containing echo and noise and the far-end reference audio frequency domain signal;

Determine an echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal;

According to the voice and audio domain signal containing echo and noise and the echo frequency domain signal, determine a voice and audio domain signal containing residual echo and noise;

Perform energy normalization processing on the amplitude spectrum of the voice and audio frequency domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal to obtain residual echo and Noise's voice and audio frequency domain signal characteristics, echo frequency domain signal characteristics and far-end reference audio frequency domain signal characteristics;

The voice and audio domain signal features containing residual echo and noise are spliced with the far-end reference audio frequency domain signal features to obtain a first splicing result, and the voice and audio domain signal features containing residual echo and noise are combined with the above. The echo frequency domain signal features are spliced to obtain a second splicing result;

The first splicing result and the second splicing result are input into the post-training feature attention model in the post-training cascade network to obtain the first attention weight and a second attention weight corresponding to the echo frequency domain signal feature;

Multiplying the remote reference audio frequency domain signal feature and the first attention weight to obtain the first fusion attention mechanism feature, and multiplying the echo frequency domain signal feature and the second attention weight to obtain the second fusion Attention mechanism features;

Splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing result;

Inputting the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain a masking estimate value of the target speech and audio domain signal;

According to the masking estimate value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise, obtain the target voice and audio frequency domain signal;

Inverse Fourier transform is performed on the target speech and audio frequency domain signal to obtain the target speech time domain signal.

An embodiment of the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Receive speech time-domain signals containing echo and noise and far-end reference acoustic time-domain signals;

Perform framing, windowing and Fourier transform on the voice time domain signal containing echo and noise and the far-end reference acoustic time domain signal respectively to obtain the voice and audio domain signal containing echo and noise and the far-end reference audio frequency domain signal;

Determine an echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal;

According to the voice and audio domain signal containing echo and noise and the echo frequency domain signal, determine a voice and audio domain signal containing residual echo and noise;

Perform energy normalization processing on the amplitude spectrum of the voice and audio frequency domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal to obtain residual echo and Noise's voice and audio frequency domain signal characteristics, echo frequency domain signal characteristics and far-end reference audio frequency domain signal characteristics;

The voice and audio domain signal features containing residual echo and noise are spliced with the far-end reference audio frequency domain signal features to obtain a first splicing result, and the voice and audio domain signal features containing residual echo and noise are combined with the above. The echo frequency domain signal features are spliced to obtain a second splicing result;

The first splicing result and the second splicing result are input into the post-training feature attention model in the post-training cascade network to obtain the first attention weight and a second attention weight corresponding to the echo frequency domain signal feature;

Multiplying the remote reference audio frequency domain signal feature and the first attention weight to obtain the first fusion attention mechanism feature, and multiplying the echo frequency domain signal feature and the second attention weight to obtain the second fusion Attention mechanism features;

Splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing result;

Inputting the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain a masking estimate value of the target speech and audio domain signal;

According to the masking estimate value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise, obtain the target voice and audio frequency domain signal;

Inverse Fourier transform is performed on the target speech and audio frequency domain signal to obtain the target speech time domain signal.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

It should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be used for The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (9)

1. a residual echo and noise cancellation method, is characterized in that, comprises: Receive speech time-domain signals containing echo and noise and far-end reference acoustic time-domain signals; Perform framing, windowing and Fourier transform on the voice time domain signal containing echo and noise and the far-end reference acoustic time domain signal respectively to obtain the voice and audio domain signal containing echo and noise and the far-end reference audio frequency domain signal; Determine an echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal; According to the voice and audio domain signal containing echo and noise and the echo frequency domain signal, determine a voice and audio domain signal containing residual echo and noise; Perform energy normalization processing on the amplitude spectrum of the voice and audio frequency domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal to obtain residual echo and Noise's voice and audio frequency domain signal characteristics, echo frequency domain signal characteristics and far-end reference audio frequency domain signal characteristics; The voice and audio domain signal features containing residual echo and noise are spliced with the far-end reference audio frequency domain signal features to obtain a first splicing result, and the voice and audio domain signal features containing residual echo and noise are combined with the above. The echo frequency domain signal features are spliced to obtain a second splicing result; The first splicing result and the second splicing result are input into the post-training feature attention model in the post-training cascade network to obtain the first attention weight and a second attention weight corresponding to the echo frequency domain signal feature; Multiplying the remote reference audio frequency domain signal feature and the first attention weight to obtain the first fusion attention mechanism feature, and multiplying the echo frequency domain signal feature and the second attention weight to obtain the second fusion Attention mechanism features; Splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing result; Inputting the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain a masking estimate value of the target speech and audio domain signal; According to the masking estimate value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise, obtain the target voice and audio frequency domain signal; Inverse Fourier transform is performed on the target speech and audio frequency domain signal to obtain the target speech time domain signal. 2. The method according to claim 1, wherein the described voice time-domain signal containing echo and noise and the far-end reference acoustic time-domain signal are respectively framed, windowed and Fourier transformations, including: Taking a preset number of sampling points as a frame of signals for the voice time-domain signal containing echo and noise and the far-end reference acoustic time-domain signal respectively; if the length is insufficient, first fill with zeros to a preset number; Windowing is performed on each frame of signal; wherein, the windowing function adopts a Hamming window; Fourier transform is performed on each frame of the windowed signal. 3. The method according to claim 1, wherein, determining the echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal, comprising: Inputting the voice and audio frequency domain signals containing echo and noise and the far-end reference audio frequency domain signals into a Kalman filter to obtain filter coefficients and the echo frequency domain signals; The echo frequency domain signal is the result of multiplying the filter coefficients and the far-end reference audio frequency domain signal. 4 . The method according to claim 1 , wherein determining the voice and audio domain signal containing residual echo and noise according to the voice and audio domain signal containing echo and noise and the echo frequency domain signal, comprising: 5 . : The echo frequency domain signal is subtracted from the voice and audio domain signal containing echo and noise to obtain the voice and audio domain signal containing residual echo and noise. 5 . The method according to claim 1 , wherein the method of comparing the amplitude spectrum of the voice domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the far-end reference audio The amplitude spectrum of the domain signal is energy normalized to obtain the voice and audio domain signal features containing residual echo and noise, the echo frequency domain signal features and the far-end reference audio frequency domain signal features, including: Determine the corresponding first function, second function and third function; According to the first function corresponding to the amplitude spectrum of the voice domain signal containing residual echo and noise, and the mean and variance of the features of the voice domain signal containing residual echo and noise, determine the residual echo and noise containing voice domain signal. Voice domain signal features; According to the second function corresponding to the amplitude spectrum of the echo frequency domain signal, and the mean value and variance of the echo frequency domain signal feature, determine the echo frequency domain signal feature; The feature of the far-end reference audio-frequency domain signal is determined according to the third function corresponding to the amplitude spectrum of the far-end reference audio-frequency domain signal, and the mean and variance of the feature of the far-end reference audio-frequency domain signal. 6. method according to claim 1, is characterized in that, after described training, cascade network is obtained by following steps training: receiving the first voice time domain signal containing echo and noise, the first remote reference acoustic time domain signal and the first target voice time domain signal; Perform framing, windowing and Fourier transform on the first voice time-domain signal containing echo and noise, the first far-end reference acoustic time-domain signal and the first target voice time-domain signal, respectively, to obtain a first voice domain signal containing echo and noise, a first remote reference voice domain signal and a first target voice domain signal; determining a first echo frequency domain signal according to the first voice frequency domain signal containing echo and noise and the first remote reference audio frequency domain signal; According to the first voice and audio domain signal containing echo and noise and the first echo frequency domain signal, determine a first voice and audio domain signal containing residual echo and noise; Perform energy normalization processing on the amplitude spectrum of the first voice-frequency domain signal containing residual echo and noise, the amplitude spectrum of the first echo frequency-domain signal, and the amplitude spectrum of the first far-end reference audio-frequency domain signal , obtain the first voice-frequency domain signal feature containing residual echo and noise, the first echo frequency-domain signal feature and the first remote reference audio-frequency domain signal feature; Splicing the first voice and audio domain signal features containing residual echo and noise with the first remote reference audio frequency domain signal features to obtain a first splicing feature, and splicing the first voice and audio signal features containing residual echo and noise. The audio domain signal feature is spliced with the first echo frequency domain signal feature to obtain a second splicing feature; The first splicing feature and the second splicing feature are input to the feature attention model in the cascade network, to jointly train the feature attention model and the residual echo and noise cancellation model in the cascade network, and the result is obtained with the first a first weight corresponding to a remote reference audio frequency domain signal feature and a second weight corresponding to the first echo frequency domain signal feature; The first remote reference audio frequency domain signal feature is multiplied by a first weight to obtain a first fusion feature, and the first echo frequency domain signal feature is multiplied by a second weight to obtain a second fusion feature; Splicing the first fusion feature, the second fusion feature, and the first voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing feature; Inputting the first fusion splicing feature into the residual echo and noise cancellation model in the cascaded network to obtain a masking estimate of the second target voice and audio domain signal; According to the masking estimate value of the second target voice domain signal and the first voice domain signal containing residual echo and noise, determine the second target voice domain signal; According to at least two loss functions, a multi-domain loss function is determined; wherein, the at least two loss functions include an energy-independent amplitude spectrum loss function and an objective speech quality assessment score loss function; the energy-independent amplitude spectrum loss function is The amplitude spectrum of the first target speech and audio domain signal is a training target, and is determined according to the second target speech and audio domain signal; the objective speech quality evaluation score loss function is determined by taking improving the audio quality of speech as a training target; The post-training cascade network is obtained by continuously reducing the multi-domain loss function iteratively with model parameters. 7. A residual echo and noise cancellation device, comprising: The receiving module is used to receive the voice time domain signal containing echo and noise and the far-end reference sound time domain signal; The processing module is used to perform framing, windowing and Fourier transform on the voice time-domain signal containing echo and noise and the far-end reference acoustic time-domain signal respectively, so as to obtain the voice and audio domain signal containing echo and noise and the far-end reference audio domain signal; a determining module, configured to determine an echo frequency domain signal according to the voice and audio frequency domain signal containing echo and noise and the far-end reference audio frequency domain signal; The determining module is further configured to determine a voice and audio domain signal containing residual echo and noise according to the voice and audio domain signal containing echo and noise and the echo frequency domain signal; An energy normalization module, configured to perform energy normalization on the amplitude spectrum of the voice and audio domain signal containing residual echo and noise, the amplitude spectrum of the echo frequency domain signal and the amplitude spectrum of the far-end reference audio frequency domain signal process to obtain the voice and audio domain signal features containing residual echo and noise, the echo frequency domain signal features and the far-end reference audio frequency domain signal features; The splicing module is used for splicing the voice and audio domain signal features containing residual echo and noise with the remote reference audio frequency domain signal features to obtain a first splicing result, and splicing the voice and audio containing residual echo and noise. The domain signal feature is spliced with the echo frequency domain signal feature to obtain a second splicing result; A weight obtaining module is used to input the first splicing result and the second splicing result into the post-training feature attention model in the post-training cascade network, and obtain the corresponding feature of the far-end reference audio frequency domain signal. a first attention weight and a second attention weight corresponding to the echo frequency domain signal feature; The fused attention mechanism feature acquisition module is used to multiply the far-end reference audio frequency domain signal feature with the first attention weight to obtain the first fused attention mechanism feature, and combine the echo frequency domain signal feature with the second The attention weights are multiplied to obtain the second fusion attention mechanism feature; The splicing module is further configured to splicing the first fusion attention mechanism feature, the second fusion attention mechanism feature, and the voice and audio domain signal features containing residual echo and noise to obtain a first fusion splicing result; a masking estimation value obtaining module, which is used to input the first fusion splicing result into the post-training residual echo and noise cancellation model in the post-training cascade network to obtain the masking estimation value of the target speech and audio domain signal; a target voice and audio domain signal obtaining module, configured to obtain the target voice and audio domain signal according to the masking estimate value of the target voice and audio domain signal and the voice and audio domain signal containing residual echo and noise; The inverse Fourier transform module is used for performing inverse Fourier transform on the target speech and audio domain signal to obtain the target speech time domain signal. 8. A residual echo and noise cancellation device, comprising at least one processor, the processor is configured to execute a program stored in a memory, and when the program is executed, the device is made to execute: The method of any one of claims 1-6. 9. A non-transitory computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the method according to any one of claims 1-6 is implemented.

Images