TWI749547B - Speech enhancement system based on deep learning - Google Patents

Abstract

A deep learning system includes a speech conversing module, a speech extracting module, a speech enhancement subsystem and a speech restoration module. The speech converting module receives and converts the first voice signal to be first voice spectrums and signal phases. The speech extracting module concatenates the first voice spectrums to be a second voice spectrum. The speech enhancement subsystem includes a speaker feature extracting model receiving the second voice spectrum to input in a first deep neuron network to acquire speaker feature code, and a speech enhancement network model receiving the speaker feature code and the second voice spectrum to input in a second deep neuron network to acquire a gain function. The gain function and the second voice spectrum are recovered to generate an enhanced speech signal spectrum. The speech restoration module combines the enhanced speech signal spectrum and the signal phases to output a second speech signal.

Description

Speech enhancement system using deep learning

The invention relates to a speech enhancement system, in particular to a speech enhancement system applying deep learning.

In the real environment, we often encounter the problem of voice communication under noise interference. For example, when using mobile phones on trains and MRT, the environmental noise signal will reduce the quality and intelligibility of the voice signal, thus reducing the communication between people and people. The communication efficiency and audio quality between people or people and machines. In order to solve this problem, some front-end voice signal processing technologies were developed accordingly, which can extract clean voice signals from noisy voice signals, reduce the noise components in the voice signal, thereby increasing the signal-to-noise ratio, and can increase The quality and comprehensibility of the speech signal, this processing method is called speech enhancement algorithm (speech enhancement). Speech enhancement algorithms can be divided into unsupervised and supervised speech enhancement algorithms, mainly based on frequency domain processing.

The unsupervised speech enhancement algorithm has the advantage of not needing to prepare tags for input data in advance. Currently widely used unsupervised speech enhancement methods such as methods based on short-term spectral restoration (spectral restoration), and its common algorithms such as spectral subtraction and Wiener filtering, are all based on spectral restoration. The method, and spectrum recovery is to estimate the gain function of the signal in the spectrum, and use the gain function to achieve speech enhancement. There are also some adaptive spectrum recovery methods. The method is to first use a noise tracking algorithm to find the noise spectrum in the speech signal, and then calculate the a priori SNR and post-noise ratio from the noise spectrum. A posteriori SNR (a posteriori SNR) algorithm, such as a minimum controlled recursive averaging (MCRA) algorithm, can then calculate the signal gain function, and use this gain function to achieve speech enhancement.

The supervised speech enhancement algorithm requires the preparation of "training data" to train the speech enhancement system in order to achieve speech enhancement. In recent years, most supervised speech enhancement algorithms are based on the construction of deep learning methods under artificial neural networks, because this method shows a strong advantage in regression analysis. Supervised speech enhancement algorithms, such as deep denoising auto-encoder (DDAE), propose a relationship model between clean signals and noisy signals, and use deep neural networks to achieve speech enhancement, which can effectively remove bands. Noise in the noise signal. In addition, the results show that the deep learning speech enhancement model under various noise conditions has good adaptability in unknown noise environments.

Therefore, it is known from the above that there is a need for how to design a speech enhancement system with good environmental adaptability in an unknown noise environment.

The technical problem to be solved by the present invention is to provide a speech enhancement system aiming at the deficiencies of the prior art, which can have good adaptability in an unknown noise environment.

In order to solve the above technical problems, one of the technical solutions adopted by the present invention is to provide a voice enhancement system using deep learning, which includes a voice conversion module, a voice capture module, a voice enhancement subsystem, and a voice enhancement system. Voice restoration module. The voice conversion module is used for receiving the first voice signal, and applying short-time Fourier transform to convert the first voice signal into multiple first voice spectra and multiple signal phases corresponding to multiple sound frames. The voice capture module is connected to the voice conversion module, and a plurality of first voice frequency spectra corresponding to a plurality of adjacent sound frames are concatenated to obtain a second voice frequency spectrum. The voice enhancement subsystem is connected to the voice capture module, and includes a speaker feature capture model and a voice enhancement network model. The speaker feature extraction model is connected to the voice capture module, and is configured to receive the second speech spectrum and input a first deep neural network to capture at least one speaker feature code of the second speech spectrum. The voice enhancement network model is connected to the voice capture module and the speaker feature capture model, and is configured to receive at least one speaker feature code and a second voice spectrum, and input a second deep neural network to pass the second depth The neural network estimates a gain function, and performs a spectrum recovery process on the gain function and the second voice spectrum to generate an enhanced voice signal spectrum. The voice restoration module is connected to the voice conversion module and the voice enhancement subsystem, receives the enhanced voice signal spectrum and multiple signal phases, combines the enhanced voice signal spectrum with multiple signal phases, and uses anti-short-time Fourier transform to output the enhanced A second voice signal.

One of the beneficial effects of the present invention is that in the speaker-perceived speech enhancement system provided by the present invention, the speaker-perceived speech enhancement system processes noisy speech, which has obvious noise reduction and speech quality and intelligibility improvement. And its performance is better than traditional unsupervised and supervised speech enhancement.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings about the present invention. However, the provided drawings are only for reference and description, and are not used to limit the present invention.

The following is a specific embodiment to illustrate the implementation of the "speech enhancement system using deep learning" disclosed in the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be based on different viewpoints and applications, and various modifications and changes can be made without departing from the concept of the present invention. In addition, the drawings of the present invention are merely schematic illustrations, and are not drawn according to actual size, and are stated in advance. The following embodiments will further describe the related technical content of the present invention in detail, but the disclosed content is not intended to limit the protection scope of the present invention. In addition, the term "or" used in this document may include any one or a combination of more of the associated listed items depending on the actual situation.

Voice activity detection is also called voice endpoint detection and voice boundary detection. The purpose of this technology is to detect the presence or absence of voice from noisy voice signals, and to locate the starting point and end point of the voice segment, which are based on voice The original processing of the data.

Voice activity detection has a wide range of applications, such as voice localization, speech coding, speech recognition, speaker identification, etc. Among the many voice activity detection algorithms, the present invention uses a statistical-based voice activity detection method. This method is to observe the change of the voice signal in the time domain frequency band and the flatness of the voice signal in the frequency domain. These methods can be used to determine whether the voice signal segment is voice or non-voice, and then the voice and non-voice can be separated.

Deep learning is a part of machine learning. It is a hierarchical machine learning method. It is usually regarded as a kind of "deep" neural network, and it is equipped with various neural network layers, such as scrolls. Convolutional Neural Networks (CNN), recurrent Neural Networks (RNN), etc.

Neural network is a kind of mathematical model to imitate the conduction in biological neuron (neuron) and nervous system. In a neural network, there are usually several layers, in each layer there are tens to hundreds of neurons, and the operation of each neuron is to add up the input of the neurons in the previous layer, and the output is It is to add an activation function to simulate the operation of nerve conduction. Among them, each neuron will be connected with the neuron of the next layer, so that the output value of the neuron of the previous layer is passed to the neuron of the next layer after weight calculation.

In general, the architecture of the similar neural network includes settings such as the number of layers, the number of neurons in each layer, the connection mode of neurons between each layer, and the type of activation function. These parameter settings need to be set manually before using the neural network. The quality of the parameter settings also greatly affects the performance of the neural network. The learning and training process of the neural network is to try to find The best weight setting. Among them, there are more than one hidden layer, and a neural network with multiple hidden layers is usually called a deep neural network.

Fig. 1 is a block diagram of a speech enhancement system applying deep learning of the present invention. As shown in FIG. 1, the speech enhancement system 10 applying deep learning of the present invention at least includes a speech conversion module 11, a speech capture module 12, a speech enhancement subsystem 13 and a speech restoration module 14.

與多個訊號相位

，

表示y中的第i個音框的頻譜強度。語音擷取模組12連接語音轉換模組11，將相鄰的多個音框對應的多個第一語音頻譜進行串接處理以獲得一第二語音頻譜

。語音增強子系統13連接語音擷取模組12，且包括語者特徵擷取模型131與語音增強網路模型132。語者特徵擷取模型131連接語音擷取模組12，經配置以接收第二語音頻譜

並輸入一第一深度神經網路133，以擷取第二語音頻譜

的至少一語者特徵編碼

。語音增強網路模型132連接語音擷取模組12及語者特徵擷取模型131，經配置以接收至少一語者特徵編碼

與第二語音頻譜

，並輸入一第二深度神經網路134，以通過第二深度神經網路134估計出一增益函數G _i，且將增益函數G _i與第二語音頻譜

進行一頻譜回復處理以產生一增強語音訊號頻譜

。語音還原模組連接語音轉換模組11及語音增強子系統13，接收增強語音訊號頻譜

及多個訊號相位

，並將增強語音訊號頻譜

與多個訊號相位

結合，並以反短時傅立葉轉換以輸出增強後的一第二語音訊號

。 The voice conversion module 11 is used to receive the first voice signal y, and apply Short-Time Fourier Transform (STFT) to convert the first voice signal y into multiple first voice frequency spectra corresponding to multiple sound frames

Phase with multiple signals

,

Represents the spectrum intensity of the i-th frame in y. The voice capture module 12 is connected to the voice conversion module 11, and a plurality of first voice frequency spectra corresponding to a plurality of adjacent sound frames are concatenated to obtain a second voice frequency spectrum

. The voice enhancement subsystem 13 is connected to the voice capture module 12 and includes a speaker feature capture model 131 and a voice enhancement network model 132. The speaker feature capture model 131 is connected to the voice capture module 12 and is configured to receive the second voice spectrum

And input a first deep neural network 133 to capture the second speech spectrum

At least one speaker feature code

. The voice enhancement network model 132 is connected to the voice capture module 12 and the speaker feature capture model 131, and is configured to receive at least one speaker feature code

With the second speech spectrum

, And input a second deep neural network 134 to estimate a gain function G _i through the second deep neural network 134, and combine the gain function G _i with the second speech frequency spectrum

Perform a spectrum recovery process to generate an enhanced voice signal spectrum

. The voice restoration module is connected to the voice conversion module 11 and the voice enhancement subsystem 13 to receive the enhanced voice signal spectrum

And multiple signal phases

And will enhance the voice signal spectrum

Phase with multiple signals

Combine and use inverse short-time Fourier transform to output an enhanced second voice signal

.

。 In the first embodiment, the speech enhancement subsystem 13, also referred to herein as the Speaker-Aware Denoising Neural Network (SaDNN), is mainly composed of two neural networks, each of which is speaker feature extraction Take the model (SpkFE) 131 and the embedded speaker characteristic speech enhancement network (SpE-DNN) 132, each has a neural network, which is different from the general speech enhancement system based on the deep neural network gain function estimation speech enhancement model The point is that the speaker-perceived speech enhancement model also integrates the speaker feature coding generated by the speaker feature extraction model

.

The short-time Fourier transform (STFT) is a time-frequency domain analysis method. The so-called short-time Fourier transform, as the name implies, is to perform Fourier transform on short-time signals. The short-term signal is a long-term signal multiplied by a short-term frame, or called a frame. In this way, a long signal is cut, multiplied by the window function, and Fourier transform is performed on each sound frame, and finally the result of each sound frame is stacked in one dimension (frame overlap), and the amplitude component is obtained to obtain a three-dimensional image. Signal spectrum graph.

First, define the first voice signal y as a voice signal with noise in the time domain, which is obtained by the sum of the clean and noise-free voice signal x and the noise signal v, so it can be expressed as a vector type:

，

,

After the above formula is Fourier transformed, the sound frame of the m-th voice signal with noise can be expressed as:

，

,

： Where k corresponds to the frequency

:

，

,

中，

與

分別為頻域中的乾淨訊號及噪聲訊號，L為音框的長度，k為頻譜中之頻率間隔(frequency bin)。 In the first speech spectrum

middle,

and

They are the clean signal and the noise signal in the frequency domain, L is the length of the sound frame, and k is the frequency bin in the spectrum.

的音功率譜密度(power spectrum density，PSD)，即可計算得到先驗訊雜比與後驗訊雜比。估計增益函數部份，根據先前計算出之先驗訊雜比與後驗訊雜比，即可估計出增益函數

。最後，增強後訊號

，是為

與

運算過後的結果。為了便於表示，後續的

、

、

及

都將以Y、S、V、G表示之。 The noise tracking part, its purpose is to calculate the noisy voice signal

The power spectrum density (PSD) can be calculated to obtain the prior signal-to-noise ratio and the posterior signal-to-noise ratio. In the part of the estimated gain function, the gain function can be estimated based on the a priori signal-to-noise ratio and the posterior signal-to-noise ratio calculated previously

. Finally, the enhanced signal

,it's for

and

The result after the operation. For ease of presentation, the subsequent

,

,

and

All will be represented by Y, S, V, G.

分解成振幅與(訊號)相位，可得： First speech spectrum by combining noise and clean speech

Decomposed into amplitude and (signal) phase, we can get:

，

,

，

,

與

皆為振福，

與

皆為相位，在本發明中訊號的振幅都將以下標a表示之。 in

and

All are Zhenfu,

and

Both are phases. In the present invention, the amplitude of the signal will be denoted by the subscript a.

In order to reconstruct the clean voice signal X from the signal Y with noise, in the present invention, the phase of the clean voice signal is estimated as follows:

，

,

的先驗密度通過均勻分佈得： in

The prior density of is obtained by uniform distribution:

， p(θ)=

,

In the interval (−π,π), according to the above equation, we can get:

；

；

Among them, the frequency spectrum of the clean voice signal can be expressed as follows:

，

,

The purpose of spectrum recovery is to estimate a gain function G.

進行頻域以及時域上的平滑處理，取一個窗函數做平均，因而增加了相鄰視窗的關聯性，其方程式如下： Because the frequency spectrum of the voice signal is obtained by Fourier transform, the error caused by the tracking of the noise signal is reduced, and the power spectrum of the voice signal is

Perform smoothing in the frequency domain and time domain, and take a window function for averaging, thus increasing the relevance of adjacent windows. The equation is as follows:

，

,

為加權係數，而窗函數的長度為

。在時域中，對音框的位置採用一階遞迴平滑處理，其方程式如下： in

Is the weighting factor, and the length of the window function is

. In the time domain, first-order recursive smoothing is used for the position of the sound frame, and the equation is as follows:

，

,

為語音訊號的平滑參數，

表示前一個所含噪聲音框的功率頻譜。接著追蹤平滑語音訊號功率頻譜中的最小值當作語音訊號存在的基準，同時估計出語音訊號存在機率，另外暫時儲存一個最小值做為下一段區間內的初始值： in

Is the smoothing parameter of the voice signal,

Represents the power spectrum of the previous noise frame. Then track the minimum value in the power spectrum of the smoothed speech signal as the basis for the existence of the speech signal, and estimate the probability of the existence of the speech signal at the same time, and temporarily store a minimum value as the initial value in the next interval:

，

,

，

,

Whenever L (m=0, 1, …, L−1) sound frames are read, the minimum value and the previously stored minimum value will be updated with the initial value. The equation is as follows, and the above equation is tracked. The minimum value in the power spectrum.

，

,

，

,

To determine whether there is voice in the mid-range of the voice signal, it can be judged by tracking the minimum value in the power spectrum of the smooth voice signal. The voice presence index is:

，

,

為1時表示語音訊號存在，

為0時表示語音訊號不存在。透過語音存在指標可以對觀測語音的語音存在機率進行追蹤： in

When it is 1, it means that the voice signal exists,

When it is 0, it means that the voice signal does not exist. The voice presence index can track the voice presence probability of the observed voice:

，

,

Based on the following two assumptions: speech and noise signals are random processes and the speech and noise signals are independent and uncorrelated, and are additive with each other, to derive the power spectral density (PSD) of the noise signal and Gain function G.

)與後驗訊雜比(

)兩參數均為統計而來的訊雜比，定義為： Priori signal-to-noise ratio (

) And the posterior signal-to-noise ratio (

) Both parameters are statistical signal-to-noise ratio, which is defined as:

,

，

,

,

=,

= E[

]分別為乾淨語音訊號X與噪聲訊號V的音功率譜密度。 in

=,

= E[

] Is the sound power spectral density of the clean speech signal X and the noise signal V, respectively.

Assuming that the clean speech signal and noise signal spectrum are constructed through Gaussian distribution, the conditional probability density function (PDF), p(Y|Xa,θ _X ) can be expressed as:

)，

),

)可以表示為： The amplitude and phase of a complex Gaussian random variable with zero mean are statistically independent, so p(

)It can be expressed as:

) = p(

) · p(

)， p(

) = p(

) · P(

),

)萊利分佈(rayleigh distribution)； Where p(

) Rayleigh distribution;

，

,

為萊利分佈機率密度中之超參數。 in

It is the hyperparameter in the probability density of the Riley distribution.

，可表示為： Maximum a posteriori spectral amplitude algorithm (Maximum A Posteriori Spectral Amplitude, MAPA) target estimated spectral amplitude

, Can be expressed as:

，

,

為MAPA的損失函數，可以表示為： in

Is the loss function of MAPA, which can be expressed as:

，

,

微分，並令其等於零，即可得到基於MAPA的增益函數： Then in the above equation

Differentiate, and make it equal to zero, you can get the gain function based on MAPA:

， G _MAPA =

,

為先驗訊雜比()與

後驗訊雜比。 in,

Is the prior signal-to-noise ratio () and

The posterior signal-to-noise ratio.

The speech signal spectrum after the enhancement of the maximum posterior spectrum amplitude algorithm is:

。

.

： The Frequency-Domain Wiener Filter (Frequency-Domain Wiener Filter) is a linear filter proposed by the mathematician Norbert Wiener (Norbert Wiener) with the minimum mean square error as the optimal criterion. Wiener filter is one of the most classic spectrum recovery methods, divided into time domain and frequency domain Wiener filters. The method of deriving the time-domain Wiener filter is the process of obtaining the reconstructed speech signal x[n] by using the estimation method. First, let a finite impulse response filter of length L be h = [h ₀ h ₁ …h _L−1 ] ^T , and then multiply the product of y[n] and this filter to obtain

:

，

,

為： After estimation, there is an error value between the reconstructed voice signal and the original voice signal. This error value

for:

，

,

And take its minimum mean square error (MMSE),

，

,

與

相比，

則需要包含更少的雜訊，故

的最佳濾波器h ₀即為時域維納濾波器，方程式如下： Let the best reconstructed voice signal be

and

compared to,

Needs to contain less noise, so

The best filter h _{0 is} the time domain Wiener filter, and the equation is as follows:

The above is part of the derivation of the time domain Wiener filter. They are all derived in the time domain. The following will convert to the frequency domain and continue to derive the frequency domain Wiener filter.

Repeating the derivation of the time-domain Wiener filtering in the frequency domain, and substituting H for H[k], we can get:

其中，

in,

，

,

After taking the partial differentiation of H in the above formula, make this formula equal to zero, and then the frequency domain Wiener filter can be calculated:

，

,

與

分別為X和Y的功率頻譜密度，因此，需計算出X和Y的功率頻譜密度方可建構出所需之濾波器，其中Y容易求得，再依據先前的假設，語音與噪聲訊號間獨立不相關，且彼此間具有可加性，即可計算出，如下式： in

and

They are the power spectral densities of X and Y respectively. Therefore, the required filter can be constructed by calculating the power spectral densities of X and Y. Y is easy to find. According to the previous assumptions, the speech and noise signals are independent of each other. Irrelevant and additivity to each other, it can be calculated as the following formula:

，

,

為雜訊V的PSD，再由上述的H ₀、

、

以及

的方程式即可推導出基於頻域維納濾波器的增益函數G _WIENER： in

Is the PSD of noise V, and then from the above H ₀ ,

,

as well as

The equation of can derive the gain function G _WIENER based on the frequency domain Wiener filter:

。

.

The purpose of the Deep Neural Network Based Speech Enhancement (DNN-SE) model is to use the neural network to enhance the signal y to reconstruct the clean signal x. The detailed system block diagram is shown in Figure 1.

進行串接處理以獲得一第二語音頻譜

，y經過短時傅立葉轉換得到第一語音頻譜

，

表示y中的第i個音框的頻譜強度，接著經過特徵擷取(feature extraction)後得到一個新的訊號

，其是由連接每個相鄰的音框並取對數成為一個連續的對數功率頻譜，可以表示為： Still referring to FIG. 1, the voice extraction module (feature extraction) 12 is connected to the voice conversion module 11, and multiple first voice spectra corresponding to multiple adjacent sound frames

Perform concatenation processing to obtain a second speech spectrum

, Y gets the first speech spectrum after short-time Fourier transform

,

Represents the spectral intensity of the i-th frame in y, and then a new signal is obtained after feature extraction

, Which consists of connecting each adjacent sound frame and taking the logarithm to become a continuous logarithmic power spectrum, which can be expressed as:

，                       方程式A，

, Equation A,

的長度為2I+1(在本發明中，假設I=5)。之後將每一個第二語音頻譜

經過基於深度神經網路的語音增強處理得到

，這個

是增強過後的對數功率頻譜，之後將對數功率頻譜經由頻譜回復(spectral restoration)轉換為原先的強度頻譜(magnitude spectrum)

，並與原始訊號y的訊號相位

合併得到新的頻譜

，最後

經過反短時傅立葉轉換得到增強過後的時域訊號

。 The symbol ";" means vertical connection, and the second speech frequency spectrum

The length of is 2I+1 (in the present invention, it is assumed that I=5). After that, each second speech spectrum

After speech enhancement processing based on deep neural network,

,this

It is the enhanced logarithmic power spectrum, and then the logarithmic power spectrum is converted to the original magnitude spectrum through spectral restoration.

, And the signal phase of the original signal y

Combine to get a new spectrum

,At last

After anti-short-time Fourier transform, the enhanced time-domain signal is obtained

.

。假設這裡的深度神經網路具有L層，其中任何一層l的輸入與輸出關係： The deep neural network block in Figure 1 is composed of deep neural networks and used to enhance the input signal

. Assume that the deep neural network here has L layers, and the input and output relationship of any layer l:

方程式B，

Equation B,

與

分別為任一 l層的啟動函數與線性回歸函數，而輸入層與輸出層分別對應到的第一層與第L層。因此，可以從深度神經網路方塊中可以得到

=

和

=

。 in

and

They are the startup function and linear regression function of any l layer, and the input layer and output layer correspond to the first layer and the Lth layer respectively. Therefore, you can get from the deep neural network block

=

and

=

.

所組成，訓練模型之參數方式是將模型的輸入設為第二語音頻譜

，計算並最小化模型的輸出

與目標輸出

間的損失函數，這裡使用回歸分析時常用的損失函數：均方誤差(mean square error)。 Regarding the training model phase, first prepare the training data set, which is composed of noisy speech signals-clean speech signals

The parameter method of training the model is to set the input of the model to the second speech frequency spectrum

, Calculate and minimize the output of the model

And target output

Loss function between time, here is the loss function commonly used in regression analysis: mean square error.

Deep Neural Network with Gain Estimation Based Speech Enhancement (DNN-Gain)

A deep neural network is used to calculate the prior signal-to-noise ratio and the posterior signal-to-noise ratio. The detailed structure is shown in Figure 2.

，可表示為： The deep neural network based on the deep neural network speech enhancement model (DNN-SE) replaces the previous unsupervised noise tracking method. The neural network predicts ξ and γ to calculate the gain function G and compare it with the original The logarithmic power spectrum Y of the input signal is calculated to obtain the enhanced voice signal spectrum

, Can be expressed as:

。

.

The average vector of speaker characteristics (d-vector),

表1 For example, in Table 1 as shown below, the input is the Mel spectrum of speech, and then the output of the last hidden layer is L2 regularization, and then the output of the entire speech in the deep neural network is obtained. An average vector (d-vector) was obtained. Tested in a quiet and noisy environment, the average vector (d-vector) speaker verification system can reduce the error rate by 14% and 25%, respectively.

Table 1

In order to improve the general speech enhancement system's processing capabilities among different speakers, the present invention is based on the deep neural network-based gain function estimation speech enhancement model (DNN-Gain) 2 to construct speech perceived by the speaker using deep learning Enhanced system (speaker-aware denoising neural network, SaDNN) 20, the system block diagram is shown in Figure 2. The voice enhancement system 20 applying deep learning in the second embodiment also includes at least a voice conversion module 21, a voice capture module 22, a voice enhancement subsystem 23, and a voice restoration module 24. Since the voice conversion module 11, the voice capture module 12, and the voice restoration module 14 of the second embodiment are the same as the voice conversion module 21, the voice capture module 22, and the voice restoration module 24 of the first embodiment, Therefore, the description of the voice conversion module 21, the voice capture module 22, and the voice restoration module 24 of the second embodiment will not be repeated here.

。 The speech enhancement subsystem 23 of the second embodiment also includes a speaker feature extraction model 231 and a speech enhancement network model 232. The speaker feature extraction module 231 of the second embodiment is the same as the speaker feature of the first embodiment. The feature extraction module 131 is the same, which respectively includes a first deep neural network 233 and a second deep neural network 234, and the speaker enhancement subsystem 23 of the second embodiment further includes a noise feature extraction model 235 And an (embedded) speech enhancement model 236. The noise feature extraction model 235 is connected to the voice capture module 22 and the voice enhancement network model 232. The noise feature extraction model 235 receives the second speech spectrum to perform a noise feature extraction process through a third deep neural network 237 Generate at least one noise feature code

.

及至少一噪聲特徵編碼

以輸入一第四深度神經網路238，以通過第四深度神經網路238估計增益函數G。 The embedded voice enhancement model 236 is connected to the voice capture module 22, the voice enhancement network model 232, and the noise feature capture model 235, and receives the second voice spectrum and at least one speaker feature code

And at least one noise feature code

A fourth deep neural network 238 is input to estimate the gain function G through the fourth deep neural network 238.

The speaker-perceived speech enhancement system of the present invention effectively integrates the characteristics of different speakers, and the research results show that the evaluation index after speech enhancement is better than the speech enhancement system based on the deep neural network gain function estimation speech enhancement model. Further considering that there are several kinds of noise interference (non-human voice signals) in the environment where the speaker is located, based on the speech enhancement model based on the gain function estimation of the deep neural network, an environment with both speaker characteristics and noise characteristics is derived Perceptual speech enhancement system (speaker and speaking environment-aware denoising neural network, SEADNN). The environment-aware speech enhancement system combines a speech enhancement system based on a deep neural network, the personal characteristics of a specific speaker, and the noise characteristics of the speaker's environment.

模型之環境感知的語音增強系統20的輸入是由連接每個相鄰的音框並取對數成為一個連續的對數特徵頻譜(context feature)所組成，可表示如上述的方程式A。在本發明的第二實施例中，基於深度神經網路之語音增強系統20特別的地方是，是由三個深度神經網路(deep neural networks，DNNs)模型所組成，分別為語者特徵擷取模型(SpkFE module)231、環境的噪聲特徵擷取模型(NoeFE module)235以及嵌入式語音增強模型(例如如嵌入式環境整合語音增強神經網路模型(EE-DNN module)等)236。 Similar to the speech enhancement system 10 applying deep learning mentioned in the first embodiment of the previous chapter, in the second implementation,

The input of the speech enhancement system 20 for environment perception of the model is composed of connecting each adjacent sound frame and taking the logarithm to form a continuous logarithmic feature spectrum (context feature), which can be expressed as the above-mentioned equation A. In the second embodiment of the present invention, the special feature of the deep neural network-based speech enhancement system 20 is that it is composed of three deep neural networks (DNNs) models, each of which is speaker feature extraction The acquisition model (SpkFE module) 231, the environmental noise feature extraction model (NoeFE module) 235, and the embedded speech enhancement model (for example, the embedded environment integrated speech enhancement neural network model (EE-DNN module), etc.) 236.

和語者特徵擷取模型產生的語者特徵編碼

的深度神經網路。帶噪聲語音之特徵

放置在輸入層(第1層)，語者特徵編碼

將與特定層(第l層)之輸出連接，因此輸入的特徵到下一層隱藏層30(第l+1層)可以表示為

。因此本發明的嵌入式語者特徵語音增強網路132或232相似於傳統的基於深度神經網路之增益函數估計語音增強網路模型，不同之處在於(嵌入式語者特徵)語音增強網路132或232在特定隱藏層加入了語者特徵。 Figure 3 is a schematic diagram of the (embedded speaker feature) speech enhancement network model. As shown in Figure 3, the (embedded speaker feature) speech enhancement network model 132 or 232 is a feature that combines noisy speech

Speaker feature coding generated by the speaker feature extraction model

Deep neural network. Features of noisy speech

Placed in the input layer (layer 1), speaker feature coding

Will be connected to the output of a specific layer (layer l), so the input features to the next hidden layer 30 (layer l+1) can be expressed as

. Therefore, the embedded speaker feature speech enhancement network 132 or 232 of the present invention is similar to the traditional deep neural network-based gain function estimation speech enhancement network model, and the difference lies in the (embedded speaker feature) speech enhancement network 132 or 232 adds speaker characteristics to a specific hidden layer.

}、乾淨語音之特徵{

}以及語者特徵擷取模型產生之語者特徵{

}組建而成。訓練以輸入{

}和{

}並產生輸出為{

}之增益函數輸出。訓練方式如最小化損失函數之誤差等方式。 Before training (embedded speaker features) speech enhancement network 132 or 232, the training data set is composed of noisy speech features {

}, the characteristics of clean voice {

}And the speaker features generated by the speaker feature extraction model{

} Formed. Train to enter {

}and{

} And produce the output as {

} The gain function output. Training methods such as minimizing the error of the loss function, etc.

為語者特徵，p(spk _ij)為語者類別。

表2 In order to extract speaker characteristics from the speaker’s information, the speech enhancement system proposed by the present invention includes a speaker feature extraction model (SpkFE) 131 or 231, as shown in Table 2 below, where

It is the characteristic of the speaker, and p(spk _ij ) is the category of the speaker.

Table 2

，作法為將如下所述。語音訊號

中的每個音框的特徵分類為該語者的類別p(spk _j)，因此語者的數量N決定了深度神經網路的輸出大小與維度，另外考慮到訓練資料集中的非語音之音框類別，因此深度神經網路的輸出類別有(N+1)種，故j=1, 2, ..., N+1。另外類別p(spk _j)為語音訊號

經過語音活性檢測(Voice Activity Detection，VAD)處理後得到的語者類別，將作為語者特徵擷取模型中深度神經網路的期望輸出。類別p(spk _j)為一有效編碼(One-Hot編碼)後的(N+1)維向量，其中非零元素對應到相應的語者。 The main purpose of constructing the speaker feature extraction model (SpkFE) 131 or 231 is to extract the speaker feature code in the speech signal

, The practice will be as follows. Voice signal

The feature of each sound box in is classified as the speaker's category p(spk _j ), so the number of speakers N determines the output size and dimension of the deep neural network, and also takes into account the non-speech sounds in the training data set Box category, so there are (N+1) output categories of deep neural networks, so j=1, 2, ..., N+1. In addition, the category p (spk _j ) is a voice signal

The speaker category obtained after voice activity detection (Voice Activity Detection, VAD) processing will be used as the expected output of the deep neural network in the speaker feature extraction model. The category p(spk _j ) is an (N+1)-dimensional vector after an effective encoding (One-Hot encoding), in which non-zero elements correspond to the corresponding speakers.

，表示輸入

中各音框之向量的語者特徵，因此語者特徵擷取模型有兩個輸出，而取得的

將被送入嵌入式環境整合語音增強神經網路模型132中進一步處理。值得注意的是，選取語者特徵編碼

的方式，對於那些未知的語者而言，語者特徵編碼

具有比輸出層的輸出更能夠表示語者資訊的概括能力，同時這個做法在初步的研究中提供了噪聲特徵的環境感知語音增強系統更好的語音增強性能。 Figure 4 shows the architecture of the first deep neural network in the speaker feature extraction model 131. The input and output relationship between each hidden layer 40 in the network can be expressed as Equation B above, and the activation function setting of the output layer It is a normalized exponent (softmax) function, and the activation function of the input layer and all hidden layers is set to a rectified linear unit (ReLU). When the deep neural network training is completed, select the output of the last hidden layer (that is, the second-to-last layer), and define this output as the speaker feature code

For input

The speaker feature of the vector in each frame, so the speaker feature extraction model has two outputs, and the obtained

It will be sent to the embedded environment integrated speech enhancement neural network model 132 for further processing. It’s worth noting that the speaker feature code is selected

Way, for those unknown speakers, the speaker feature encoding

It has the ability to summarize speaker information better than the output of the output layer. At the same time, this approach provides better speech enhancement performance of the environment-aware speech enhancement system with noise characteristics in the preliminary research.

，其示意下表3所示。

表3 The noise feature extraction model based on deep neural network, the purpose is to extract the noise feature code in the environment from the environment where the speaker is located

, Its schematic is shown in Table 3 below.

table 3

中的每個噪聲特徵分類為該噪聲的類別，因此噪聲的數量M決定了噪聲特徵擷取模型的輸出大小與維度，故k = 1, 2, ..., M。另外p(noe ^k)為語音訊號

中的噪聲類別，將作為噪聲特徵擷取模型的期望輸出；p(noek)為One-Hot編碼後的M維向量，其中非零元素對應到相應的噪聲。 The noise feature extraction model converts the speech signal

Each noise feature in is classified as the category of the noise, so the amount of noise M determines the output size and dimension of the noise feature extraction model, so k = 1, 2, ..., M. In addition, p(noe ^k ) is the voice signal

The noise category in will be used as the expected output of the noise feature extraction model; p(noek) is the M-dimensional vector after One-Hot encoding, and the non-zero elements correspond to the corresponding noise.

，表示輸入的語音訊號

中各音框之向量的語者特徵，之後噪聲特徵編碼

與

都將被送入嵌入式環境整合語音增強神經網路模型(嵌入式語音增強模型236)中進一步處理。 5 is a schematic diagram of the third deep neural network, showing the structure of the noise feature extraction model 235, the network structure has a third deep neural network 237, in which the input and output relationship between each hidden layer 50 It can be expressed as Equation B as shown above, in which the activation function of the output layer is set as a normalized exponent (softmax) function, and the activation functions of the input layer and all hidden layers are set as a rectified linear unit (ReLU). When the deep neural network training is completed, select the output of the last hidden layer (that is, the second-to-last layer), and define this output as a noise feature code

, Which means the input voice signal

The speaker characteristics of the vector of each frame in the middle, and then the noise characteristics are coded

and

All will be sent to the embedded environment integrated speech enhancement neural network model (embedded speech enhancement model 236) for further processing.

的這個方式，對於那些未知的噪聲而言，噪聲特徵編碼

具有比最後的輸出層之輸出就有更能夠表示未知噪聲特徵的概括能力，同時這個做法在初步的實驗中提供了噪聲特徵的環境感知語音增強系統更好的語音增強性能。 Comparing the noise feature extraction model 235 and the speaker feature extraction model 231, the purpose is to extract the characteristics of specific speech signals to improve the system performance of the embedded environment integrated speech enhancement neural network model (embedded speech enhancement model 236) , Select noise feature code

In this way, for those unknown noises, the noise feature encoding

Compared with the output of the final output layer, it has the generalization ability that can express unknown noise characteristics. At the same time, this approach provides a better speech enhancement performance of the environment-aware speech enhancement system with noise characteristics in the preliminary experiment.

In the embedded environment integrated speech enhancement neural network (Environment Embedded Denoising Neural Network, EE-DNN) (embedded speech enhancement model 236), based on the gain function of the deep neural network to estimate the speech enhancement model system using the characteristics of noisy speech The frequency spectrum is used as input, and the embedded environment integrated speech enhancement neural network 236 additionally adds the speaker and noise features generated by the speaker feature extraction model and the noise feature extraction model. These two features are collectively called environmental features, embedded The architecture of the integrated speech enhancement neural network 236 in the integrated environment is shown in FIG. 6.

、語者特徵

以及噪聲特徵編碼

。帶噪語音之特徵

放置在輸入層，而噪聲特徵編碼

和語者特徵

將分別與特定層(第l1與第l2層)之輸出連接，因此輸入的特徵到下一層隱藏層60(第l1+1)與(第l2+1)層可以表示為

與

。 Fig. 6 is a schematic diagram of the fourth deep neural network. As shown in Fig. 6, the input of the embedded environment integrated speech enhancement neural network 236 contains the characteristics of noisy speech

, Speaker characteristics

And noise feature coding

. Features of noisy speech

Placed in the input layer, and the noise feature encoding

Speaker characteristics

Connect to the output of a specific layer (the 11th and 12th layers) respectively, so the input features to the next hidden layer 60 (the l1+1) and (l2+1) layers can be expressed as

and

.

}、相關的乾淨語音之特徵{

}以及Sp語者特徵擷取模型kFE產生之特徵{

}與噪聲特徵擷取模型產生之特徵{

}建構而成。訓練嵌入式環境整合語音增強神經網路模型是以輸入{

}、{

}以及{

}並產生輸出為{

}之增益函數輸出。 Therefore, the embedded environment integrated speech enhancement neural network model is similar to the traditional DNN-Gain network. The difference is that the embedded environment integrated speech enhancement neural network model adds speaker characteristics and noise characteristics to a specific hidden layer. The training data set of the embedded environment integrated speech enhancement neural network model is composed of the characteristics of noisy speech {

}, the characteristics of the related clean voice {

}And the features generated by the Sp speaker feature extraction model kFE{

}And the features generated by the noise feature extraction model{

} Constructed. Training the embedded environment to integrate the speech enhancement neural network model is based on the input {

}, {

}as well as{

} And produce the output as {

} The gain function output.

Fig. 7A is a distribution diagram of speaker feature codes, and Fig. 7B is a distribution diagram of speaker noise. First, the training data is divided into a noisy data set (Noisy for short) and a clean data set (Clean for short). There are 24 different speakers (24 × 8 = 192 audio files) in the noisy data set. And mix it into 4 kinds of noise with 3 different signal-to-noise ratios, so there are 2304 audio files (192×3×4 = 2304), and there are 24 different speakers (24 × 8 = 192) in the clean data set. Audio files), and the test data is a collection of noisy data sets and clean data sets (2304 + 192 = 2496 audio files). Then train the SpkFE model with the noisy data set and the clean data set respectively, and the two models obtained are referred to as the model of the noisy data set (Noisy model) and the model of the clean data set (Clean model), and finally used The test data tests two models, and obtains two speaker characteristic codes.

The speaker features as shown in FIG. 7B cannot effectively distinguish different noise ratios (SNR), because these feature points are often interleaved with each other. However, some trends can still be observed from the figure. Under high-noise ratio (SNR) conditions, its distribution is biased toward the periphery, and conversely, under low-noise ratio (SNR) ratios, it is biased toward the center. This result indicates that the speaker's model is more susceptible to the noise ratio (SNR), compared to the noise type.

Fig. 7C is a histogram of the original noisy speech and the speech enhancement system using deep learning of the present invention. As shown in FIG. 7C, these results show the effectiveness of the speaker feature extraction model and the noise feature extraction model in the first or second embodiment of the present invention, and therefore present changes in speech for the entire speech enhancement process. And the robustness of environmental noise changes.

The present invention proposes a novel speaker-perceived speech enhancement system, which reduces the distortion in the speech under the influence of different speakers and different environments, thereby increasing the quality of the speech signal. The speaker-perceived speech enhancement system is composed of three deep neural networks (DNN). The first one obtains the characteristics of each speaker, the second one obtains the characteristics of the environmental noise in the environment of the speaker at the time, and the third This uses the voice signal features captured by the first and second DNNs to restore the noisy voice to a clean voice close to the original. In particular, the speaker-perceived speech enhancement system proved to be effective and well enhanced the unknown speakers and the utterances generated in the unknown environment.

[Beneficial effects of the embodiment]

One of the beneficial effects of the present invention is that in the speaker-perceived speech enhancement system provided by the present invention, the speaker-perceived speech enhancement system processes noisy speech, which has obvious noise reduction and speech quality and intelligibility improvement. And its performance is better than traditional unsupervised and supervised speech enhancement.

The content disclosed above is only a preferred and feasible embodiment of the present invention, and does not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the description and schematic content of the present invention are included in the application of the present invention. Within the scope of the patent.

:第一語音頻譜

:訊號相位

:第二語音頻譜

:語者特徵編碼

:語者特徵編碼

:增益函數

:增益函數

:噪聲特徵編碼

:噪聲特徵編碼

:增強語音訊號頻譜

:第二語音訊號 20:語音增強系統 21:語音轉換模組 22:語音擷取模組 23:語音增強子系統 231:語者特徵擷取模型 232:語音增強網路模型 233:第一深度神經網路 234:第二深度神經網路 235: 噪聲特徵擷取模型 236: 嵌入式語音增強模型 237:第三深度神經網路 238:四深度神經網路 24:語音還原模組 30:隱藏層 40:隱藏層 50:隱藏層 60:隱藏層 PESQ:語音品質 10: Speech enhancement system 11: Speech conversion module 12: Speech capture module 13: Speech enhancement subsystem 131: Speaker feature extraction model 132: Speech enhancement network model 133: First deep neural network 134: No. Two deep neural network 14: voice restoration module y: the first voice signal

: The first speech spectrum

: Signal phase

: The second speech spectrum

: Speaker feature coding

: Speaker feature coding

: Gain function

: Gain function

: Noise feature coding

: Noise feature coding

: Enhanced voice signal spectrum

: Second voice signal 20: voice enhancement system 21: voice conversion module 22: voice capture module 23: voice enhancement subsystem 231: speaker feature extraction model 232: voice enhancement network model 233: first deep nerve Network 234: Second deep neural network 235: Noise feature extraction model 236: Embedded speech enhancement model 237: Third deep neural network 238: Four-deep neural network 24: Speech restoration module 30: Hidden layer 40 : Hidden layer 50: Hidden layer 60: Hidden layer PESQ: Voice quality

FIG. 1 is a block diagram of a speech enhancement system applying deep learning according to the first embodiment of the present invention.

Fig. 2 is a block diagram of a speech enhancement system applying deep learning according to a second embodiment of the present invention.

Figure 3 is a schematic diagram of a voice enhancement network model.

FIG. 4 is a schematic diagram of the first deep neural network architecture in the speaker feature extraction model.

Figure 5 is a schematic diagram of the third deep neural network architecture.

Fig. 6 is a schematic diagram of the fourth deep neural network architecture.

Figure 7A is a distribution diagram of speaker feature codes.

Figure 7B shows the speaker noise distribution diagram.

Fig. 7C is a histogram of the original noisy speech and the speech enhancement system using deep learning of the present invention.

:第一語音頻譜

:訊號相位

:第二語音頻譜

:語者特徵編碼

:增強語音訊號頻譜

:第二語音訊號 G _i:增益函數 10: Speech enhancement system 11: Speech conversion module 12: Speech capture module 13: Speech enhancement subsystem 131: Speaker feature extraction model 132: Speech enhancement network model 133: First deep neural network 134: No. Two deep neural network 14: voice restoration module y: the first voice signal

: The first speech spectrum

: Signal phase

: The second speech spectrum

: Speaker feature coding

: Enhanced voice signal spectrum

: The second voice signal G _i : gain function

Claims (7)

A speech enhancement system using deep learning includes: a speech conversion module for receiving a first speech signal, and applying short-time Fourier transform to convert the first speech signal into a plurality of first speech signals corresponding to a plurality of sound frames A voice frequency spectrum and a plurality of signal phases; a voice capture module connected to the voice conversion module, and a plurality of the first voice frequency spectrum corresponding to a plurality of adjacent sound frames are concatenated to obtain A second voice frequency spectrum; a voice enhancement subsystem, connected to the voice capture module, and includes: a speaker feature capture model, connected to the voice capture module, configured to receive the second voice Frequency spectrum and input a first deep neural network to extract at least one speaker feature code of the second speech spectrum; and a voice enhancement network model to connect the voice capture module and the speaker feature An extraction model, configured to receive the at least one speaker feature code and the second speech frequency spectrum, and input a second deep neural network to estimate a gain function through the second deep neural network, And perform a spectrum recovery process on the gain function and the second voice spectrum to generate an enhanced voice signal spectrum; and a voice restoration module connected to the voice conversion module and the voice enhancement subsystem to receive the The enhanced voice signal spectrum and the plurality of signal phases, and the enhanced voice signal spectrum and the plurality of signal phases are combined, and an inverse short-time Fourier transform is used to output an enhanced second voice signal; wherein, The voice enhancement network model further includes: a noise feature extraction model connecting the voice capture module and the voice enhancement network model, and the noise feature extraction model receives the second The speech frequency spectrum is processed by performing a noise feature extraction process through a third deep neural network to generate at least one noise feature code; an embedded speech enhancement model connected to the speech capture module, the speech enhancement network model, and The noise feature extraction model receives the second speech frequency spectrum, the at least one speaker feature code, and the at least one noise feature code to input a fourth deep neural network to pass the fourth deep neural network The network estimates the gain function; wherein the noise feature extraction model classifies the noise features in the second speech spectrum into noise categories, and determines the number of hidden layers of the deep neural network according to the number of noises, And output the noise feature code in the last hidden layer; wherein, the gain function is to apply Maximum A Posteriori Spectral Amplitude (MAPA) or Wiener Filter (Frequency-Domain Wiener Filter). ) To calculate the gain function based on the maximum a posteriori spectral amplitude algorithm or the Wiener filter. The speech enhancement system applying deep learning according to claim 1, wherein the speaker feature extraction model encodes at least one speech feature extracted from the second speech frequency spectrum into multiple speaker categories, and The number of hidden layers of the deep neural network is determined according to the number of speakers, and the speaker feature code is output in the last hidden layer. The speech enhancement system applying deep learning according to claim 2, wherein, in the speaker feature extraction model, the input and output relationship of the deep neural network is: z ^{( l )} = σ ^{( l )} ( h ^{( l )} ( z ^{( l -1)} )), l =1,..., L , where σ ^{( l )} (.) and h ^{( l )} (.) are any of the hidden layers l The activation function and linear regression function of the input layer and the output layer respectively correspond to the first hidden layer and the Lth hidden layer, the second speech spectrum is set in the input layer, and at least one The speaker feature encoding is set in any one of the hidden layers other than the input layer and the output layer. The speech enhancement system applying deep learning according to claim 2, wherein, in the noise feature extraction model, the input and output relationship of the deep neural network is: z ^{( l )} = σ ^{( l )} ( h ^{( l )} ( z ^{( l -1)} )), l =1,..., L , where σ ^{( l )} (.) and h ^{( l )} (.) are the values of any one of the hidden layers l The activation function and the linear regression function, and the input layer and the output layer respectively correspond to the first hidden layer and the Lth hidden layer, and the activation function of the output layer is set to a normalized index (softmax) Function, and the activation function of the input layer and all the hidden layers is set to a rectified linear unit (ReLU), and the second speech spectrum is set in the input layer. The voice enhancement system using deep learning as described in claim 1, further comprising a spectrum recovery module for combining the second voice spectrum represented by the logarithmic power spectrum and the signal phase of the first voice signal to obtain After the second speech frequency spectrum represented by the new frequency spectrum is converted to the second speech frequency spectrum represented by the new frequency spectrum by inverse short-time Fourier transformation, an enhanced second speech signal is output. The speech enhancement system applying deep learning according to claim 1, wherein the first deep neural network, the second deep neural network, the third deep neural network, and the fourth deep neural network The network is a convolutional neural network or a recurrent neural network. The speech enhancement system applying deep learning according to claim 1, wherein the second speech spectrum is formed by connecting each adjacent sound frame and taking the logarithm to form a continuous logarithmic power spectrum.

Images