CN110751957B - A speech enhancement method using stacked multi-scale modules - Google Patents

Abstract

The present invention discloses an end-to-end speech enhancement method using stacked multi-scale modules, comprising the following steps: S1: constructing a cascaded end-to-end speech enhancement framework, and splicing the stacked multi-scale modules into a network structure; S2: in the preprocessing stage, transforming the time domain signal into a two-dimensional feature; S3: enhancing the two-dimensional feature using a speech enhancement module; S4: in the post-processing stage, transforming the enhanced feature representation into a one-dimensional time domain signal through decoding synthesis. In order to further improve the performance of the algorithm, the evaluation indicators STOI and SDR of speech enhancement are integrated into the loss function using a multi-objective joint optimization training strategy. Experiments show that the method proposed in the present invention can significantly improve the speech enhancement effect, and has good noise resistance under unknown noise and low signal-to-noise ratio conditions.

Description

A speech enhancement method using stacked multi-scale modules

technical field

The invention belongs to the technical field of speech enhancement, and in particular relates to an end-to-end speech enhancement method using stacked multi-scale modules.

Background technique

Speech enhancement refers to the task of removing or attenuating additional noise in noisy speech, improving the overall perceptual quality of speech and speech intelligibility by suppressing and separating noise, in robust speech recognition, hearing aid design, speaker verification, etc. Has a wide range of applications. Traditional speech enhancement methods include spectral subtraction, Wiener filtering, statistical model-based methods, and subspace-based methods. In the past few years, deep learning-based supervised speech enhancement methods have gradually become the main focus of scholars. research direction.

Some scholars consider processing the time domain signal of speech directly instead of relying on the frequency domain representation of the speech signal, avoiding the switch back and forth between the time domain and the frequency domain of the speech signal, and making full use of the time domain feature representation of speech. Based on the WaveNet framework, Qian et al. proposed the method of introducing speech prior distribution for speech enhancement, and Rethage et al. predicted the target through acausal dilated convolution. Pascual et al. proposed SEGAN, which used convolutional networks to directly enhance speech in the time domain. Fu et al. proposed a fully convolutional neural network for whole sentence speech enhancement in time domain. In order to solve real-time speech enhancement, Pandey et al. A modular network is combined with a codec architecture to process time-domain signals.

These end-to-end methods directly map the one-dimensional time-domain waveform to the target speech. However, the time-domain waveform signal itself cannot show obvious characteristic structure, and it is difficult to directly model the time-domain signal. In a low signal-to-noise ratio environment The next modeling difficulty will be further increased.

SUMMARY OF THE INVENTION

The present invention provides an end-to-end speech enhancement method using stacked multi-scale modules, aiming to solve the above-mentioned problems.

The present invention is achieved in this way, a speech enhancement method using stacked multi-scale modules, comprising the following steps:

S1: Build a cascaded end-to-end speech enhancement framework and stitch stacked multi-scale modules into the network structure;

S2: In the preprocessing stage, transform the time domain signal into two-dimensional features;

S3: Use the speech enhancement module to enhance the two-dimensional features;

S4: In the post-processing stage, the enhanced feature representation is transformed into a one-dimensional time domain signal by decoding synthesis.

Further, the cascaded end-to-end speech enhancement architecture includes speech time-domain signal preprocessing, speech enhancement module and post-processing of target speech synthesis; specific steps include:

a. In the time-domain signal preprocessing stage, one-dimensional convolution is used to perform the convolution operation on the input speech segment, and the results of each convolution check on the noisy speech y are stacked row by row to form a two-dimensional The real-valued feature Y is inspired by the processing method of the convolutional neural network on the pixel value of the image, and the two-dimensional feature is separated to obtain the absolute value feature and the sgn mask;

将其与sgn mask相乘合成目标语音的特征表示：b. The absolute value feature of the noisy speech y is input to the speech enhancement module for enhancement, and the estimation of the absolute value feature is obtained

Multiply it with the sgn mask to synthesize the feature representation of the target speech:

变换为时域信号

c. After transposed convolution, the

Transform to time domain signal

Further, the multi-scale module includes an average pooling layer, convolution kernels with convolution kernels of 1×1 and 3×3, and dilated convolutions with different dilation rates.

Further, the training strategy of multi-objective joint optimization is used to integrate the speech enhancement evaluation indicators STOI and SDR into the loss function.

Further, the specific steps to incorporate the STOI indicator into the loss function include:

首先去除对语音可懂度无贡献的无声区域，然后用STFT将时域信号变换到时频域，通过将两个信号分割为50％重叠的带汉宁窗的帧；1) STOI input is pure speech X and degenerate speech

First remove the silent regions that do not contribute to speech intelligibility, then use STFT to transform the time-domain signal to the time-frequency domain by splitting the two signals into 50% overlapping frames with a Hanning window;

2) Carry out 1/3 octave frequency band analysis, divide a total of 15 1/3 octave frequency bands, in which the center frequency range of the frequency band is 4.3kHz to 150Hz, and the short-time envelope x _{j, m} of pure speech is expressed as follows:

[X _j (m-L+1), X _j (m-L+2), ...X _j (m)] ^T

where X∈R is the 1/3 octave band obtained by x, M is the total number of frames of a speech, m is the index of the frame, j is the index of the 1/3 octave band, and L corresponds to the length of the speech;

可懂度表示为两个时间包络之间的相关系数：3) Normalize and crop the speech to obtain the envelope representation of the degraded speech

Intelligibility is expressed as the correlation coefficient between two temporal envelopes:

Among them, ||·|| ₂ is the L2 norm, and μ(·) is the mean vector of the corresponding sample.

4) Calculate the average of the intelligibility of all bands and frames, and the STOI calculation indicator can be obtained:

带入到STOI计算公式中，即可得到训练过程中的STOI计算指标：5) Will enhance the voice

Bring it into the STOI calculation formula, and you can get the STOI calculation indicator during the training process:

Among them, d _{j, m} represents the correlation coefficient between the enhanced speech and the pure speech temporal envelope.

Further, the specific steps for incorporating the SDR indicator into the loss function include:

增强语音的SDR计算过程如下：1) The input of SDR is pure voice x and enhanced voice

The SDR calculation process for enhanced speech is as follows:

2) Equivalently transform the SDR optimization objective to simplify the calculation to obtain:

Among them, the process of maximizing the evaluation index SDR is equivalent to minimizing

Further, the specific steps of integrating the STOI and SDR evaluation indicators into the loss function include:

1) Calculate the conventional root mean square error, the process is as follows:

Among them, M and N are the number of sampling points of each voice and the total number of voices.

2) Combine the root mean square error with the loss function of the evaluation index based on STOI and SDR:

In the formula, α, β, γ correspond to the coefficients of different parts in the loss function.

where X∈R is the 1/3 octave band obtained by x, M is the total number of frames of a speech, m is the index of the frame, and j∈{1, 2,...15} is the index of the 1/3 octave band , L=30 corresponds to the length of the analyzed speech being 384ms.

Compared with the prior art, the beneficial effect of the present invention is that in order to improve the direct processing capability of the neural network for the time domain speech signal, the present invention proposes a new multi-scale end-to-end speech enhancement framework. In the preprocessing stage, the time domain signal is transformed into a two-dimensional feature representation, then the two-dimensional feature is enhanced by the speech enhancement module, and finally the enhanced feature representation is transformed into a one-dimensional time domain signal through decoding and synthesis. In order to further improve the performance of the algorithm, the training strategy of multi-objective joint optimization is used to integrate the evaluation indicators STOI and SDR of speech enhancement into the loss function. Experiments show that the method proposed in the present invention can significantly improve the effect of speech enhancement, and has better noise immunity under unknown noise and low signal-to-noise ratio conditions.

Description of drawings

Fig. 1 is the overall schematic diagram of the present invention;

2 is a schematic diagram of stacking multi-scale modules in the present invention;

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

In the description of the present invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", The orientations or positional relationships indicated by "horizontal", "top", "bottom", "inside", "outside", etc. are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than An indication or implication that the referred device or element must have a particular orientation, be constructed and operate in a particular orientation, is not to be construed as a limitation of the invention. In addition, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

Example

Referring to Figures 1-2, the present invention provides a technical solution: an end-to-end speech enhancement method using stacked multi-scale modules, comprising the following steps:

S1: Build a cascaded end-to-end speech enhancement framework and stitch stacked multi-scale modules into the network structure;

S2: In the preprocessing stage, transform the time domain signal into two-dimensional features;

S3: Use the speech enhancement module to enhance the two-dimensional features;

S4: In the post-processing stage, the enhanced feature representation is transformed into a one-dimensional time domain signal by decoding synthesis.

The end-to-end speech enhancement framework proposed by the present invention includes speech time-domain signal preprocessing, speech enhancement module and post-processing of target speech synthesis, as shown in FIG. 1 .

Assuming that the time domain clean speech is x and the noise signal is n, then the noisy speech y can be expressed as:

y=x+n

In the time-domain signal preprocessing stage, one-dimensional convolution is used to perform a convolution operation on the input speech segment, and the results of each convolution kernel on the noisy speech y are stacked row by row to form a two-dimensional real value. The feature Y is inspired by the way the convolutional neural network processes the pixel value of the image, and the two-dimensional feature is separated to obtain the absolute value feature and the sgn mask, where sgn represents the sign function (sign function), that is, the symbol of Y is taken, the two-dimensional feature Y is expressed as the product of absolute value features and sgnmask:

Y=abs(Y)⊙sgn(Y)

将其与sgn mask相乘合成目标语音的特征表示：where ⊙ represents the multiplication of the corresponding elements, and then the absolute value feature of the noisy speech y is input to the speech enhancement module for enhancement, and the estimation of the absolute value feature is obtained.

Multiply it with the sgn mask to synthesize the feature representation of the target speech:

变换为时域信号

Finally, through transposed convolution, the

Transform to time domain signal

Among them, the speech enhancement module adopted in this framework is based on the fully convolutional network. In the encoding process of this module, each layer of convolution reduces the feature size by half, but the channels are doubled, and the feature is encoded into a small and deep representation through multi-layer convolution, and correspondingly in the decoding process, the feature is expanded step by step. size, and finally returns to its original size. In the process of enlarging the feature size, we upsampling by bilinear interpolation to obtain higher resolution.

Add skip connections between layers of the same hierarchy in the speech enhancement module, which allows to preserve a high level of detail through the copy operation. The direct inflow of low-level information to high-level information can effectively guide the model to model high-resolution features.

The symmetrical structure of the speech enhancement module ensures that its input and output have the same shape size, which makes it naturally suitable for any pixel-intensive prediction task, especially the prediction task of each pixel label in an image.

In order to make full use of the multi-scale context information in speech features, we carefully designed multi-scale blocks and stacked them, as shown in Figure 2, SMB (Stacked Multi-scale Block, stacked multi-scale Block) contains an average pooling layer, Ordinary 1×1 and 3×3 convolutions, and dilated convolutions with different dilation rates; in order to effectively preserve the original information, we stack the original features with multi-scale features.

Speech enhancement methods based on deep learning often use mean squared error (MSE) as the loss function for training, but in the process of speech enhancement, the model is often tested by evaluating the intelligibility and speech quality of the enhanced speech. Whether the performance is good or bad, this inconsistency between the loss function and the evaluation metric does not guarantee the optimal model.

其首先去除对语音可懂度无贡献的无声区域，然后用STFT将时域信号编程时频域，通过将两个信号分割为50％重叠的带汉宁窗的帧。接着进行1/3倍频带分析，划分共15个1/3倍频带，其中频带中心频率范围为4.3kHz至150Hz。纯净语音的短时时间包络x_j，m可表示如下：In order to calculate the loss function from the perspective of magnitude value, we use RMSE (Root Mean Square Error, root mean square error); STOI is used to evaluate the intelligibility of speech, and its input is pure speech X and degenerate speech

It first removes the silent regions that do not contribute to speech intelligibility, and then uses STFT to program the time-domain signal into the time-frequency domain by dividing the two signals into 50% overlapping frames with Hanning windows. Then carry out 1/3 octave band analysis, and divide a total of 15 1/3 octave frequency bands, in which the center frequency of the frequency band ranges from 4.3kHz to 150Hz. The short-time envelope x _j,m of pure speech can be expressed as follows:

[X _j (m-L+1), X _j (m-L+2), ...X _j (m)] ^T

可懂度表示为两个时间包络之间的相关系数：where X∈R is the 1/3 octave band obtained by x, M is the total number of frames of a speech, m is the index of the frame, j is the index of the 1/3 octave band, and L corresponds to the length of the speech; Normalization and clipping. Normalization is used to compensate for global differences that should not have an impact on speech intelligibility; clipping ensures an upper bound for STOI evaluation on severely degraded speech. The normalized and cropped temporal envelope of degenerate speech is expressed as

Intelligibility is expressed as the correlation coefficient between two temporal envelopes:

where ||·|| ₂ is the L2 norm, and μ(·) represents the mean vector of the corresponding sample. Calculate the average of intelligibility of all bands and frames, and the STOI calculation index of degenerate speech can be obtained:

带入到STOI计算公式中，即可得到训练过程中的STOI计算指标：will enhance the voice

Bring it into the STOI calculation formula, and you can get the STOI calculation indicator during the training process:

Among them, d _{j, m} represents the correlation coefficient between the enhanced speech and the pure speech temporal envelope.

中干净分量

与其他分量的能量比值，干净分量

是x在

上的投影：On the other hand, SDR is Enhanced Speech

medium clean

Energy ratio to other components, clean component

is x in

Projection on :

SDR is defined as:

Combining the above two formulas can get:

Equivalent transformation of the SDR optimization objective to simplify the calculation:

Finally, we combine these two metrics with RMSE to form a loss function:

In the formula, α, β, γ correspond to the coefficients of different parts in the loss function.

Test example

The speech data used in the experiment comes from the TIMIT data set, and the noise data set adopts ESC-50 as the training set. In order to verify the generalization performance of the model proposed in this paper, we also use the Noisex92 noise data set for testing.

In this experiment, the TIMIT dataset contains a total of 6300 speeches, which were obtained by recording 10 sentences for each of 630 people, and the ratio of male to female is 7:3. Among them, 7 sentences recorded by each person are repeated. In order to remove the influence of repeated sentences on model training and testing, only 1890 speeches with different sentences were selected in this experiment. About 80% of the speech is used as the training set, and the other 20% is used as the test speech, and the ratio of male to female is the same as the overall distribution of TIMIT. The ESC-50 dataset contains a collection of 2000 labeled environmental recordings divided into 5 main categories: animals, natural soundscapes and water sounds, non-speech vocals, indoor sounds, and urban sounds. All speech was resampled to 16kHz, and all speech was truncated to 2 seconds length. The Adam optimizer is used for stochastic gradient descent (SGD) based optimization. The learning rate is set to a constant value equal to 1×10 ⁻⁴ .

For the baseline model, several typical encoder-decoder solutions are selected for comparison with the method proposed in the present invention, including spectral mapping-based and end-to-end methods, and we also use noisy speech as a baseline comparison: (a ) noisy speech, (b) AET, (c) CED, (d) R-CED, (e) noSMB-SE, (f) SMB-SE. Among them, AET is an end-to-end speech enhancement architecture, CED and R-CED are speech enhancement methods in the time-frequency domain of convolutional neural networks, and noSMB-SE is the basic framework of our proposed SMB-free version, which simply converts low-level information Connect to the high tier, while SMB-SE adds 4 SMBs based on noSMB-SE.

All models are trained at 0dB SNR and evaluate performance at -15dB, -10dB, -5dB, 0dB and 5dB SNR. To evaluate the generalization performance of the proposed framework, we also The proposed framework is tested on the Noisex-92 noise dataset.

TABLE I

Test results under visible noise conditions. Bold is best performance

TABLE II

Test results under invisible noise conditions. Bold is best performance

The present invention proposes an end-to-end speech enhancement framework using stacked multi-scale modules. First, the original time domain waveform is encoded into a two-dimensional feature representation, and then the speech enhancement module is used to learn the mapping relationship from noisy speech to clean speech. Decode the synthesized time-domain speech signal. The end-to-end framework proposed by the present invention can effectively extract the characteristic information of the time domain signal, the application of the SMB module helps the model to mine more information, and the integration of STOI, SDR and RMSE can effectively improve the overall enhancement performance of the model. The framework exhibits noise immunity under low SNR conditions and good generalization in unknown noise environments.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention. Inside.

Claims (7)

1. an end-to-end speech enhancement method using stacked multi-scale modules, is characterized in that, comprises the following steps: S1: Build a cascaded end-to-end speech enhancement framework and stitch stacked multi-scale modules into the network structure; S2: In the preprocessing stage, transform the time domain signal into two-dimensional features; S3: Use the speech enhancement module to enhance the two-dimensional features; S4: In the post-processing stage, the enhanced feature representation is transformed into a one-dimensional time domain signal by decoding synthesis. 2. speech enhancement method according to claim 1 is characterized in that: described cascading end-to-end speech enhancement framework comprises speech time domain signal preprocessing, speech enhancement module and target speech synthesis post-processing; Concrete steps comprise: a. In the time-domain signal preprocessing stage, one-dimensional convolution is used to perform the convolution operation on the input speech segment, and the results of each convolution check on the noisy speech y are stacked row by row to form a two-dimensional The real-valued feature Y is inspired by the processing method of the convolutional neural network on the pixel value of the image, and the two-dimensional feature is separated to obtain the absolute value feature and the sgn mask; b. The absolute value feature of the noisy speech y is input to the speech enhancement module for enhancement, and the estimation of the absolute value feature is obtained

Multiply it with the sgn mask to synthesize the feature representation of the target speech:

c. After transposed convolution, the

Transform to time domain signal

3. The speech enhancement method according to claim 1, wherein the multi-scale module comprises an average pooling layer, the convolution kernels are 1×1 and 3×3 convolutions, and dilated volumes with different dilation rates product. 4 . The speech enhancement method according to claim 1 , further comprising the steps of: incorporating the evaluation indicators STOI and SDR of speech enhancement into the loss function using a training strategy of multi-objective joint optimization. 5 . 5. speech enhancement method according to claim 4, is characterized in that, the concrete step of incorporating STOI index into the loss function comprises: 1) STOI input is pure speech x and degenerate speech

First remove the silent regions that do not contribute to speech intelligibility, then use STFT to transform the time-domain signal to the time-frequency domain by splitting the two signals into 50% overlapping frames with a Hanning window; 2) Carry out 1/3 octave frequency band analysis, divide a total of 15 1/3 octave frequency bands, in which the center frequency range of the frequency band is 4.3kHz to 150Hz, and the short-time envelope x _{j, m} of pure speech is expressed as follows: [X _j (m-L+1), X _j (m-L+2), ...X _j (m)] ^T where X∈R is the 1/3 octave band obtained by x, M is the total number of frames of a speech, m is the index of the frame, j is the index of the 1/3 octave band, and L corresponds to the length of the speech; 3) Normalize and crop the speech to obtain the envelope representation of the degraded speech

Intelligibility is expressed as the correlation coefficient between two temporal envelopes:

where ||·|| ₂ is the L2 norm, and μ(·) represents the mean vector of the corresponding sample. 4) Calculate the average of the intelligibility of all bands and frames, and the STOI calculation index of the degraded speech can be obtained:

5) Will enhance the voice

Bring it into the STOI calculation formula, and you can get the STOI calculation indicator during the training process:

Among them, d _{j, m} represents the correlation coefficient between the enhanced speech and the pure speech temporal envelope. 6. speech enhancement method according to claim 4, is characterized in that, the concrete step that SDR index is incorporated in loss function comprises: 1) The input of SDR is pure voice x and enhanced voice

The SDR calculation process for enhanced speech is as follows:

2) Equivalently transform the SDR optimization objective to simplify the calculation to obtain:

Among them, the process of maximizing the evaluation index SDR is equivalent to minimizing

7. speech enhancement method according to claim 4 is characterized in that, STOI and SDR evaluation index are fused into loss function, and concrete steps comprise: 1) Calculate the conventional root mean square error, the process is as follows:

Where M and N are the number of sampling points of each voice and the total number of voices; 2) Combine the root mean square error with the loss function of the evaluation index based on STOI and SDR:

In the formula, α, β, γ correspond to the coefficients of different parts in the loss function.

Images