CN116013344A - A Speech Enhancement Method in Multiple Noise Environments - Google Patents

Abstract

The invention discloses a voice enhancement method under multiple noise environments, which comprises the following steps: 1) Finishing the pretreatment of the audio and the data enhancement operation; 2) Extracting multi-level audio features by using a multi-scale encoder based on a transducer architecture, and strengthening key features by means of a feature lifting module; 3) Capturing long-term and short-term characteristics on different dimensions by means of a long-term and short-term sensing module based on a two-way architecture; 4) Obtaining a clean speech signal using a residual decoder and a mask estimation module; 5) The network model is jointly trained by means of a mean square error loss term and a signal to noise ratio loss term. The method has strong robustness and high instantaneity, and can effectively process ten common noises such as whistling, noisy, applause, bird song and the like, thereby improving the user experience of applications such as short videos, network live broadcast, video conferences, voice calls and the like. Compared with a part of mainstream voice enhancement model, the method can averagely improve 16% on the relevant evaluation index.

Description

A method for speech enhancement in various noise environments

Technical Field

The invention belongs to the technical field of speech noise reduction, and in particular relates to a speech enhancement method in multiple noise environments.

Background Art

Whether it is a short video or a live webcast, it faces a big problem: when the shooter is speaking, the surrounding background noise will also be collected, which will greatly reduce the actual experience of the audience. In addition, different shooters are in different surrounding environments, so the types of noise are also varied, such as: car horns, square music, children crying, construction site machinery, crowd noise, etc. The interference of the surrounding environment and the complexity of the application scenarios require the use of a robust speech enhancement technology to process noisy audio.

Of course, the application of speech enhancement technology is not limited to short videos or live webcasts, but can also serve a variety of downstream voice-related tasks, including: intelligent voice interaction, voice sentiment analysis, intelligent voice input, etc. In the field of intelligent voice interaction, common ones include smart speakers. In the field of intelligent voice input, common ones include voice input methods. Taking smart homes as an example, users can use voice to give instructions, thereby truly freeing their hands and avoiding direct contact with the device. Although voice-based intelligent interaction is becoming the mainstream human-computer interaction method, due to the complex noise environment in which users live, it still cannot completely replace keyboards or touch screens for input in daily life. Therefore, it is crucial to use speech enhancement technology to obtain pure speech from noisy mixed audio in real time.

At present, speech enhancement algorithms are mainly divided into the following according to different processing methods: harmonic enhancement method, which is only applicable to the removal of stationary white noise and cannot accurately estimate the pitch period of speech; spectral subtraction method, which is more effective in processing broadband noise, but the result after enhancement will have residual noise components; Wiener filtering method, the residual noise after enhancement is similar to white noise rather than musical noise; enhancement method based on speech model parameters, which has poor performance under low signal-to-noise ratio conditions and often requires multiple iterations; signal subspace method, which requires a large amount of calculation and is difficult to meet real-time requirements; enhancement method based on wavelet transform, which has poor denoising ability for non-stationary noise; method based on deep learning, which directly estimates pure speech signals with the help of data-driven, and has strong robustness and real-time performance. Compared with traditional methods, methods based on deep learning have incomparable performance advantages, so they have become the mainstream method of speech enhancement.

However, the current deep learning methods used for speech enhancement still face problems such as the inability to effectively capture long-term and short-term features and enhance key features, resulting in poor noise removal and weak robustness.

Summary of the invention

In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a speech enhancement method in a variety of noise environments, so as to more effectively remove noise in speech and have strong robustness and real-time performance.

In order to achieve the above object, the technical solution adopted by the present invention is:

A method for speech enhancement in a variety of noise environments, characterized by comprising the following steps:

Step 1: preprocessing and data enhancement operations are performed on the acquired audio data, and the processed audio data is input into a long-term and short-term perception enhancement model; the long-term and short-term perception enhancement model includes: a multi-scale encoder, a long-term and short-term perception module, and a residual decoder;

Step 2: for the processed audio data, extracting its deep audio features using the multi-scale encoder;

Step 3: Using the long-term and short-term perception modules to capture features in different dimensions respectively;

Step 4: Reconstruct the speech signal using the residual decoder, and estimate the mask of the clean speech using the mask estimation module, and multiply it with the original input audio to obtain the enhanced clean speech.

In one embodiment, the preprocessing operation includes one or more of the following operations: resampling the audio, trimming the audio length, and compressing the audio channels;

The data enhancement operation includes one or more of the following operations: mixing noise audio according to a random signal-to-noise ratio, randomly changing the volume of audio, and randomly adding a reverberation effect.

In one embodiment, the multi-scale encoder is based on the Transformer architecture, is composed of a plurality of stacked feature capture modules, and achieves feature downsampling with the help of a pooling operation; each feature capture module includes: a feature enhancement module, a normalization layer, and a feedforward neural network;

The feature enhancement module is used to capture key audio features and the relationship between features in the global scope. It uses convolutional layers, fully connected layers and sigmoid functions to obtain attention weights, and uses matrix corresponding element multiplication to achieve key feature enhancement, and uses a multi-head attention mechanism to capture the relationship between features in the global scope; the normalization layer performs normalization operations; the feedforward neural network uses a bidirectional gated recurrent unit to capture long-term and short-term features, and combines a fully connected layer to extract deep features;

Among them, different feature capture modules use different dilated convolution operations to capture features of different scales.

In one embodiment, the calculation formula of the feature capture module is as follows:

和

分别为特征捕获模块的输入特征、中间过程特征和输出特征，LayerNorm(·)为层归一化操作，FBM(·)为特征提升模块操作，FNN(·)为前馈神经网络；In the formula,

and

are the input features, intermediate process features and output features of the feature capture module, LayerNorm(·) is the layer normalization operation, FBM(·) is the feature enhancement module operation, and FNN(·) is the feedforward neural network;

The calculation formula of the feature enhancement module is as follows:

和

分别为特征提升模块的输入特征、中间特征和输出特征；C_1D(·)、FC(·)和R(·)分别为一维卷积、全连接层和调整通道操作；⊙和

分别表示矩阵对应元素相乘与相加操作；σ表示激活函数Sigmoid；MAM(·)表示多头注意力机制操作。In the formula,

and

are the input features, intermediate features and output features of the feature enhancement module respectively; C _1D (·), FC(·) and R(·) are the one-dimensional convolution, fully connected layer and channel adjustment operations respectively; ⊙ and

They represent the multiplication and addition operations of the corresponding elements of the matrix respectively; σ represents the activation function Sigmoid; MAM(·) represents the multi-head attention mechanism operation.

分别获得队列Q_i、键K_i、值V_i，计算公式如下：In one embodiment, the multi-head attention mechanism operates by first using a learnable linear transformation based on the input features.

Get the queue _Qi , key _Ki , and value _Vi respectively, and the calculation formula is as follows:

Where ^, _WiQ , _WiK ^and _WiV are the weights of the fully connected layers respectively ^;

Secondly, the similarity between the queue and the key value is calculated using the dot product method, and divided by the scaling factor;

Then, apply the Softmax activation function to obtain the weight corresponding to each value and multiply it with the corresponding value;

Finally, the results obtained from all heads are concatenated and linearly projected again to obtain the final output;

The specific calculation formula of the multi-head attention mechanism is as follows:

MAM(Q,K,V)=Concat(head ₁ ,...,head _h )W ^mh

Where ^Wmh is the linear transformation matrix, h is the number of parallel attention layers, and d is the scaling factor;

The output of the multi-head attention mechanism is used as the input of the feedforward neural network to obtain the final output features;

The feedforward neural network includes a gated recurrent unit, an activation function, and a fully connected layer. The calculation formula is as follows:

Where W _fc and b _fc represent the weights of the fully connected layer and the corresponding bias, δ represents the activation function ReLU, and the gated recurrent unit includes an update gate and a reset gate. The calculation formula is as follows:

z _t =σ(W _z ·[h _t-1 ,x _t ])r _t =σ(W _r ·[h _t-1 ,x _t ])

Where σ and γ represent activation functions Sigmoid and Tanh respectively, x _t , h _t-1 and h _t are the input features at this moment, the hidden state at the previous moment and the hidden state at the current moment respectively.

In one embodiment, the long-term and short-term perception module adopts a dual-path architecture, including a gated recurrent unit, a one-dimensional convolution module, an immediate layer normalization module and a channel adjustment module; the gated recurrent unit captures the long-term and short-term features of the features, the one-dimensional convolution module extracts deep features, and the immediate layer normalization module performs feature normalization processing.

In one embodiment, the calculation formula of the long-term and short-term perception module is as follows:

和

分别为长短期感知模块的输入特征、中间特征以及输出特征；In the formula, GRU(·) is a gated recurrent unit, C _1D (·) is a one-dimensional convolution operation, iLN(·) is an immediate normalization operation, and R(·) is a channel adjustment operation.

and

They are the input features, intermediate features, and output features of the long- and short-term perception modules respectively;

The calculation formula of the instant layer normalization module is as follows:

和

分别为均值操作和方差操作，符号ε和β分别为可学习的参数，符号λ为正则化参数。In the formula, _Xtf is the input feature, N and K are the dimensions of the feature,

and

are mean operation and variance operation respectively, the symbols ε and β are learnable parameters respectively, and the symbol λ is the regularization parameter.

和同级特征捕获模块的输出

其计算公式如下：In one embodiment, the residual decoder includes a plurality of decoding units, each of which includes a one-dimensional deconvolution module, a normalization module and an activation function; the input of each decoding unit is the output of the previous decoding unit.

And the output of the feature capture module at the same level

The calculation formula is as follows:

为当前解码单元的输出特征，解码器的输出为重构的语音信号。Where TC _1D (·) is a one-dimensional deconvolution operation, B(·) is a batch normalization operation, θ is the activation function PReLU,

is the output feature of the current decoding unit, and the output of the decoder is the reconstructed speech signal.

In one embodiment, the mask estimation module is composed of a one-dimensional convolution module and a plurality of different activation functions, and its calculation formula is as follows:

和

分别为掩码估计模块的输入特征、中间过程特征以及输出的掩码，γ、δ和σ分别为激活函数Tanh、ReLU和Sigmoid；In the formula,

and

are the input features, intermediate process features and output masks of the mask estimation module, γ, δ and σ are the activation functions Tanh, ReLU and Sigmoid respectively;

The output feature of the mask estimation module is multiplied by the original input speech signal to obtain the pure speech signal estimated by the model. The calculation formula is as follows:

Where _Xin is the original input audio signal, and _Xest is the pure speech estimated by the model.

In addition, the present invention utilizes a joint loss function to train the long-term and short-term perception enhancement model, wherein the joint loss function is composed of a mean square error loss term and a signal-to-noise ratio loss term, wherein the mean square error loss term is used to achieve optimization on a speech waveform graph, and the signal-to-noise ratio loss term is used to achieve optimization on a speech spectrum graph; wherein the mean square error loss term takes a logarithm to ensure that it has the same order of magnitude as the signal-to-noise ratio loss term.

Compared with the prior art, the present invention has the following beneficial effects:

(1) The present invention proposes a real-time speech denoising method based on the long-term and short-term perception enhancement model with the help of deep learning. The method has a small number of parameters, strong robustness, high real-time performance, and can be well applied to various noise scenes.

(2) The present invention proposes an encoder based on the Transformer architecture, which introduces an attention mechanism and a gated recurrent unit, which is conducive to solving the problem of capturing key features and the dependency of long-term and short-term features.

(3) The present invention proposes a feature enhancement module based on the attention mechanism, which can effectively capture the relationship between audio features in different ranges, thereby enhancing key audio features.

(4) The present invention proposes a long-term and short-term perception module based on a dual-path architecture, which can realize the extraction of long-term and short-term features in different dimensions, thereby providing more discriminative features for speech enhancement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the real-time speech enhancement method of the present invention.

FIG2 is a framework diagram of the medium- to short-term perception enhancement model of the present invention.

FIG3 is a schematic diagram of a feature capture module of a multi-scale encoder in the present invention.

FIG4 is a schematic diagram of the multi-head attention mechanism in the present invention.

FIG5 is a schematic diagram of a feature enhancement module based on an attention mechanism in the present invention.

FIG6 is a schematic diagram of a gated cycle unit in the present invention.

FIG7 is a schematic diagram of a long-term and short-term perception module based on a dual-path architecture in the present invention.

FIG8 is a diagram showing the effects of speech enhancement under various noises in the present invention.

DETAILED DESCRIPTION

The following is a detailed description of how the present invention uses technical means to solve technical problems and achieve the implementation process of technical effects in conjunction with the accompanying drawings and examples. It should be clear that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, as long as there is no conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other, and the technical solutions formed are all within the protection scope of the present invention.

The present invention discloses a method for speech enhancement in a variety of noise environments, as shown in FIG1 , comprising the following steps:

Step 1: Obtain audio data and perform preprocessing and data enhancement operations.

Step 1.1: Complete audio preprocessing

As a data-driven supervised learning method, speech enhancement technology based on deep learning requires that the input audio data has a fixed length, so the audio needs to be divided into fixed-length segments. Considering that the sampling rates of different audios are different, they need to be resampled first. With the help of the audio processing library librosa, the sampling rate of the audio can be adjusted to 16KHz and stored in WAV format. Since some audio may be multi-channel, channel compression operations are required to convert them into single-channel audio data. In order to simplify the calculation, the fusion strategy of multi-channel addition and averaging is directly adopted here. The specific calculation formula is as follows:

Wherein, K is the number of audio channels, S _mono is the processed single-channel audio, and _Si is the audio of a specific channel. Through channel compression, the multi-channel audio signal is compressed into a single-channel audio signal.

In addition, assuming that the input audio length of the model is 4 seconds, the audio length needs to be trimmed according to the audio trimming algorithm to ensure that the length of each audio segment is 4 seconds. Since the sampling rate of the audio is 16KHz, each audio segment contains 64,000 sampling points. Assuming that the total number of audio sampling points is T, the specific audio trimming calculation formula is as follows:

Where l is a positive integer, S _start and S _end are the IDs of the starting sampling point and the ending sampling point, respectively. When the number of audio sampling points after cropping does not meet 64,000 but the total number of sampling points is greater than 50,000, linear interpolation can be used to fill it up to 64,000 sampling points. When the number of cropped sampling points is less than 50,000, the cropped audio segment can be directly discarded.

Step 1.2: Complete the audio data augmentation operation

Considering the complexity and variability of model application scenarios, data enhancement technology is needed to improve the robustness of the model. In order to enhance the complexity of audio, three audio data enhancement methods are introduced here, which mainly include: random signal-to-noise ratio mixed noise audio, random change of audio volume, and random addition of reverberation effect.

The random mixed noise audio operation is mainly to introduce other additional background noise data and mix the input audio according to the random signal-to-noise ratio. For example, you can choose a variety of common noises such as electric drill sounds, whistle sounds, noise, dog barking, applause, bird sounds, gunshots, frog sounds, machine sounds, music sounds, etc. The specific operation process of this data enhancement is to first use the uniform random sampling method to generate a signal-to-noise ratio in the range of [-15,15], multiply the random signal-to-noise ratio with the original speech, and add the multiplied result to the noise audio to obtain the noisy mixed audio.

The random volume change operation mainly uses a random scaling factor to amplify or reduce the volume of the input audio. It mainly uses random uniform sampling to generate an audio scaling factor in the range of [0,2], and multiplies the scaling factor with the original audio to obtain audio with randomly adjusted volume.

The operation flow of randomly adding reverberation effects includes the following aspects: creating the room (defining the room size, the required reverberation time, the wall material, and the maximum number of reflections allowed), creating a signal source in the room, placing a microphone in the room, creating a room impulse response, simulating sound propagation, and synthesizing reverberation effects. In this embodiment, the reverberation effect of voice data can be added directly with the help of the Pyroomacoustics library.

Step 2: Extract deep audio features with the help of a multi-scale encoder.

The present invention designs an efficient long-term and short-term perception enhancement model with the help of deep learning technology, and inputs the audio processed in step 1 into the long-term and short-term perception enhancement model, thereby realizing real-time speech enhancement under various noises. Figure 2 shows the overall architecture of this model. The model mainly includes a multi-scale encoder, a long-term and short-term perception module, and a residual decoder. The multi-scale encoder is mainly used to realize the compression of audio features and the extraction of deep features, and the residual decoder is mainly used to realize the reconstruction of audio signals. In this embodiment, the multi-scale encoder is based on the Transformer architecture, which is mainly composed of a plurality of feature capture modules stacked, which are 5 in this embodiment. Each feature capture module also includes: a feature enhancement module, a normalization layer, and a feedforward neural network.

Figure 3 shows the details of the feature capture module in the multi-scale encoder based on the Transformer architecture. The specific calculation formula is as follows:

和

分别为特征捕获模块的输入特征、中间过程特征和输出特征。LayerNorm(·)为层归一化操作，FBM(·)为特征提升模块操作，FNN(·)为前馈神经网络。此外，特征捕获模块引入了残差连接保持原始特征，并使用基于注意力机制的特征提升模块实现关键特征的捕获与强化。图4展示了此模块中所采用的多头注意力机制的细节信息。对于特征捕获模块的整体流程而言，为了有效地捕获关键的音频特征和解决长短期特征依赖，首先将获取后的特征X_ie_n输入至特征提升模块，利用基于注意力机制的特征提升模块捕获关键的长短期特征，进而借助层归一化操作实现特征归一化，然后利用前馈神经网络实现深层特征的捕获，最后利用层归一化操作进行处理。此外，不同特征捕获模块之间借助最大池化操作实现特征的下采样。同时，不同特征捕获模块采用不同的膨胀卷积操作捕获不同尺度的特征。In the formula,

and

They are the input features, intermediate process features and output features of the feature capture module respectively. LayerNorm(·) is a layer normalization operation, FBM(·) is a feature enhancement module operation, and FNN(·) is a feedforward neural network. In addition, the feature capture module introduces a residual connection to maintain the original features, and uses a feature enhancement module based on an attention mechanism to capture and enhance key features. Figure 4 shows the details of the multi-head attention mechanism used in this module. For the overall process of the feature capture module, in order to effectively capture key audio features and solve long-term and short-term feature dependencies, the acquired _{feature Xien} is first input into the feature enhancement module, and the feature enhancement module based on the attention mechanism is used to capture key long-term and short-term features, and then the layer normalization operation is used to achieve feature normalization, and then the feedforward neural network is used to capture deep features, and finally the layer normalization operation is used for processing. In addition, the maximum pooling operation is used to achieve feature downsampling between different feature capture modules. At the same time, different feature capture modules use different dilated convolution operations to capture features of different scales.

The feature enhancement module is the core component of the feature capture module. The present invention uses the feature enhancement module to capture key audio features and the relationship between features in the global range, that is, effectively capture and strengthen important features. Figure 5 shows the detailed architecture of this module. This module mainly uses convolutional layers, fully connected layers and Sigmoid functions to obtain attention weights, and uses matrix corresponding elements to multiply to achieve key feature enhancement. At the same time, it can also capture the relationship between features in a larger range with the help of a multi-head attention mechanism, thereby eliminating harmonics as much as possible. The specific calculation formula is as follows:

和

分别为特征提升模块的输入特征、中间特征和输出特征。C_1D(·)、FC(·)和R(·)分别为一维卷积、全连接层和调整通道操作。⊙和

分别表示矩阵对应元素相乘与相加操作。此外，σ表示激活函数Sigmoid，从而便于求取关键特征对应的权重。MAM(·)则表示多头注意力机制操作。这里，使用卷积核大小为1的一维卷积操作实现特征通道的压缩，进而借助全连接层与激活函数Sigmoid获取相应的权重矩阵，最后利用矩阵对应元素相乘实现关键特征的强化。对于多头注意力机制而言，首先借助可学习的线性变换根据输入特征

分别获得队列Q_i、键K_i、值V_i，其具体的计算公式如下：In the formula,

and

are the input features, intermediate features, and output features of the feature enhancement module, respectively. _{C 1D} (·), FC(·), and R(·) are the one-dimensional convolution, fully connected layer, and channel adjustment operations, respectively. ⊙ and

Respectively represent the multiplication and addition operations of the corresponding elements of the matrix. In addition, σ represents the activation function Sigmoid, which makes it easier to obtain the weights corresponding to the key features. MAM(·) represents the multi-head attention mechanism operation. Here, a one-dimensional convolution operation with a convolution kernel size of 1 is used to compress the feature channel, and then the corresponding weight matrix is obtained with the help of the fully connected layer and the activation function Sigmoid. Finally, the corresponding elements of the matrix are multiplied to enhance the key features. For the multi-head attention mechanism, firstly, a learnable linear transformation is used to calculate the weights of the input features.

Get the queue _Qi , key _Ki , and value _Vi respectively. The specific calculation formula is as follows:

^In the formula, _WiQ , _WiK ^and _WiV are the weights of the fully connected layer. Secondly, the similarity between the queue and the ^key value is calculated by the dot product method, and it is also necessary to divide it by the scaling factor. Then, the Softmax activation function is applied to obtain the weight corresponding to each value and multiply it with the corresponding value. Finally, the results obtained by all heads need to be connected in series, and the linear projection operation is performed again to obtain the final output. The specific calculation formula of the multi-head attention mechanism is as follows:

MAM(Q,K,V)=Concat(head ₁ ,...,head _h )W ^mh

Where W ^mh is the linear transformation matrix and h is the number of parallel attention layers. The output of the multi-head attention module will be used as the input of the feedforward neural network to obtain the final output features. In this module, residual connections and layer normalization operations are also introduced to further improve the feature extraction effect.

The feedforward neural network mainly includes: gated recurrent unit, activation function and fully connected layer. It mainly uses the bidirectional gated recurrent unit to capture long-term and short-term features, and combines the fully connected layer to extract deep features. The specific calculation formula is as follows:

Where W _fc and b _fc represent the weights of the fully connected layer and the corresponding bias, and δ represents the activation function ReLU. Here, a bidirectional gated recurrent unit is used to capture audio features, which can not only effectively capture long-term and short-term features, but also avoid the problem of complex LSTM calculations. In addition, compared with simply using a fully connected layer, this method can often achieve more satisfactory results. At the same time, compared with one-dimensional convolution, the bidirectional gated recurrent unit can perceive the relationship between farther features and automatically focus on more important features. Figure 6 shows the specific implementation details of the gated recurrent unit, which mainly includes an update gate and a reset gate. The specific calculation formula is as follows:

z _t =σ(W _z ·[h _t-1 ,x _t ])r _t =σW _r ·[h _t-1 ,x _t ])

Where σ and γ represent activation functions Sigmoid and Tanh respectively, x _t , h _t-1 and h _t are the input features at this moment, the hidden state at the previous moment and the hidden state at the current moment respectively.

Step 3: Use the long-term and short-term perception modules to capture features in different dimensions.

For the speech features extracted by the multi-scale encoder, it is necessary to further process the relationship between the features in different dimensions. Therefore, the present invention designs a long-term and short-term perception module using a dual-path architecture, which can effectively realize the capture of long-term and short-term audio features in different dimensions, thereby effectively solving the long-term and short-term dependencies between features. As shown in Figure 7, the detailed architecture of the long-term and short-term perception module is shown. This module mainly uses gated cyclic units, one-dimensional convolution operations, instant layer normalization operations and channel adjustment operations to respectively realize the capture of long-term and short-term features in the time dimension and feature dimension. It is worth noting that this embodiment uses an instant layer normalization operation to replace the traditional layer normalization operation to reduce the sensitivity of the model to the energy of the input signal. At the same time, in order to maintain the original features, this module also introduces the idea of residual connection. Whether it is the time dimension or the feature dimension, it uses the gated cyclic unit to realize the extraction of long-term and short-term features in different ranges, and uses the one-dimensional convolution operation to realize the capture of deep features, and then uses the instant normalization operation to realize the normalization of features.

The specific calculation formula of this module is as follows:

和

分别为此模块的输入特征、中间特征以及输出特征。当特征输入至网络中，首先利用GRU实现时间维度上长短期特征的捕获，进而利用一维卷积操作实现深层特征的提取，然后借助即时层归一化操作进行特征归一化处理。这里之所以采用GRU，是因为其与LSTM相比所需要的计算资源与时间成本更少，却能够达到相同的效果。GRU仅包含控制重置的门控和控制更新的门控，其有效地解决了长短期记忆的问题。另外，此模块所使用的即时层归一化操作的具体计算公式如下：In the formula, GRU(·) is a gated recurrent unit, C _1D (·) is a one-dimensional convolution operation, iLN(·) is an immediate normalization operation, and R(·) is a channel adjustment operation. In addition,

and

They are the input features, intermediate features, and output features of this module respectively. When the features are input into the network, GRU is first used to capture the long-term and short-term features in the time dimension, and then the one-dimensional convolution operation is used to extract the deep features, and then the feature normalization is performed with the help of the instantaneous layer normalization operation. The reason why GRU is used here is that it requires less computing resources and time cost than LSTM, but can achieve the same effect. GRU only contains the gate for controlling reset and the gate for controlling update, which effectively solves the problem of long-term and short-term memory. In addition, the specific calculation formula of the instantaneous layer normalization operation used in this module is as follows:

和

分别为均值操作和方差操作。此外，符号ε和β分别为可学习的参数，符号λ为正则化参数。该归一化操作可以降低模型对输入信号能量的敏感程度。为了实现特征维度上长短期特征的捕获，需要将特征的两个通道进行调换，进而利用GRU捕获长期特征，其次借助一维卷积实现深层特征的提取，并使用即时归一化操作进行处理，最后借助通道调整操作获得输出特征。In the formula, _Xtf is the input feature, N and K are the dimensions of the feature. In addition,

and

are mean operation and variance operation respectively. In addition, the symbols ε and β are learnable parameters, and the symbol λ is a regularization parameter. This normalization operation can reduce the sensitivity of the model to the energy of the input signal. In order to capture long-term and short-term features in the feature dimension, it is necessary to swap the two channels of the feature, and then use GRU to capture the long-term features. Secondly, use one-dimensional convolution to extract deep features, and use instant normalization operation for processing. Finally, the output features are obtained with the help of channel adjustment operation.

Step 4: Use the residual decoder to obtain the enhanced clean speech.

另一个是来自于同级特征捕获模块的输出

解码单元使用一维反卷积同时实现特征的提取与上采样操作，并借助激活函数PReLU增加模型的非线性能力。其具体的计算公式如下：In order to obtain pure speech, it is necessary to first reconstruct the speech signal with the help of a residual decoder. This residual decoder mainly includes multiple decoding units, 5 in this embodiment, which can gradually realize the estimation of the spectrum mask. For each decoding unit, it is mainly composed of a one-dimensional deconvolution operation, a normalization operation and an activation function. At the same time, in order to better reconstruct the speech signal, the input of each decoding unit includes two parts: one is the output from the previous decoding unit

The other is the output from the feature capture module at the same level

The decoding unit uses one-dimensional deconvolution to simultaneously extract features and perform upsampling operations, and uses the activation function PReLU to increase the nonlinearity of the model. The specific calculation formula is as follows:

为当前解码单元的输出特征，解码器的输出为重构的语音信号。此时，需要借助掩码估计模块处理解码器输出的重构语音信号，估计纯净语音信号的掩码，从而实现纯净语音掩码的生成。此掩码估计模块由一维卷积操作和多个不同的激活函数构成，其具体的计算公式如下：Where TC _1D (·) is a one-dimensional deconvolution operation, which is mainly used to implement feature extraction and upsampling operations. B(·) is a batch normalization operation, and θ is the activation function PReLU. In addition,

is the output feature of the current decoding unit, and the output of the decoder is the reconstructed speech signal. At this time, it is necessary to use the mask estimation module to process the reconstructed speech signal output by the decoder, estimate the mask of the pure speech signal, and thus realize the generation of the pure speech mask. This mask estimation module consists of a one-dimensional convolution operation and multiple different activation functions, and its specific calculation formula is as follows:

和

分别为掩码估计模块的输入特征、中间过程特征以及输出的掩码。此外，γ、δ和σ分别为激活函数Tanh、ReLU和Sigmoid。将掩码估计模块的输出特征与原始输入的语音信号相乘即可获得模型估计的纯净语音信号，其计算公式如下：In the formula,

and

are the input features, intermediate process features, and output masks of the mask estimation module. In addition, γ, δ, and σ are activation functions Tanh, ReLU, and Sigmoid, respectively. The output features of the mask estimation module are multiplied by the original input speech signal to obtain the pure speech signal estimated by the model. The calculation formula is as follows:

Where _Xin is the original input audio signal, and _Xest is the pure speech estimated by the model.

The model and process of the present invention are as described above. Furthermore, the model needs to be trained or tested to obtain a model that meets the requirements.

Specifically, in order to complete the supervised training of the model, the present invention introduces a joint loss function, which includes two parts: the signal-to-noise ratio loss term f(·) and the mean square error loss term MSE(·). The former is mainly used to achieve optimization on the speech waveform graph, and the latter is mainly used to achieve optimization on the speech spectrogram. In addition, the logarithm of the mean square error loss term needs to be taken to ensure that it has the same order of magnitude as the signal-to-noise ratio loss term.

The specific expression of the loss function is as follows:

分别为纯净的音频与模型估计的音频，S_r和

分别为纯净的频谱图的实部与模型估计的频谱图的实部，S_i和

分别为纯净的频谱图的虚部与模型估计的频谱图的虚部，|S|和

分别为纯净的频谱图的幅值与模型估计的频谱图的幅值。此外，均方误差损失项可以测量模型估计频谱图和真实频谱图之间的实部、虚部以及幅度的差异。同时，对均方误差损失项取对数，以确保其与信噪比损失项具有相同的数量级。信噪比损失项则可以约束输出的振幅，避免输入和输出之间的电平偏移。该损失项具体的计算公式如下：In the formula, s and

are the pure audio and the model estimated audio, S _r and

are the real part of the pure spectrum and the real part of the spectrum estimated by the model, _Si and

are the imaginary parts of the pure spectrum and the model-estimated spectrum, respectively, |S| and

are the amplitude of the pure spectrum and the amplitude of the spectrum estimated by the model, respectively. In addition, the mean square error loss term can measure the difference in real part, imaginary part and amplitude between the model estimated spectrum and the true spectrum. At the same time, the logarithm of the mean square error loss term is taken to ensure that it has the same order of magnitude as the signal-to-noise ratio loss term. The signal-to-noise ratio loss term can constrain the amplitude of the output and avoid level offset between the input and output. The specific calculation formula of this loss term is as follows:

In order to prove the effectiveness of the method proposed in the present invention, relevant experimental tests were carried out. A large amount of noise audio is integrated on the basis of the existing pure speech to simulate the speech collected under various noises. The types of noise selected here are: electric drill sound, whistle sound, noise, dog barking, applause, bird sound, gunshot sound, frog sound, machine sound, and music sound. At the same time, the effect of speech enhancement is measured by means of three commonly used evaluation indicators for speech enhancement, namely: perceptual speech quality evaluation (PESQ), short-term speech intelligibility (STOI) and source artifact ratio (SAR). Among them, PESQ and STOI are both perceptual level evaluation methods, and the larger the value, the better the effect of speech enhancement. For STOI, its calculation process mainly includes three steps: removing silent frames; completing 1/3 octave band decomposition of DFT for the signal; calculating the correlation coefficient before and after the time envelope of enhancement and taking the average. For PESQ, it requires a noisy attenuated signal and an original reference signal, and the calculation process includes preprocessing, time alignment, perceptual filtering, masking effect and other operations. It can provide a subjective prediction value for objective speech quality assessment and can be mapped to the MOS scale range, with a score range of -0.5–4.5. In addition, the evaluation index SAR can be regarded as an evaluation index of the signal level. The larger the value, the better the speech enhancement effect. The specific calculation formula is as follows:

Wherein, e _interf , e _noise and e _artif are the error signals introduced by interference, noise and artifacts respectively, and s _target is the target signal. Table 1 shows the comparison of the effects of the present invention with the mainstream methods on the above evaluation indicators. It is not difficult to find that it can improve the PESQ evaluation index by about 16% compared with the mainstream speech enhancement model Demucs, and improve the SAR evaluation index by about 16% compared with the mainstream speech enhancement model MannerNet. At the same time, it can achieve an excellent performance of 0.94 on the evaluation index STOI. In addition, for ten different noise interference environments, Figure 8 shows the speech effect diagram after denoising based on the long-term and short-term perception enhancement model proposed by the present invention, which can achieve satisfactory results.

Table 1 Comparison of the effects of the long-term and short-term perception enhancement model of the present invention and the mainstream speech enhancement model

PESQ STOI SAR Demucs 2.08 0.93 18.70 MannerNet 2.22 0.94 17.41 Long-term and short-term perception enhancement model 2.41 0.94 20.27

Claims (10)

1. A method for speech enhancement in a variety of noise environments, characterized in that it comprises the following steps: Step 1: preprocessing and data enhancement operations are performed on the acquired audio data, and the processed audio data is input into a long-term and short-term perception enhancement model; the long-term and short-term perception enhancement model includes a multi-scale encoder, a long-term and short-term perception module, and a residual decoder; Step 2: for the processed audio data, extracting its deep audio features using the multi-scale encoder; Step 3: Using the long-term and short-term perception modules to capture features in different dimensions respectively; Step 4: Reconstruct the speech signal using the residual decoder, and estimate the mask of the clean speech using the mask estimation module, multiply it with the original input audio to obtain the enhanced clean speech. The model training is completed with the help of the joint loss function. 2. According to the method for speech enhancement in multiple noise environments of claim 1, the preprocessing operation comprises one or more of the following operations: resampling the audio, trimming the audio length, and compressing the audio channel; The data enhancement operation includes one or more of the following operations: mixing noise audio according to a random signal-to-noise ratio, randomly changing the volume of audio, and randomly adding a reverberation effect. 3. According to the method for speech enhancement in a variety of noise environments described in claim 1, it is characterized in that the multi-scale encoder is based on the Transformer architecture, is composed of a plurality of feature capture modules stacked, and achieves feature downsampling by means of a pooling operation; each feature capture module includes: a feature enhancement module, a normalization layer and a feedforward neural network; The feature enhancement module is used to capture key audio features and the relationship between features in the global scope. It uses convolutional layers, fully connected layers and Sigmoid functions to obtain attention weights, and uses matrix corresponding element multiplication to achieve key feature enhancement, and uses a multi-head attention mechanism to capture the relationship between features in the global scope; the normalization layer performs normalization operations; the feedforward neural network uses a bidirectional gated recurrent unit to capture long-term and short-term features, and combines a fully connected layer to extract deep features; Among them, different feature capture modules use different dilated convolution operations to capture features of different scales. 4. According to the method for speech enhancement in multiple noise environments as claimed in claim 3, it is characterized in that the calculation formula of the feature capture module is as follows:

In the formula,

and

are the input features, intermediate process features and output features of the feature capture module, LayerNorm(·) is the layer normalization operation, FBM(·) is the feature enhancement module operation, and FNN(·) is the feedforward neural network; The calculation formula of the feature enhancement module is as follows:

In the formula,

and

are the input features, intermediate features and output features of the feature enhancement module respectively; C _1D (·), FC(·) and R(·) are the one-dimensional convolution, fully connected layer and channel adjustment operations respectively; ⊙ and

They represent the multiplication and addition operations of the corresponding elements of the matrix respectively; σ represents the activation function Sigmoid; MAM(·) represents the multi-head attention mechanism operation. 5. According to the method of claim 4, the multi-head attention mechanism is characterized in that the multi-head attention mechanism first uses a learnable linear transformation according to the input feature

Get the queue _Qi , key _Ki , and value _Vi respectively, and the calculation formula is as follows:

Where ^, _WiQ , _WiK ^and _WiV are the weights of the fully connected layers respectively ^; Secondly, the similarity between the queue and the key value is calculated using the dot product method, and divided by the scaling factor; Then, apply the Softmax activation function to obtain the weight corresponding to each value and multiply it with the corresponding value; Finally, the results obtained from all heads are concatenated and linearly projected again to obtain the final output; The specific calculation formula of the multi-head attention mechanism is as follows:

MAM(Q,K,V)=Concat(head ₁ ,...,head _h )W ^mh Where ^Wmh is the linear transformation matrix, h is the number of parallel attention layers, and d is the scaling factor; The output of the multi-head attention mechanism is used as the input of the feedforward neural network to obtain the final output features; The feedforward neural network includes: gated recurrent unit, activation function and fully connected layer, and its calculation formula is as follows:

Where W _fc and b _fc represent the weights of the fully connected layer and the corresponding bias, δ represents the activation function ReLU, and the gated recurrent unit includes an update gate and a reset gate. The calculation formula is as follows: z _t =σ(W _z ·[h _t-1 ,x _t ])r _t =σ(W _r ·[h _t-1 ,x _t ])

Where σ and γ represent activation functions Sigmoid and Tanh respectively, x _t , h _t-1 and h _t are the input features at this moment, the hidden state at the previous moment and the hidden state at the current moment respectively. 6. According to the method for speech enhancement in multiple noise environments described in claim 1, it is characterized in that the long-term and short-term perception module adopts a dual-path architecture, including a gated recurrent unit, a one-dimensional convolution module, an instantaneous layer normalization module and a channel adjustment module; the gated recurrent unit captures the long-term and short-term features of the features, the one-dimensional convolution module extracts deep features, and the instantaneous layer normalization module performs feature normalization. 7. The method for speech enhancement in multiple noise environments according to claim 6, wherein the calculation formula of the long-term and short-term perception module is as follows:

In the formula, GRU(·) is a gated recurrent unit, C _1D (·) is a one-dimensional convolution operation, iLN(·) is an immediate normalization operation, and R(·) is a channel adjustment operation.

and

They are the input features, intermediate features, and output features of the long- and short-term perception modules respectively; The calculation formula of the instant layer normalization module is as follows:

In the formula, _Xtf is the input feature, N and K are the dimensions of the feature,

and

are mean operation and variance operation respectively, the symbols ε and β are learnable parameters respectively, and the symbol λ is the regularization parameter. 8. According to the method for speech enhancement in a variety of noise environments described in claim 1, it is characterized in that the residual decoder includes a plurality of decoding units, each decoding unit includes a one-dimensional deconvolution module, a normalization module and an activation function; the input of each decoding unit is the output of the previous decoding unit

And the output of the feature capture module at the same level

The calculation formula is as follows:

Where TC _1D (·) is a one-dimensional deconvolution operation, B(·) is a batch normalization operation, θ is the activation function PReLU,

is the output feature of the current decoding unit, and the output of the decoder is the reconstructed speech signal. 9. According to the method for speech enhancement in multiple noise environments of claim 1, the mask estimation module is composed of a one-dimensional convolution module and a plurality of different activation functions, and its calculation formula is as follows:

In the formula,

and

are the input features, intermediate process features and output masks of the mask estimation module, respectively. γ, δ and σ are the activation functions Tanh, ReLU and Sigmoid, respectively. The output feature of the mask estimation module is multiplied by the original input speech signal to obtain the pure speech signal estimated by the model. The calculation formula is as follows:

Where _Xin is the original input audio signal, and _Xest is the pure speech estimated by the model. 10. According to claim 1, a method for speech enhancement in multiple noise environments is characterized in that the long-term and short-term perception enhancement model is trained using a joint loss function, and the joint loss function is composed of a mean square error loss term and a signal-to-noise ratio loss term, the mean square error loss term is used to achieve optimization on the speech waveform graph, and the signal-to-noise ratio loss term is used to achieve optimization on the speech spectrum graph; wherein the mean square error loss term takes a logarithm to ensure that it has the same order of magnitude as the signal-to-noise ratio loss term.

Images