KR102358151B1 - Noise reduction method using convolutional recurrent network - Google Patents

Abstract

A noise removal method according to a preferred embodiment of the present invention comprises: a step of converting a digital input signal into frequency components; a variance normalization step of size-normalizing the variance of the power value of each frequency component of the signal converted into the frequency component on the time axis; a forward propagation step of passing the normalized signal through a convolutional neural network having a plurality of hidden layers that processes and outputs an input using a weight and a bias value; a power compensation step of compensating for the power value of the signal subtracted in the variance normalization step for the signal that has passed through the forward propagation step; a step of converting the frequency component signal into a digital output signal; and a cost function step of calculating a cost function from the digital output signal and updating the weight and bias to minimize the cost function. According to the present invention, provided is a noise removal method having high noise removal efficiency compared to the prior art.

Description

{Noise reduction method using convolutional recurrent network}

The present invention relates to a noise removal method using a convolutional neural network, and more particularly, to a noise removal method using a convolutional neural network with improved noise removal efficiency.

Artificial Neural Network (ANN) has developed a lot after the development of Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), etc.

With the development of artificial intelligence technology, artificial intelligence technology has been widely applied in the field of speech processing. For example, Deep Neural Network (DNN), a machine learning technique, is showing excellent performance in various speech enhancement and speech recognition studies. The deep neural network shows excellent performance by effectively modeling the nonlinear relationship between the input feature vector and the output feature vector through multiple hidden layers and hidden nodes.

Korean Patent Registration No. 10-0762596 relates to a voice signal preprocessing system and a method for extracting voice signal characteristic information, and describes a technique for preprocessing a voice signal using a neural network recognition method.

Korean Patent Registration No. 10-1934636 describes a technology for integrally removing noise and echo from a voice signal by using noise information and echo information, which are statistical information of a voice signal, as additional inputs to a deep neural network (DNN).

Convolutional Recurrent Network (CRN), which combines CNN and RNN, is also applied to the field of extracting the target sound by removing noise from the input signal including noise. In addition, the Deep Complex Convolution Recurrent Network (DCCRN), which can implement the CRN algorithm when entering a complex number, is showing good performance.

In artificial intelligence technology, noise is removed by training an input signal containing a noise signal. However, in an actual situation, the magnitude of the input signal, especially the magnitude of the noise, is not constant. Therefore, it is necessary to learn noise by assuming various cases such as 20, 15, 10, 5, 0, -5, -10 dB for the signal-to-noise ratio (hereinafter referred to as 'SNRi') of the input signal. However, when a signal with untrained SNRi is input, the noise is not effectively removed with the weights and bias values learned from the existing data, and residual noise remains in the final output sound, making it difficult to hear.

The present invention has been made in view of this point, and an object of the present invention is to improve noise removal efficiency by improving the conventional CRN and DCCRN algorithms, and to provide a method in which the final output sound is naturally audible.

A noise removal method according to a preferred embodiment of the present invention comprises the steps of: converting a digital input signal into a frequency component; Step, a forward propagation step of passing a normalized signal through a convolutional neural network having a plurality of hidden layers that process an input using a weight and a bias value and output it, and subtract the distributed normalization step for the signal that has passed the forward propagation step A power compensation step of compensating for the power value of the converted signal, a step of converting a frequency component signal into a digital output signal, and a cost function of calculating a cost function from the digital output signal and updating the weight and bias to minimize the cost function steps are provided.

In the variance normalization step, the following equation

It is possible to obtain a signal obtained by normalizing the magnitude of the variance of the power value of each frequency component along the time axis. Here, epsi is an experimental constant value to prevent division by too small values when the input value is small.

Assuming that the power value of each frequency component of the input signal x(n) is X(k), and E [ X(k) ] is the ensemble average of X(k) for the past M frames,

is,

는 0<

<1 인 스무딩 상수이고, X _av (k)는 각 주파수 성분의 파워값에 대한 Long-Term IIR(Infinite Impulse Response) 평균이다. as,

is 0<

<1 is a smoothing constant, and X _av ( k ) is the Long-Term Infinite Impulse Response (IIR) average of the power values of each frequency component.

The power compensating step compensates for a difference in power value between the input signal X ( k ) and NVar_X ( k ).

In the forward propagation phase, each hidden layer uses a chance function to determine the value to pass to the next hidden layer. In an embodiment, a smoothing function of a Gaussian Error Linear Unit (GELU) function may be used as the activation function. Specifically, when ρ is a smoothing constant of 0 < ρ < 1, the activation function

can be used

In this case, the output y _t passed through the t-th hidden layer is

is expressed by

In an embodiment, as the cost function, a Euclidean distance between a frequency component of an output signal and a frequency component of a signal obtained by passing the correct answer signal through a variance normalization step may be used.

는 유클리디언 거리(Euclidean distance)를 나타내고,

와

를 각각

와

를 Mel 필터 뱅크에 통과시킨 벡터라 할 때,M _i denotes the i-th Mel filter bank, and the subscripts t and f denote signals in the waveform domain and spectral domain, respectively,

represents the Euclidean distance,

Wow

each

Wow

Assuming that is a vector passed through the Mel filter bank,

The cost function can be calculated by

According to the present invention, there is provided a noise removal method having high noise removal efficiency compared to the prior art. Also, according to the present invention, there is provided a method in which the final output sound is heard naturally.

1 is a block diagram of a general CRN algorithm.
2 is a block diagram showing a noise removal method using a convolutional recurrent neural network according to a preferred embodiment of the present invention.
3 is a block diagram of a hidden layer constituting a forward propagation step.
4 is a graph showing representative activation functions.
5 is a graph showing an output signal when the conventional CRN algorithm and the NV-CRN algorithm of the present invention are applied when a noise signal is input.
6 is a table showing performance indicators when the conventional CRN algorithm and the NV-CRN algorithm of the present invention are applied when a noise signal is input.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

을 얻는다. 비용 함수(Loss Function) 단계(16)에서는 비용함수를 계산하여 이 비용함수를 최소화하는 가중치 w와 바이어스 bias를 갱신해나감으로써 출력신호

가 s(n)에 근접되도록 만들게 한다.1 is a block diagram of a general CRN algorithm 10 . The digital input signal x(n) is a signal including noise, and a Short-Term Fast Fourier Transform is performed in the ST-FFT step 11 to convert the digital input signal into a frequency component. In the convolution step 12, convolution with a pre-designed filter is performed, and after passing the activation function in the activation function step 13, in the pooling step 14, the data dimension is reduced. Finally, in the ST-iFFT step (15), a Short-Term Inverse Fast Fourier Transform is performed to the output signal

to get In the loss function step 16, the output signal is calculated by calculating the cost function and updating the weight w and the bias bias that minimize the cost function.

to be close to s(n) .

The input signal x(n) including noise can be expressed as in Equation 1. In Equation 1, s(n) is a target (correct answer) signal, and v(n) is a noise signal.

The activation function step 13 and the pooling step 14 in the box indicated by the dotted line in FIG. 1 are performed several times, and the weight w and the bias bias used in each step are in the direction of minimizing the cost function. is updated with

However, in an actual situation, the magnitude of the input signal, particularly the magnitude of the noise, is not constant. Therefore, it is necessary to learn noise by assuming various cases such as 20, 15, 10, 5, 0, -5, -10 dB for the signal-to-noise ratio (hereinafter referred to as 'SNRi') of the input signal. However, if a signal with an SNRi of 30 dB, -15 dB, etc. is input, the noise is not effectively removed with the weight and bias values learned from the existing data, and residual noise remains in the final output sound, making it difficult to hear. phenomenon may occur.

In order to solve this problem, in the present invention, after ST-FFT the input signal, the power of each frequency component is calculated, the change amount of this value is calculated, and the change amount is normalized and then input to the CRN algorithm. The signal input to the CRN algorithm becomes a normalized variation value in which the DC value is deleted from the original signal. Since this value is normalized by removing the DC value from the original signal, the dynamic range is much reduced than that of the original signal. Accordingly, there is an effect that the number of cases of data to be learned is significantly reduced, and the amount of calculation is greatly reduced even in real-time implementation. In the following description, the algorithm according to the present invention is referred to as a Normailed Variance (NV)-CRN algorithm.

2 is a block diagram showing a noise removal method using a convolutional recurrent neural network according to a preferred embodiment of the present invention.

In the method shown in FIG. 2 , a signal obtained by performing a short-time fast Fourier transform of an input signal in the ST-FFT step 110 is subjected to a dispersion normalization step 115 and then a forward propagation step is performed. In the variance normalization step 115, the magnitude of the variance of the power value of each frequency component of the input signal converted into the frequency component is normalized on the time axis. In the forward propagation step, it goes through a plurality of hidden layers. In each hidden layer, the input x is processed using the weight W and the bias b value, the output y is calculated, and it is transferred to the next hidden layer.

If the input signal x(n) is squared by ST-FFT, the power value of each frequency component can be obtained as shown in Equation (2).

The time-axis variance of X(k) in the current frame can be calculated as in Equation (3).

Here, E [ X(k) ] represents the ensemble average of X(k) for the past M frames.

The average of the Long-Term Infinite Impulse Response (IIR) for the power value of each frequency component can be obtained using Equation (4).

는 0<

<1 인 스무딩 상수이다.From here

is 0<

<1 is the smoothing constant.

Equation 5 is calculated in order to size-normalize the variance of the power value of each frequency component on the time axis.

Here, epsi is an experimental constant value to prevent division by too small values when the input value is small.

The signal obtained by Equation 5 becomes the output of the variance normalization step 115, and this signal is input to the forward propagation step.

A block diagram of the hidden layers constituting the forward propagation stage is shown in FIG. 3 . In the forward propagation step, it goes through a plurality of hidden layers. In each hidden layer, the input x is processed using the weight W and the bias b value, the output y is calculated, and it is transferred to the next hidden layer.

In FIG. 3 , the calculation formula of the hidden layer of the current t may be expressed as Equation (6).

Here, tanh is a hyperbolic tangent function, and it is an activation function that processes the current value and decides which value to transfer to the next hidden layer. The activation function plays a very important role in the artificial neural network algorithm. As the activation function, in addition to the tanh-like sigmoid and ELU (Exponential Linear Unit), ReLU (Rectified Linear Unit) and GELU (Gaussian Error Linear Unit) are known. 4 shows examples of representative activation functions.

The GELU function, which is known to show good results among the activation functions, can be expressed as in Equation 7.

When the GELU function of Equation 7 is applied to the hidden layer of FIG. 3 , the output y _t passing through the current hidden layer can be expressed as Equation 8.

However, assuming that the same activation function is applied to all hidden layers, there is no guarantee that the learning rate will not decrease and problems such as overfitting or gradient vanishing will not occur. The gradient annihilation problem means that as the propagation progresses, the h calculated in the current hidden layer hardly reflects the past input vector x.

Equation 9 obtained by smoothing the GELU function may be used as an activation function in order to alleviate the determination of the output y with respect to the current value x when the amount of change on the time and frequency axes is large, such as an acoustic or voice signal.

where ρ is a smoothing constant with 0 < ρ < 1.

The output y _t passing through the hidden layer to which the activation function of Equation 9 is applied can be expressed as Equation 10.

When the output is calculated as in Equation 10, it is possible to obtain an output that is comfortable to the auditory sense with respect to an input vector having a rapid change amount on the time axis.

After performing the forward propagation step, the power compensation step 155 is performed. In the power compensation step 155 , the power value of the signal subtracted from the normalization process by subtracting the DC component of the input signal in the dispersion normalization step 115 is compensated. That is, the difference in power value between the input signal X ( k ) and NVar_X ( k ) of Equation 5 is compensated.

을 얻을 수 있다. 비용함수(Loss Function) 단계(160)에서는 출력신호로부터 비용함수를 계산하여 이 비용함수를 최소화하는 가중치 w와 바이어스 bias를 갱신해나감으로써 출력신호

가 s(n)에 근접되도록 만들게 한다.When ST-iFFT (150) is executed on the signal that has passed the power compensation step (155), the output signal

can be obtained In the cost function step 160, the output signal is calculated by calculating the cost function from the output signal and updating the weight w and the bias bias to minimize the cost function.

to be close to s(n) .

의 주파수 성분

와 수학식 1의 정답 신호인 s(k)의 주파수 성분 사이의 유클리디언 거리를 수학식 11과 같이 계산할 수 있다. 이 값을 지각 비용함수(perceptual loss function)라 정의한다.output signal

frequency component of

and Euclidean distance between the frequency component of s(k) , which is the correct signal of Equation 1, can be calculated as in Equation 11. This value is defined as a perceptual loss function.

는 유클리디언 거리(Euclidean distance)를 나타낸다.Here, M _i means the i-th Mel filter bank, and the subscripts t and f are abbreviations of time and frequency, respectively, and mean signals in the waveform domain and the spectrum domain, respectively.

represents the Euclidean distance.

와

를 각각 Mel 필터 뱅크에 통과시킨 벡터

와

에 대한 지각 비용함수이다.In an embodiment, a perceptual cost function as in Equation 12 may be used. Equation 12 is

Wow

vectors passed through each Mel filter bank

Wow

is the perceptual cost function for .

In another embodiment, as shown in Equation 13, the sum of the cost function of Equation 11 and the cost function of Equation 12 may be used as the cost function.

5 shows the output signal when the conventional CRN algorithm and the NV-CRN algorithm of the present invention are applied when the noise signal is input, and FIG. 6 shows the performance index at that time.

In FIG. 6 , Perceptual Evaluation Of Speech Quality (PESQ) and STOI were used as performance comparison indicators. For both PESQ and STOI, a larger value indicates better performance. PESQ is a performance index used in standards such as ITU-T P.862, and STOI is a performance index obtained by the following equation, where X(n) represents the original sound signal and Y(n) represents the microphone signal.

It can be seen from FIG. 6 that the algorithm of the present invention shows better performance than the conventional CRN algorithm in both the case where the signal-to-noise ratio (SNRi) is 0 dB and the case of 10 dB.

As mentioned above, although the present invention has been described with a few examples, the present invention is not necessarily limited to these embodiments, even though it has been described that all components constituting the embodiment of the present invention are combined or operated as one. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, although all the components may be implemented as one independent hardware, some or all of the components are selectively combined to perform some or all of the functions of the combined hardware in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable storage medium (Computer Readable Media), read and executed by the computer, thereby implementing the embodiment of the present invention. In addition, it is also possible to implement an operation described as being performed in the frequency domain by modifying it to be performed in the time domain, or by modifying the operation described as being performed in the time domain to be performed in the frequency domain.

Terms such as "include", "compose" or "have" described above mean that the corresponding component may be inherent unless otherwise stated, so it does not exclude other components. It should be construed as being able to further include other components.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

110 short-time fast Fourier transform steps,
115 variance normalization steps;
120 convolution steps,
130 activation function step,
140 pooling steps;
150 short-time fast Fourier inverse transform steps,
155 power compensation stage,
160 Cost Function Steps.

Claims (8)

delete converting the digital input signal into frequency components;
A variance normalization step of size-normalizing the variance of the power value of each frequency component of the signal converted to the frequency-order component on the time axis;
A forward propagation step of passing a normalized signal through a convolutional neural network having a plurality of hidden layers that processes and outputs an input using a weight and a bias value;
A power compensation step of compensating for the power value of the signal subtracted in the dispersion normalization step for the signal that has passed through the forward propagation step;
converting the frequency component signal into a digital output signal;
A cost function step of calculating a cost function from the digital output signal and updating the weight and bias to minimize the cost function
is provided,
In the variance normalization step, the following equation

A signal obtained by normalizing the magnitude of the variance of the power value of each frequency component on the time axis by
epsi is an experimental constant value to prevent division by too small values when the input value is small,
Assuming that the power value of each frequency component of the input signal x(n) is X(k), and E [ X(k) ] is the ensemble average of X(k) for the past M frames,

is,

as,

is 0<

<1 is a smoothing constant, X _av ( k ) is the Long-Term Infinite Impulse Response (IIR) average for the power value of each frequency component,
How to remove noise.
According to claim 2, wherein the power compensation step,
Compensating for the difference in power value between the input signal X ( k ) and NVar_X ( k ), the noise removal method.
3. The method of claim 2,
Each hidden layer determines the value to be passed to the next hidden layer using a causation function,
As the activation function, a smoothing function of the Gaussian Error Linear Unit (GELU) function is used.
How to remove noise.
5. The method of claim 4,
When ρ is a smoothing constant of 0 < ρ < 1, the activation function

How to remove noise using .
6. The method of claim 5,
The output y _t passed through the t-th hidden layer is

Represented by , the noise removal method.
4. The method of claim 2 or 3,
As the cost function, the Euclidean distance between the frequency component of the output signal and the frequency component of the signal in which the correct signal has passed the variance normalization step is used, the noise removal method.
8. The method of claim 7,
M _i denotes the i-th Mel filter bank, and the subscripts t and f denote signals in the waveform domain and spectral domain, respectively,

represents the Euclidean distance,

Wow

each

Wow

Assuming that is a vector passed through the Mel filter bank,

A denoising method that calculates the cost function by

Images