KR102401959B1 - Joint training method and apparatus for deep neural network-based dereverberation and beamforming for sound event detection in multi-channel environment - Google Patents

Abstract

A method and apparatus for combined learning of reverberation cancellation, beamforming, and acoustic recognition models based on deep neural networks using multi-channel acoustic signals are presented. A method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a computer-implemented multi-channel acoustic signal according to an embodiment deepens a multi-channel acoustic signal input in an environment in which noise and reverberation exist. removing reverberation using a neural network-based WPE (Weighted Predicted Error) reverberation technology; removing noise from the multi-channel sound signal from which the reverberation has been removed using a deep neural network-based MVDR (Minimum Variance Distortionless Response) beamforming technology; and inputting the improved acoustic signal of the single channel from which the noise has been removed into an acoustic recognition model based on a convolutional recurrent neural network, wherein the reverberation cancellation model for removing the reverberation and the beamforming are performed. The beamforming model, and the acoustic recognition model may operate as one combined neural network.

Description

JOINT TRAINING METHOD AND APPARATUS FOR DEEP NEURAL NETWORK-BASED DEREVERBERATION AND BEAMFORMING FOR SOUND EVENT DETECTION IN MULTI-CHANNEL ENVIRONMENT}

The following embodiments relate to a method and apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal.

Acoustic recognition is a technique for finding the start position (onset) and end position (offset) of a sound generated from a given microphone input signal and classifying the sound type (eg, cough sound, keyboard sound, speech sound, ringtone, etc.).

In (Non-Patent Document 1), various methods for systematically combining the deep neural network-based reverberation cancellation and beamforming module were studied, and speech recognition experiments were conducted in an environment where noise and reverberation were present. In (Non-Patent Document 2), a method for learning by combining a deep neural network-based reverberation cancellation and beamforming module with an x-vector-based speaker verification module was proposed. (Non-Patent Document 3) proposes a method of recognizing a sound and estimating the direction of the sound using a convolutional recurrent neural network.

In the conventional acoustic recognition technology, research has been mainly conducted to improve the performance of the neural network model itself (eg, convolutional neural network) or to simultaneously execute other functions (eg, direction estimation). Unlike voice signals used in similar fields, such as voice recognition or speaker verification, sound signals have very diverse characteristics. There weren't many attempts.

As described above, the conventional technology simply uses a convolutional neural network-based acoustic recognition model, and thus is not robust in noise and reverberation environments, and does not utilize the advantage of using a multi-channel acoustic signal.

“Integrating neural network based beamforming and weighted prediction error dereverberation,” L. Drude, C. Boeddeker, J. Heymann, R. Haeb-Umbach, K. Kinoshita, M. Delcroix, and T. Nakatani, University of Paderborn & NTT, Interspeech, 2018. “Joint optimization of neural acoustic beamforming and dereverberation with x-vectors for robust speaker verification,” J.-Y. Yang and J.-H. Chang, Hanyang University, Interspeech, 2019. “Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, Tampere University of Technology, IEEE Journal of Selected Topics in Signal Processing, 2018.

The embodiments describe a method and apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal, and more specifically, using a multi-channel acoustic signal in an environment in which noise and reverberation exist. This provides a technology to construct a robust acoustic perception system.

The embodiments provide an environment in which noise and reverberation exist by combining and learning a deep neural network-based reverberation, beamforming, and sound event detection model using a multi-channel sound signal. An object of the present invention is to provide a method and apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal, which can obtain improved performance in a multi-channel acoustic signal.

A method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a computer-implemented multi-channel acoustic signal according to an embodiment deepens a multi-channel acoustic signal input in an environment in which noise and reverberation exist. removing reverberation using a neural network-based WPE (Weighted Predicted Error) reverberation technology; removing noise from the multi-channel sound signal from which the reverberation has been removed using a deep neural network-based MVDR (Minimum Variance Distortionless Response) beamforming technology; and inputting the improved acoustic signal of the single channel from which the noise has been removed into an acoustic recognition model based on a convolutional recurrent neural network, wherein the reverberation cancellation model for removing the reverberation and the beamforming are performed. The beamforming model, and the acoustic recognition model may operate as one combined neural network.

The method may further include performing joint learning of the reverberation cancellation model, the beamforming model, and the acoustic perception model based on an output of the acoustic recognition model and an error calculated by a target.

The step of removing the reverberation from the multi-channel sound signal by using the deep neural network-based WPE reverberation removal technique includes converting the multi-channel sound signal into a value on the frequency axis from the time axis through short-time Fourier transform (STFT). ; and improving the STFT coefficient of the transformed frequency axis by the spectral mask estimated using the WPE reverberation cancellation technique based on the deep neural network.

In the step of removing noise from the multi-channel sound signal from which the reverberation has been removed by using the deep neural network-based MVDR beamforming technology, the STFT coefficient of the multi-channel sound signal from which the reverberation has been removed is determined by the deep neural network-based MVDR beamforming technology. It can be improved by two kinds of spectral masks estimated using

The step of inputting the improved acoustic signal of the single channel from which the noise has been removed into the acoustic recognition model based on the convolutional recurrent neural network may include: It may be converted into a filter bank (mel filter bank) and input to a sound recognition model based on a convolutional recurrent neural network.

The step of inputting the improved acoustic signal of the single channel from which the noise has been removed into the acoustic recognition model based on the convolutional recurrent neural network includes: In the convolution module, the method of performing convolution by considering only the frequency axis and the method of performing convolution by considering the frequency axis and the time axis sequentially can be connected in parallel and then combined.

The step of combining the reverberation cancellation model, the beamforming model, and the acoustic perception model by the output of the acoustic recognition model and the error calculated by the target is to set the loss function of the neural network to the focal loss to increase the amount of data. imbalance can be alleviated.

A deep neural network-based reverberation removal, beamforming, and acoustic recognition model combined learning apparatus using a multi-channel acoustic signal according to another embodiment is a WPE based on a deep neural network for a multi-channel acoustic signal input in an environment in which noise and reverberation exist (Weighted Predicted Error) A reverberation cancellation model that removes reverberation using reverberation cancellation technology; a beamforming model that removes noise from the multi-channel sound signal from which the reverberation has been removed by using a MVDR (Minimum Variance Distortionless Response) beamforming technology based on a deep neural network; and an acoustic recognition model inputting the improved acoustic signal of the single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic perception model, wherein the reverberation cancellation model, the beamforming model, and the The acoustic perception model can operate as a single combined neural network.

The reverberation cancellation model, the beamforming model, and the acoustic recognition model may further include a learning unit that performs combined learning based on an error calculated by an output of the acoustic recognition model and a target.

The reverberation cancellation model converts the multi-channel acoustic signal from a time axis to a frequency axis value through short-time Fourier transform (STFT), and the STFT coefficient of the converted frequency axis uses the deep neural network-based WPE reverberation cancellation technology. Therefore, it can be improved by the estimated spectral mask.

The acoustic recognition model is based on a convolutional recurrent neural network in which the STFT coefficients of the improved acoustic signal of the single channel from which the noise has been removed are converted into a log scale mel filter bank. can be input as an acoustic perception model of

According to embodiments, noise and reverberation are present by combining and learning a deep neural network-based reverberation, beamforming, and sound event detection model using a multi-channel sound signal. It is possible to provide a method and apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal, which can acquire improved performance in an environment where a multi-channel acoustic signal is used.

1 is a diagram for describing a method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an embodiment.
2 is a flowchart illustrating a method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an exemplary embodiment.
3 is a block diagram illustrating an apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an exemplary embodiment.
4 is a diagram for explaining a convolutional recurrent neural network model according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present embodiment is not limited by the embodiments described below. In addition, various embodiments are provided in order to more completely explain the present embodiment to those of ordinary skill in the art. The shapes and sizes of elements in the drawings may be exaggerated for clearer description.

In the following embodiments, noise and reverberation exist by combining and learning a deep neural network-based reverberation, beamforming, and sound event detection model using a multi-channel sound signal. It aims to obtain improved performance in the environment where

1 is a diagram for describing a method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an embodiment.

Referring to FIG. 1 , a combined learning method of the deep neural network-based reverberation cancellation model 110 , the beamforming model 120 , and the acoustic recognition model 130 using a multi-channel acoustic signal is shown. First, the reverberation cancellation model 110 . ) converts a multi-channel sound signal input in an environment where noise and reverberation exist into values on the frequency axis from the time axis through short-time Fourier transform (STFT). The STFT coefficient of the transformed frequency axis is improved by the spectral mask estimated using the WPE (Weighted Predicted Error) reverberation technique based on the deep neural network.

Next, the STFT coefficients of the multi-channel enhanced acoustic signal from which the reverberation has been removed from the beamforming model 120 are two types of spectral masks (source masks) estimated using the MVDR (Minimum Variance Distortionless Response) beamforming technique based on the deep neural network. and noise mask).

As a result, the enhanced acoustic signal STFT coefficients of the single channel from which the noise has been removed are converted to a log-scale mel filter bank, and a convolutional recurrent neural network-based acoustic perception model ( 130) is entered.

The three models of the reverberation cancellation model 110, the beamforming model 120, and the acoustic recognition model 130 operate as one combined neural network, and are calculated by the output of the acoustic recognition model 130 and the target during training. It is jointly learned by error. In this case, the focal loss is used for error calculation in order to alleviate the phenomenon that the learning ability of the model is deteriorated due to the amount of unbalanced acoustic data.

As a result of the experiment, the deep neural network-based reverberation cancellation model 110, beamforming model 120, and acoustic perception model 130 jointly trained using focal loss as a loss function improved acoustic perception in an environment where noise and reverberation exist. performance was shown.

2 is a flowchart illustrating a method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an exemplary embodiment.

Referring to FIG. 2 , a method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a computer-implemented multi-channel acoustic signal according to an embodiment is an input method in an environment in which noise and reverberation are present. Step of removing reverberation using WPE (Weighted Predicted Error) reverberation cancellation technology based on deep neural network for multi-channel acoustic signal (S110), deep neural network-based MVDR (Minimum Variance Distortionless Response) for multi-channel acoustic signal The step of removing noise using beamforming technology (S120), and the step of inputting the improved acoustic signal of a single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic recognition model (S130) Including, the reverberation cancellation model for removing reverberation, the beamforming model for performing beamforming, and the acoustic recognition model may operate as one combined neural network.

The reverberation cancellation model, the beamforming model, and the acoustic recognition model may further include a step ( S140 ) of joint learning based on an error calculated by an output of the acoustic recognition model and a target.

Hereinafter, each step of a method for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an embodiment will be described.

The deep neural network-based reverberation cancellation, beamforming, and combined learning method of an acoustic recognition model using a multi-channel acoustic signal according to an embodiment includes the deep neural network-based reverberation cancellation, beamforming and An example of a joint learning apparatus for a sound recognition model may be described in more detail.

3 is a block diagram illustrating an apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an exemplary embodiment.

Referring to FIG. 3 , an apparatus 300 for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an embodiment includes a reverberation cancellation model 310 and a beamforming model 320 . ) and an acoustic perception model 330 . According to an embodiment, the combined learning apparatus 300 may further include a learning unit 340 .

In step S110, the reverberation removal model 310 may remove the reverberation using a WPE (Weighted Predicted Error) reverberation removal technique based on a deep neural network based on a multi-channel acoustic signal input in an environment in which noise and reverberation exist. . Here, the reverberation cancellation model 310 converts a multi-channel acoustic signal from a time axis to a frequency axis value through short-time Fourier transform (STFT), and then, the STFT coefficient of the converted frequency axis is a WPE reverberation cancellation technology based on a deep neural network. It can be improved by the spectral mask estimated using

In step S120 , the beamforming model 320 may remove noise from the multi-channel acoustic signal from which the reverberation has been removed by using a deep neural network-based Minimum Variance Distortionless Response (MVDR) beamforming technique. Here, in the beamforming model 320 , the STFT coefficient of the multi-channel acoustic signal from which the reverberation is removed may be improved by two types of spectral masks estimated using the deep neural network-based MVDR beamforming technique.

In operation S130 , the acoustic recognition model 330 may input an improved acoustic signal of a single channel from which noise has been removed into a convolutional recurrent neural network-based acoustic perception model. The reverberation cancellation model 310 , the beamforming model 320 , and the acoustic recognition model 330 may operate as one combined neural network. Here, the STFT coefficients of the improved acoustic signal of the single channel from which the noise has been removed in the acoustic recognition model 330 are converted into a log scale mel filter bank, and a convolutional recurrent neural network ) based on the acoustic recognition model 330 .

Acoustic perception model 330 is a convolutional recurrent neural network In the convolution module, the method of performing convolution by considering only the frequency axis and the method of performing convolution by considering the frequency axis and the time axis sequentially can be connected in parallel and then combined.

In step S140 , the learning unit 340 performs the reverberation cancellation model 310 , the beamforming model 320 , and the acoustic perception model 330 according to the output of the acoustic recognition model 330 and the error calculated by the target. It can be combined learning. In this case, the learning unit 340 may set the loss function of the neural network as focal loss to alleviate the imbalance in the amount of data.

Hereinafter, a method and apparatus for combined learning of a deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using a multi-channel acoustic signal according to an embodiment will be described in more detail as an example.

In this embodiment, a combined reverberation cancellation, beamforming, and acoustic recognition module using a multi-channel acoustic signal was studied as one loss function based on focal loss with reference to the conventional techniques in the field of speech recognition and speaker verification. At this time, unlike the prior art, it showed additional performance improvement by modifying the detailed module of the convolutional neural network. Also, unlike the conventional technique for directly estimating the STFT coefficient when removing deep neural network-based reverberation, it is between 0 and 1 By estimating a spectral mask with a value of , smooth learning was enabled.

Example

WPE reverberation cancellation based on signal models and deep neural networks

Assuming that acoustic signals are collected through D microphones in an environment where noise and reverberation exist, the acoustic signal input to the microphone can be expressed as the sum of the acoustic signal with reverberation and noise, and the equation is as follows.

[Equation 1]

(1)

(One)

In this case, x and n are the STFT (short-time Fourier transform) coefficients of the acoustic signal and the noise signal degraded by reverberation, and t, f, and d are the time frame index, frequency-bin index, and microphone index, respectively. , y denotes a microphone input signal obtained by adding an acoustic signal and a noise signal. In addition, the superscripts (early) and (late) mean an early reflection signal and a late reflection signal, respectively, and each signal indicates that the original signal is deteriorated by the echo characteristic of the space. It is assumed that possible deterioration is possible, and in the latter case, the time interval is relatively long, so it is considered to be greatly deteriorated and needs to be removed. At this time, a signal to be obtained through reverberation cancellation is calculated based on the linear prediction filter as follows.

[Equation 2]

(2)

(2)

는 선형 예측 기법을 통해 추정한 early reflection 신호의 추정값을,

는 선형 예측 알고리즘의 delay를,

와 G는 각각 마이크 입력 신호의 STFT 계수와 선형 예측 필터 계수를 현재 프레임 t를 기준으로 과거

번째 프레임부터 과거

번째 프레임까지 쌓아 놓은 stacked representation이다. 고전적인 WPE 잔향 제거 기술은 입력 신호의 late reflection 성분을 추정하기 위한 선형 예측 필터를 아래와 같은 반복적인(iterative) 방식으로 추정한다.here,

is the estimated value of the early reflection signal estimated through the linear prediction technique,

is the delay of the linear prediction algorithm,

and G are the STFT coefficients and linear prediction filter coefficients of the microphone input signal, respectively, in the past with respect to the current frame t.

from the second frame to the past

It is a stacked representation stacked up to the second frame. The classical WPE reverberation cancellation technique estimates the linear prediction filter for estimating the late reflection component of the input signal in the following iterative manner.

[Equation 3]

(3)

(3)

[Equation 4]

(4)

(4)

[Equation 5]

(5)

(5)

[Equation 6]

(6)

(6)

는 추정한 early reflection 신호의 시간-주파수 축에서의 파워를, R는 선형 예측 필터의 차수(order)를 의미한다. At this time,

is the estimated power on the time-frequency axis of the early reflection signal, and R is the order of the linear prediction filter.

의 파워를 입력 받아 late reflection 성분이 제거된

성분의 파워를 추정하도록 학습된다. 이는 음향 성분과 잡음 성분 모두에서 late reverberation 제거하는 것을 목적으로 심화 신경망을 학습하는 방법이며, 본 실시예에서는 학습을 더욱 원활하게 하기 위하여 값의 범위가 상대적으로 넓은 STFT 계수 대신에 0~1 사이의 값을 가지는 spectral mask를 추정하여 STFT 계수에 곱하는 방법으로 향상된 STFT 계수를 추정한다. Mask의 값이 0~1 사이가 되도록 하기 위해서 mask 추정을 위한 심화 신경망의 출력 레이어에서는 ReLU(Rectified Linear Unit) 함수를 사용한 뒤 출력값이 최대 1이 되도록 상한값을 설정해 준다. 심화 신경망의 학습이 끝나면 심화 신경망을 이용하여 각 마이크로폰 채널별로 early reflection 신호의 파워 추정값을 계산한 뒤, 모든 채널에 대해 평균을 취하여 수학식 3의 좌변을 대신할 수 있는 파워 추정값을 계산하고, 수학식 4 내지 수학식 6의 과정을 거쳐 최종적으로는 수학식 2를 통해 early reflection 신호의 STFT 계수를 추정할 수 있다.On the other hand, WPE reverberation cancellation using deep neural network replaces a part of the classical WPE algorithm with logic using deep neural network. The part for estimating the power of the early reflection signal in Equation 3 above is replaced by a deep neural network, and in this case, the deep neural network is a microphone input signal

received the power of , and the late reflection component is removed.

It is learned to estimate the power of a component. This is a method of learning a deep neural network for the purpose of removing late reverberation from both the acoustic component and the noise component. An improved STFT coefficient is estimated by estimating a spectral mask having a value and multiplying the STFT coefficient. In order to ensure that the mask value is between 0 and 1, the output layer of the deep neural network for mask estimation uses the Rectified Linear Unit (ReLU) function and sets the upper limit so that the output value is at most 1. After learning of the deep neural network is completed, the power estimate of the early reflection signal is calculated for each microphone channel using the deep neural network, and then the power estimate that can be substituted for the left side of Equation 3 is calculated by taking the average of all channels, After the process of Equation 4 to Equation 6, the STFT coefficient of the early reflection signal may be estimated through Equation 2 finally.

MVDR beamforming based on deep neural network

The classical MVDR beamforming technique aims to minimize the power of residual noise remaining in the output signal while the output signal to which the beamforming is applied has no distortion. By solving such a minimization problem, the following MVDR gain can be obtained.

[Equation 7]

(7)

(7)

와

는 각각 음향 성분과 잡음 성분의 power spectral density(PSD) 행렬을 나타내며,

는 출력 채널 선택을 위한 one-hot 벡터를 나타낸다. MVDR 빔포머의 출력은 MVDR 이득(gain)과 입력 신호를 곱하여 얻을 수 있으며, 그 식은 아래와 같이 표현될 수 있다.here,

Wow

represents the power spectral density (PSD) matrix of the acoustic component and the noise component, respectively,

denotes a one-hot vector for output channel selection. The output of the MVDR beamformer can be obtained by multiplying the MVDR gain by the input signal, and the equation can be expressed as follows.

[Equation 8]

(8)

(8)

은 시간-주파수 축에서의 빔포머 출력 음향 신호를 의미한다. here,

denotes a beamformer output sound signal on the time-frequency axis.

On the other hand, MVDR beamformer using deep neural network replaces a part of the classical beamformer algorithm with logic using deep neural network. In this embodiment, a spectral mask-based MVDR beamformer is used, and the beamformer estimates the spectral mask of the time-frequency axis for the voice component and the noise component using a deep neural network, and then uses the estimated mask to estimate the voice component and A PSD matrix for the noise component is calculated. At this time, since mask estimation is performed independently for each microphone channel, the same number of masks as the number of microphones is calculated for the voice component and the noise component, respectively. In order to ensure that the mask value is between 0 and 1, the output layer of the deep neural network for mask estimation uses a sigmoid function to design the model so that the output value appears between 0 and 1. Since the spectral mask corresponding to the acoustic component and the noise component is estimated respectively, the deep neural network for mask estimation is configured separately, and when the deep neural network is trained, the PSD matrix of the acoustic and noise component signals is estimated using the following equation can do.

[Equation 9]

(9)

(9)

은 심화신경망을 통해 각 마이크로폰 채널별로 얻은 mask의 추정값을 모든 채널에 대해 평균을 취한 average mask이다. At this time,

is the average mask obtained by averaging the mask estimates obtained for each microphone channel through the deep neural network for all channels.

The MVDR gain can be obtained by substituting the PSD matrix obtained in this way into Equation (7). The deep neural network for mask estimation is trained to receive the log-scale power spectra (LPS) calculated from the input signal and output the mask.

Acoustic Recognition based on Convolutional Recurrent Neural Network

After the deep neural network-based WPE reverberation cancellation and MVDR beamforming are sequentially completed, the improved STFT coefficients of the output single channel are input to the acoustic recognition model based on the convolutional recurrent neural network. In this case, a log scale mel filter bank is used as an input to the neural network, similar to the prior art, and the start and end positions of the sound are found using a convolutional recurrent neural network model, and the corresponding Estimate the type of sound. Here, the mel filter bank is an energy value calculated by applying a mel filter to the magnitude value of the STFT coefficient, and the mel filter is a non-linear imitation of a human ear. it's a filter

3 is a diagram for explaining a convolutional recurrent neural network model according to an embodiment.

Referring to FIG. 3 , a comparison of a convolutional recurrent neural network model according to the prior art and an embodiment is shown. In this embodiment, the performance is improved by modifying the convolution part in the convolutional recurrent neural network used in the prior art. In the existing convolution part, a method of extracting features of the time-frequency axis using multiple layers of 3x3 convolution filters was used, but in this embodiment, the method of convolution by considering only the frequency axis and the frequency axis and the time axis are considered sequentially Thus, by using the method of concatenating the convolution method in parallel and then combining it, it is possible to extract features more effectively and obtain improved performance.

Deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model combined learning method

Deep neural network-based reverberation cancellation, beamforming, and acoustic recognition models are all composed of only differentiable operations, so they are connected to a single neural network and combine learning is possible. In this case, for error backpropagation of neural network learning, focal loss may be used as a loss function. Acoustic data varies greatly from short to long sounds due to the characteristics of sound, and it can be said that the amount of data is very diverse as it has not been long since it has been systematically collected and managed for research like voice. In general, in neural network learning, efficient learning is possible only when the amount of data in each class is the same. In acoustic recognition, there have always been difficulties in neural network learning due to an imbalance in the amount of data, and various methods have been proposed to alleviate it. In this embodiment, the focal loss proposed in the imaging field was introduced to the acoustic recognition field to automatically compensate for the imbalance in the amount of data to improve performance.

[Equation 10]

(10)

(10)

[Equation 11]

(11)

(11)

는 focal loss의 focusing 파라미터를 나타낸다. Here, gt represents the correct answer, that is, 1 means that the sound is present at the corresponding time, and 0 means that there is no sound at the corresponding time. p is the acoustic presence probability estimated by the neural network model, and

Is It represents the focusing parameter of focal loss.

에 의하여 에러가 강제로 작게 반영되게 된다.Focal loss is a loss function that works by penalizing data for which classification is very easy. In general, in the case of a class with a large amount of data, classification becomes very easy, and at this time, the value of the acoustic presence probability p estimated by the neural network model becomes very large. According to Equation 10, if the value of p is very large, the focusing parameter

by forcing the error to be reflected small.

On the other hand, since both the WPE reverberation removal and MVDR beamforming algorithms in joint learning consist of operations that process STFT coefficients having complex values, it is necessary to separately calculate the real and imaginary parts of the complex number to actually implement joint learning. there is In particular, the inverse operation of the complex matrix shown in Equations 6 and 7 can be solved through the following formula.

[Equation 12]

(12)

(12)

[Equation 13]

(13)

(13)

는 복소수 행렬이며,

와

는 각각 역행렬이 존재하는 실수 행렬이다.here,

is a complex matrix,

Wow

is a real matrix with inverse matrices, respectively.

Experiment result

For the evaluation of this embodiment, the TAU Spatial Sound Events 2019-Microphone Array public dataset consisting of sound signals recorded with multi-channel microphones in an indoor environment where noise and reverberation exist was used. This dataset was recorded with a 4-channel microphone with a tetrahedral structure and consists of 500 files. Each file is 1 minute long, the sampling frequency is 48,000 Hz, and the signal-to-noise ratio is 30 dB, so the noise is not strong. For evaluation in a noisy environment, noise recorded indoors was added to a signal-to-noise ratio of 10 dB and used for additional evaluation. A total of 11 types of sound are considered in this dataset: throat clearing, coughing, knocking, door closing, drawer opening and closing, laughter, keyboard sound, placing a key on a table, turning a bookshelf, It is a phone ring sound and a speech sound, and 20 files were used for each type. Since the learning effect of neural network learning varies greatly depending on the amount of data, two data amplification methods, pitch shifting and block mixing, were used to improve the effect of neural network learning.

Acoustic recognition algorithms are generally evaluated with two indicators, F-score and error rate (ER). There may be various methods of calculating the index, but in this embodiment, the calculation was performed by shifting segments in units of 1 second without overlapping parts and comparing the results with the correct answers. F-score can be defined as follows.

[Equation 14]

(14)

(14)

At this time, K is the total number of segments, TP is the number of true positives (when both the correct answer and the result are 1), FP is the number of false positives (when the correct answer is 0 but the result is 1), and FN is Indicates the number of false negatives (when the correct answer is 1 but the result is 0).

Next, ER may be defined as follows.

[Equation 15]

(15)

(15)

At this time, N(k) represents the total number of things with the correct answer being 1, and S(k), D(k), and I(k) are abbreviations of substitution, deletion, and insertion, respectively, and may be defined as follows.

[Equation 16]

(16)

(16)

[Equation 17]

(17)

(17)

[Equation 18]

(18)

(18)

In the most ideal condition, the F-score becomes 1 and ER becomes 0.

Table 1 shows the F-score and ER results using the TAU Spatial Sound Events 2019-Microphone Array dataset according to an embodiment.

[Table 1]

Table 2 shows F-score and ER results using a dataset in which room noise is added by 10 dB to the TAU Spatial Sound Events 2019-Microphone Array dataset according to an embodiment.

[Table 2]

As a result of the experiment, it can be seen that the F-score was improved by 1-2% compared to the prior art due to the deformation of the convolutional part used in the acoustic recognition model, and the ER was at a similar level. In addition, there was a significant performance improvement by combining learning with deep neural network-based WPE reverberation cancellation and MVDR beamforming. In particular, the performance was the best when three modules were combined learning. In particular, in an environment with a signal-to-noise ratio of 10 dB where noise and reverberation are relatively strong, the F-score is improved by 10% and ER by 0.2 level compared to the prior art. In addition, when focal loss is used as the loss function of the neural network during joint learning, the performance is somewhat improved compared to when the conventional cross entropy is used as the loss function.

In the case of inputting and processing sound through a multi-channel microphone in an environment where noise and reverberation exist by using the deep neural network-based WPE reverberation cancellation, MVDR beamforming, and combined learning technique of the acoustic recognition model according to this embodiment, especially indoors Improved performance can be expected as it is applied to acoustic recognition-based applications that can be applied to smartphones, artificial intelligence speakers, robots, and CCTVs in the environment.

As described above, embodiments relate to a method of constructing a robust acoustic recognition system using a multi-channel acoustic signal in an environment in which noise and reverberation exist. According to embodiments, the performance of sound recognition can be improved by being applied to smartphones, artificial intelligence speakers, robots, CCTVs, etc. that require sound recognition in an environment in which noise and reverberation exist, and in order to reflect the microphone characteristics, the corresponding microphone Optimized performance can be obtained by using the collected acoustic signal for learning or by using an adaptation technique that adapts to microphone characteristics.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

A method for combined learning of deep neural network-based reverberation cancellation, beamforming, and acoustic recognition models using computer-implemented multi-channel acoustic signals, the method comprising:
removing reverberation from a multi-channel acoustic signal input in an environment in which noise and reverberation exist using a WPE (Weighted Predicted Error) reverberation removal technology based on a deep neural network;
removing noise from the multi-channel sound signal from which the reverberation has been removed using a deep neural network-based MVDR (Minimum Variance Distortionless Response) beamforming technology; and
inputting the improved acoustic signal of the single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic perception model;
including,
The reverberation cancellation model for removing the reverberation, the beamforming model for performing the beamforming, and the acoustic recognition model operate as one combined neural network,
Combining learning of the reverberation cancellation model, the beamforming model, and the acoustic recognition model based on an output of the acoustic recognition model and an error calculated by a target;
Further comprising, a combined learning method. delete A method for combined learning of deep neural network-based reverberation cancellation, beamforming, and acoustic recognition models using computer-implemented multi-channel acoustic signals, the method comprising:
removing reverberation from a multi-channel acoustic signal input in an environment in which noise and reverberation exist using a WPE (Weighted Predicted Error) reverberation removal technology based on a deep neural network;
removing noise from the multi-channel sound signal from which the reverberation has been removed using MVDR (Minimum Variance Distortionless Response) beamforming technology based on a deep neural network; and
inputting the improved acoustic signal of the single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic perception model;
including,
The reverberation cancellation model for removing the reverberation, the beamforming model for performing the beamforming, and the acoustic recognition model operate as one combined neural network,
The step of removing the reverberation of the multi-channel acoustic signal by using the deep neural network-based WPE reverberation removal technology,
converting the multi-channel sound signal into a value on the frequency axis from the time axis through short-time Fourier transform (STFT); and
The STFT coefficient of the transformed frequency axis is improved by the spectral mask estimated using the WPE reverberation cancellation technique based on the deep neural network.
Containing, a combined learning method. A method for combined learning of deep neural network-based reverberation cancellation, beamforming, and acoustic recognition models using computer-implemented multi-channel acoustic signals, the method comprising:
removing reverberation from a multi-channel acoustic signal input in an environment in which noise and reverberation exist using a WPE (Weighted Predicted Error) reverberation removal technology based on a deep neural network;
removing noise from the multi-channel sound signal from which the reverberation has been removed using a deep neural network-based MVDR (Minimum Variance Distortionless Response) beamforming technology; and
inputting the improved acoustic signal of the single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic perception model;
including,
The reverberation cancellation model for removing the reverberation, the beamforming model for performing the beamforming, and the acoustic recognition model operate as one combined neural network,
The step of removing noise from the multi-channel sound signal from which the reverberation has been removed using MVDR beamforming technology based on a deep neural network includes:
The STFT coefficient of the multi-channel sound signal from which the reverberation has been removed is improved by two types of spectral masks estimated using the MVDR beamforming technique based on the deep neural network.
characterized in that, a combined learning method. According to claim 1,
The step of inputting the improved acoustic signal of the single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic recognition model comprises:
The STFT coefficient of the improved acoustic signal of the single channel from which the noise has been removed is converted into a log scale mel filter bank and input to a convolutional recurrent neural network-based acoustic perception model. to be
characterized in that, a combined learning method. According to claim 1,
The step of inputting the improved acoustic signal of the single channel from which the noise has been removed into a convolutional recurrent neural network-based acoustic recognition model comprises:
of the convolutional recurrent neural network. In the convolution module, the method of convolution by considering only the frequency axis and the method of performing convolution by considering the frequency axis and the time axis sequentially are connected in parallel and then combined.
characterized in that, a combined learning method. According to claim 1,
The step of combining and learning the reverberation cancellation model, the beamforming model, and the acoustic perception model by an error calculated by an output of the acoustic recognition model and a target,
Setting the loss function of the neural network as the focal loss to alleviate the imbalance in the amount of data
characterized in that, a combined learning method. In the combined learning apparatus of the deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using multi-channel acoustic signals,
A reverberation cancellation model that removes reverberation using WPE (Weighted Predicted Error) reverberation technology based on deep neural network for multi-channel acoustic signals input in an environment where noise and reverberation exist;
a beamforming model that removes noise from the multi-channel sound signal from which the reverberation has been removed by using a MVDR (Minimum Variance Distortionless Response) beamforming technology based on a deep neural network; and
Acoustic perception model in which the improved acoustic signal of the single channel from which the noise has been removed is input into a convolutional recurrent neural network-based acoustic perception model
including,
The reverberation cancellation model, the beamforming model, and the acoustic perception model operate as one combined neural network,
A learning unit that combines the reverberation cancellation model, the beamforming model, and the acoustic recognition model based on an output of the acoustic recognition model and an error calculated by a target
Further comprising, a combined learning device. delete In the combined learning apparatus of the deep neural network-based reverberation cancellation, beamforming, and acoustic recognition model using multi-channel acoustic signals,
A reverberation cancellation model that removes reverberation using WPE (Weighted Predicted Error) reverberation technology based on deep neural network for multi-channel acoustic signals input in an environment where noise and reverberation exist;
a beamforming model that removes noise from the multi-channel sound signal from which the reverberation has been removed by using a MVDR (Minimum Variance Distortionless Response) beamforming technique based on a deep neural network; and
Acoustic perception model in which the improved acoustic signal of the single channel from which the noise has been removed is input into a convolutional recurrent neural network-based acoustic perception model
including,
The reverberation cancellation model, the beamforming model, and the acoustic perception model operate as one combined neural network,
A learning unit that combines the reverberation cancellation model, the beamforming model, and the acoustic perception model based on an output of the acoustic recognition model and an error calculated by a target
further comprising,
The reverberation removal model is
The multi-channel sound signal is transformed from the time axis to the frequency axis value through short-time Fourier transform (STFT), and the STFT coefficient of the converted frequency axis is applied to the spectral mask estimated using the WPE reverberation cancellation technique based on the deep neural network. to be improved by
characterized in that, a combined learning device.

Images