KR20220144117A - Apparatus and method for separating audio sources using denselstm - Google Patents

Abstract

The present specification relates to a device and method for separating an audio source. The method for separating the audio source according to one embodiment of the present specification comprises: a step of input-receiving the learning data comprising the target data without a static noise and the static noise data with the static noise; a step of learning a target separation model by combining a feature map corresponding to each of a plurality of frequency bands regarding to a learning spectrogram based on the learning data; and a step of extracting a target signal using a mask filter and an input signal outputted from the target separation model. Therefore, the present invention is capable of having an improved audio source separation performance.

Description

Audio source separation apparatus and method using DenseLSTM

This specification relates to an audio source separation apparatus and method. More particularly, it relates to an apparatus and method for separating an audio source using DenseLSTM.

When an unwanted signal such as severe noise is mixed in a system using an acoustic signal as a target, system performance is degraded. However, when a person listens to several mixed signals in a real environment at the same time, he or she can concentrate on the desired signal and hear it selectively. This phenomenon is called auditory scene analysis or cocktail party effect.

However, this phenomenon is very difficult to implement with a computer. Recently, as the copyright market grows, the need for technology development for accurately calculating copyright fees increases, and the demand for voice recognition technology is increasing due to the development of deep learning technology.

Accordingly, there is a need for a technology for precisely separating a desired signal from a mixed signal through a computer.

An object of the present specification is to provide an audio source separation apparatus and method having improved audio source separation performance by using time series information by applying an LSTM block.

In addition, an object of the present specification is to provide an audio source separation apparatus and method capable of increasing the reception range by using a dilated block including a time dilated convolution and a frequency dilated convolution connected in parallel.

The objects of the present specification are not limited to the above-mentioned objects, and other objects and advantages of the present specification that are not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present specification. It will also be readily apparent that the objects and advantages of the present specification may be realized by the means and combinations thereof indicated in the claims.

A method of separating an audio source according to an embodiment of the present specification includes receiving training data including target data without noise and noise data including noise as input, and a plurality of frequencies for a training spectrogram based on the training data. Learning a target separation model by combining feature maps corresponding to each band, and extracting a target signal using a mask filter output from the target separation model and an input signal.

In an embodiment of the present specification, the plurality of frequency bands includes a low band, a middle band, a high band, and a full band.

In an embodiment of the present specification, the step of learning the target separation model includes at least one compression block, at least one DDB (Dilated Dense Block) and at least one LSTM block (Long-Short Term Memory block) based on the DenseNet structure. and learning the target separation model using

In one embodiment of the present specification, the DDB includes a dilated block and a dense block, and the dilated block includes FD Convolution (Frequency Dilated Convolution) and TD Convolution (Time Dilated Convolution) arranged in parallel.

In an embodiment of the present specification, the step of learning the target separation model includes first learning spectrogram and converting into a second learning spectrogram, calculating the sizes of the first and second learning spectrograms using absolute values, and the calculated sizes of the first and second learning spectrograms and learning the target separation model by receiving as an input.

In an embodiment of the present specification, extracting the target signal includes converting the input signal into an input spectrogram through a short-time Fourier transform, calculating the size of the input spectrogram that is an absolute value of the input spectrogram, and the input spectrogram. obtaining a mask filter from the target separation model using the magnitude of, multiplying an input spectrogram and the mask filter and applying the phase information of the input spectrogram to generate a corrected spectrogram; and extracting a target signal by performing an inverse short-time Fourier transform (ISTFT, Inverse STFT).

An audio source separation apparatus according to an embodiment of the present specification includes a data input unit that receives training data including target data without noise and noise data including noise, and a plurality of learning spectrograms based on the training data. It includes a model learning unit for learning a target separation model by combining feature maps corresponding to each frequency band, and a signal extraction unit for extracting a target signal using a mask filter output from the target separation model and an input signal.

In an embodiment of the present specification, the plurality of frequency bands includes a low band, a middle band, a high band, and a full band.

In an embodiment of the present specification, the model learning unit uses at least one compression block, at least one DDB (Dilated Dense Block), and at least one LSTM block (Long-Short Term Memory block) based on the DenseNet structure to separate the target model It includes the step of learning.

In one embodiment of the present specification, the DDB includes a dilated block and a dense block, and the dilated block includes FD Convolution (Frequency Dilated Convolution) and TD Convolution (Time Dilated Convolution) arranged in parallel.

In an embodiment of the present specification, the model learning unit converts each of the target data and the mixed data in which the target data and the noise data are mixed into a first learning spectrogram and a second learning spectrogram through a Short Term Fourier Transform (STFT). gram, calculate the size of each of the first learning spectrogram and the second learning spectrogram using absolute values, and input the calculated sizes of the first learning spectrogram and the second learning spectrogram as an input to the Train the target separation model.

In an embodiment of the present specification, the signal extractor converts the input signal into an input spectrogram through a short-time Fourier transform, calculates the size of the input spectrogram that is the absolute value of the input spectrogram, and uses the size of the input spectrogram to obtain a mask filter from the target separation model, multiply the input spectrogram and the mask filter, and apply the phase information of the input spectrogram to generate a corrected spectrogram, and inverse short-time Fourier for the generated modified spectrogram Transform (ISTFT, Inverse STFT) is performed to extract a target signal.

The audio source separation apparatus and method according to an embodiment of the present specification may have improved audio source separation performance by using time series information by applying an LSTM block.

In addition, the audio source separation apparatus and method according to an embodiment of the present specification may increase the reception range by using a dilated block including a time dilated convolution and a frequency dilated convolution connected in parallel.

1 is a block diagram of an apparatus for separating an audio source according to an embodiment of the present specification.
2 is a block diagram illustrating a detailed configuration of an apparatus for separating an audio source.
3 is a block diagram specifically showing the target separation model of the present specification.
4 is a diagram illustrating a sub-model of a full band among target separation models.
5 is a diagram showing the internal structure of the DDB in detail.
6 is a diagram illustrating an internal structure of a composite function.
7 is a diagram illustrating FD convolution and TD convolution.
8 is a diagram illustrating an internal structure of a compression block.
9 is a diagram showing the internal structure of an LSTM block.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of an apparatus for separating an audio source according to an embodiment of the present specification, and FIG. 2 is a block diagram illustrating a detailed configuration of an apparatus for separating an audio source. Hereinafter, it will be described with reference to FIGS. 1 and 2 .

Referring to the drawing, the audio source separation apparatus 100 includes a data input unit 110 , a model learning unit 120 , and a signal extraction unit 130 .

The data input unit 110 receives learning data to be learned through a model learning unit 120 to be described later. Specifically, the training data includes target data 112 that does not include noise and noise data that includes noise (Non-target dataset). Here, the target data may be one and the noise data may be plural. In addition, the learning data may be sound data or voice data, but is not limited thereto, and includes all audio data including sound.

The model learning unit 120 trains the target separation model based on the training data. Specifically, the model learning unit 120 performs a first learning spectrogram for each of the target data 112 and the mixed data 122 in which the target data and the noise data are mixed through a Short Term Fourier Transform (STFT). and a second learning spectrogram. In this case, the spectrogram is a tool for visualizing and understanding sound or waves by combining the characteristics of a waveform and a spectrum. The X axis represents time and the Y axis represents frequency.

Thereafter, the model learning unit 120 calculates the size of each of the first learning spectrogram and the second learning spectrogram by using the absolute value. In addition, the model learning unit 120 trains the target separation model by inputting the calculated sizes of the first learning spectrogram and the second learning spectrogram as an input to perform a target separation model (Target separation model) 124 . create

If the target separation model 124 learned through this process is used, a target signal, which is a signal to be extracted from the input signal, can be extracted as will be described later.

The signal extractor 130 extracts a target signal by using the mask filter output from the target separation model and the input signal. Specifically, the signal extractor 130 converts the input signal into an input spectrogram through a short-time Fourier transform, and calculates the size of the input spectrogram, which is an absolute value of the input spectrogram.

Then, when the signal extractor 130 obtains the mask filter 132 from the target separation model 124 using the calculated size of the input spectrogram, the signal extractor 130 selects the input spectrogram and the mask filter. Multiply and apply the phase information of the input spectrogram to generate a modified spectrogram.

Specifically, the signal extractor 130 uses the size of the input spectrogram as an input of the target separation model 124 , and obtains the mask filter 132 through the target separation model 124 . The mask filter 132 is a filter that removes unwanted sounds, such as white noise or pink noise, from the input signal in order to hide the unwanted sound. If the mast filter 132 is used for the input signal, the target signal can be easily separated.

That is, the signal extractor 130 generates a corrected spectrogram by multiplying the input spectrogram converted from the input signal by the mask filter 132 . At this time, since the phase value of the input signal is required, when the input spectrogram is multiplied by the mask filter 132, the phase information of the input spectrogram is also applied to generate a corrected spectrogram.

The signal extractor 130 may extract a target signal by performing an Inverse Short Term Fourier Transform (ISTFT) on the modified spectrogram thus generated.

3 is a block diagram specifically illustrating a target separation model of the present specification, and FIG. 4 is a diagram illustrating a full band sub-model of the target separation model. Hereinafter, it will be described with reference to FIGS. 3 and 4 .

The model learning unit 120 learns a target separation model by combining a feature map corresponding to each of a plurality of frequency bands with respect to a learning spectrogram based on the training data.

Referring to FIG. 3 , a training spectrogram 123 is used as an input to the target separation model. The Y-axis of the learning spectrogram has a frequency value, and in general, an acoustic signal has a pattern of different characteristics for each frequency band. Accordingly, the model learning unit 120 classifies the frequency band of the training spectrogram 123 into a low band, a middle band, and a high band from a low frequency band. Accordingly, one learning spectrogram 123 may include a plurality of frequency bands, and the plurality of frequency bands may be arranged in parallel. The plurality of frequency bands may include, for example, a low band, a middle band, a high band, and a full band.

As described above, the model learning unit 120 of the present specification classifies one spectrogram 123 into a plurality of frequency bands to arrange sub-models suitable for the characteristics of each frequency band, and perform effective target separation model learning. can do.

In addition, the model learning unit 120 may remove the distortion occurring at the boundary of the frequency band by combining the low band, the middle band, and the high band together with the full band into one tensor.

Each frequency band has a different sub-model, and the model learning unit 120 outputs a feature map corresponding to each frequency band through the sub-model. The model learning unit 120 outputs a mask filter 132 by combining the output feature map into one tensor 125 . In this case, the output mask filter 132 may have a size corresponding to the size of the input learning spectrogram 123 .

On the other hand, each sub-model includes at least one compression block, at least one DDB (Dilated Dense Block), and at least one LSTM block (Long-Short Term Memory block) based on a densely connected convolutional networks (DenseNet) structure. .

Here, the dense block is an image model block used in CNN (Convolutional Neural Network), and the DDB outputs a feature map from the learning spectrogram in the form of an expanded dense block. The detailed internal structure of the DDB will be described later in detail.

The compression block reduces the number of feature maps by compressing the information of the feature maps output from the DDB. In addition, the LSTM block outputs and stores target information using time series information along the time axis of the learning spectrogram.

Referring to FIG. 4 , as an example of a sub model, a full band sub model includes 9 DDBs, 9 compression blocks, and 2 LSTM blocks. That is, the full band sub-model has a structure in which multi-scale features are obtained through down-sampling and up-sampling processes using a large number of dense blocks. The model learning unit 120 obtains, as an output, a mask filter to be multiplied by the input spectrogram when a learning spectrogram is input to the sub-model.

Specifically, a plurality of down-sampling and a plurality of up-sampling occur while the model learning unit 120 obtains the mask filter as an output. Downsampling is performed to obtain compressed information, and as signal separation is a regression task, the output of the network must be equal to the input size, so the upsampling process is absolutely necessary after downsampling. Here, downsampling may use average pooling of a 2x2 kernel, and upsampling may use transposed convolution of a 2x2 kernel.

In an embodiment of the present specification, the model learning unit 120 outputs and stores target information using a total of six LSTM blocks in the downsampling and upsampling processes. Since the LSTM block outputs target information using time series information along the time axis of the learning spectrogram, it is possible to output more detailed target information.

5 is a diagram showing the internal structure of the DDB in detail, FIG. 6 is a diagram showing the internal structure of the composite function, FIG. 7 is a diagram showing FD convolution and TD convolution, and FIG. 8 is a diagram showing the internal structure of a compression block and FIG. 9 is a view showing the internal structure of the LSTM block. Hereinafter, the internal structures of the DDB and LSTM blocks will be described with reference to FIGS. 5 to 9 .

Referring to FIG. 5 , the DDB includes a dilated block and a dense block. Here, the dense block may be composed of a plurality of composite functions (non-linear transformation functions). Dense block is a DenseNet structure, and feature maps of input and output of a plurality of composite functions may be connected. Through this DenseNet structure, the output of each function is accumulated in the feed-forward process, so it can contain a lot of information, and it can solve the vanishing gradient problem in the error back-propagation process. There are advantages.

The DenseNet structure can be expressed by Equation 1 below.

<Equation 1>

은

layer의 출력이자

layer의 입력이고,

는 composite function,

는 0부터

층까지의 출력 특징맵이 연결된 것을 의미한다.here,

silver

the output of the layer

is the input of the layer,

is a composite function,

is from 0

It means that the output feature map up to the layer is connected.

As shown in FIG. 6 , the composite function consists of a continuation of batch normalization (BN), rectified linear unit (ReLU), and convolution of a 3×3 kernel.

On the other hand, the dilated block is to effectively increase the reception range in the spectrogram, and it is placed in front of the dense block to form a DDB together with the dense block. In the spectrogram, the time axis is affected by the firing rate, and the frequency axis changes with a cause independent of the time axis such as pitch and harmonic according to gender.

Therefore, referring again to FIG. 4 , the dilated block has a structure in which BN, ReLU, followed by FD Conv (Frequency Dilated Convolution) and TD Conv (Time Dilated Convolution) are arranged in parallel.

In addition, as shown in (a) of FIG. 7, the FD convolution has a 5 x 3 kernel with a [2, 1] dilation rate and serves to increase the reception range along the frequency axis, and is shown in (b) of FIG. As shown, the TD convolution has a 3 x 5 kernel with a dilation rate of [1, 2] and serves to increase the acceptance range along the time axis.

는 하기의 식 2와 같이 표현될 수 있다.As described above, since k feature maps are output from each layer of the DDB and all are connected, the number of final output feature maps of the DDB is

can be expressed as in Equation 2 below.

<Equation 2>

Here, m ₀ denotes the number of input feature maps, L denotes the number of composite functions, and k denotes the growth rate.

로 표현될 수 있고,

은 0에서 1사이의 값을 갖는다. 이에 따라 compression block을 통과한 최종 출력 특징맵의 개수는

이다.Meanwhile, in the feature map output from the DDB, the information of the feature map is compressed by the compression block, so that the number of feature maps is reduced. As shown in FIG. 8, the compression block is composed of a convolution of BN, ReLU, and 1 x 1 kernel. The compression rate compressed by the compression block is

can be expressed as

has a value between 0 and 1. Accordingly, the number of final output feature maps that have passed through the compression block is

to be.

9 shows a detailed structure of an LSTM block. The LSTM block consists of a 1 x 1 kernel convolution, BLSTM (bi-directional LSTM) and FCNN (fully connected neural network), and the input is connected to the output. Here, since the BLSTM is a Recurrent Neural Network (RNN) structure, the LSTM block operates from a Convolutional Neural Network (CNN) model to a Convolutional Recurrent Neural Network (CRNN) model. In addition, through the BLSTM, the LSTM block outputs the features of the feature map using time series information along the time axis of the learning spectrogram.

In addition, an objective function is required to train the target separation model. The values of the above-described parameters are iteratively learned so that the objective function becomes zero. The objective function may be expressed by Equation 3 below.

<Equation 3>

는 목적 함수, Y는 정답 신호의 스펙트로그램의 크기, X는 학습 스펙트로그램, M은 타겟 분리 모델에서 산출된 마스크 필터,

는 요소별 곱(element-wise multiplication),

은 행렬의 각 요소들의 절대값을 취하여 더한 값을 의미한다. here

is the objective function, Y is the size of the spectrogram of the correct answer signal, X is the training spectrogram, M is the mask filter calculated from the target separation model,

is an element-wise multiplication,

denotes a value added by taking the absolute values of each element of the matrix.

10 is a flowchart of a method for separating an audio source according to an embodiment of the present specification.

Referring to the drawing, the data input unit 110 of the apparatus 100 for separating an audio source receives training data including target data without noise and noise data including noise ( S110 ).

Thereafter, the model learning unit 120 learns a target separation model by combining a feature map corresponding to each of a plurality of frequency bands with respect to the learning spectrogram based on the training data ( S120 ).

In this case, the plurality of frequency bands includes a low band, a middle band, a high band and a full band, and the target separation model is based on the DenseNet structure at least one compression block, at least one DDB (Dilated Dense Block) and at least one It is learned using an LSTM block (Long-Short Term Memory block).

When the learning of the target separation model is completed, the signal extraction unit 130 separates the audio source by extracting the target signal using the mask filter output from the target separation model and the input signal ( S130 ).

As described above, the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed in the present specification. It is obvious that variations can be made. In addition, although the effect of the configuration of the present invention has not been explicitly described and described while describing the embodiment of the present invention, it is natural that the effect predictable by the configuration should also be recognized.

Claims (12)

receiving training data including target data not including noise and noise data including noise;
learning a target separation model by combining a feature map corresponding to each of a plurality of frequency bands with respect to a learning spectrogram based on the training data; and
Extracting a target signal using a mask filter output from the target separation model and an input signal,
How to separate audio sources.
According to claim 1,
The plurality of frequency bands
low band, middle band, high band and full band
How to separate audio sources.
According to claim 1,
The step of learning the target separation model is
Based on the DenseNet structure, at least one compression block, at least one DDB (Dilated Dense Block), and at least one LSTM block (Long-Short Term Memory block) comprising learning a target separation model using
How to separate audio sources.
4. The method of claim 3,
The DDB includes a Dilated block and a Dense block,
The dilated block includes FD Convolution (Frequency Dilated Convolution) and TD Convolution (Time Dilated Convolution) arranged in parallel.
How to separate audio sources.
According to claim 1,
The step of learning the target separation model is
transforming each of the target data and the mixed data in which the target data and the noise data are mixed into a first learning spectrogram and a second learning spectrogram through a Short Term Fourier Transform (STFT);
calculating sizes of the first and second learning spectrograms using absolute values;
and learning the target separation model by inputting the calculated sizes of the first learning spectrogram and the second learning spectrogram as input.
How to separate audio sources.
According to claim 1,
The step of extracting the target signal
converting the input signal into an input spectrogram through a short-time Fourier transform;
calculating a size of an input spectrogram that is an absolute value of the input spectrogram;
obtaining a mask filter from the target separation model using the size of the input spectrogram;
generating a corrected spectrogram by multiplying the input spectrogram and the mask filter and applying phase information of the input spectrogram;
performing an inverse short-time Fourier transform (ISTFT, Inverse STFT) on the generated modified spectrogram to extract a target signal
How to separate audio sources.
a data input unit for receiving training data including target data without noise and noise data including noise;
a model learning unit for learning a target separation model by combining a feature map corresponding to each of a plurality of frequency bands with respect to a learning spectrogram based on the training data; and
A signal extraction unit for extracting a target signal using the mask filter output from the target separation model and the input signal,
Audio source separation device.
8. The method of claim 7,
The plurality of frequency bands
low band, middle band, high band and full band
Audio source separation device.
8. The method of claim 7,
The model learning unit
Based on the DenseNet structure, at least one compression block, at least one DDB (Dilated Dense Block), and at least one LSTM block (Long-Short Term Memory block) comprising learning a target separation model using
Audio source separation device.
10. The method of claim 9,
The DDB includes a Dilated block and a Dense block,
The dilated block includes FD Convolution (Frequency Dilated Convolution) and TD Convolution (Time Dilated Convolution) arranged in parallel.
Audio source separation device.
8. The method of claim 7,
The model learning unit
Each of the target data and the mixed data in which the target data and the noise data are mixed is transformed into a first learning spectrogram and a second learning spectrogram through Short Term Fourier Transform (STFT), and the absolute value is used to calculate the size of each of the first learning spectrogram and the second learning spectrogram, and learn the target separation model by using the calculated sizes of the first learning spectrogram and the second learning spectrogram as input
Audio source separation device.
8. The method of claim 7,
The signal extraction unit
The input signal is converted into an input spectrogram through a short-time Fourier transform, the size of the input spectrogram that is the absolute value of the input spectrogram is calculated, and a mask filter is formed from the target separation model using the size of the input spectrogram. obtaining, multiplying the input spectrogram and the mask filter, and applying the phase information of the input spectrogram to generate a modified spectrogram, and perform an inverse short-time Fourier transform (ISTFT, Inverse STFT) on the generated modified spectrogram to extract the target signal
Audio source separation device.

Images