KR102579470B1 - Method And Apparatus for Processing Audio Signal - Google Patents

Abstract

A method and device for processing audio signals are disclosed. An audio signal processing method according to an embodiment of the present invention includes obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; calculating the difference between the initial audio signal and the final audio signal in the time domain; converting the initial audio signal and the final audio signal into a mel spectrum; calculating the difference between the mel spectrum of the initial audio signal and the final audio signal in the frequency domain; training the plurality of neural network models based on results calculated in the time domain and frequency domain; and generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.

Description

{Method And Apparatus for Processing Audio Signal}

The present invention relates to a method and device for processing audio signals, and more specifically, in training a neural network model for encoding and decoding audio signals, by calculating a loss function for training the neural network model using a psychoacoustic model. It relates to a method and device for processing signals.

In the process of encoding an audio signal, decoding the encoded audio signal, and restoring the audio signal, a difference occurs between the restored audio signal and the initially input audio signal due to loss of the audio signal.

In order to reduce the loss of these audio signals, research is being actively conducted on Neural Audio Coding, which applies a neural network model in deep learning, one of the artificial intelligence technologies, to the encoding and decoding of audio signals. there is. However, technology that takes psychoacoustic factors into account when learning a neural network model to minimize audio signal loss is required.

The present invention provides a method and device for minimizing audio signal loss by using a psychoacoustic model in the learning process of the neural network model when processing an audio signal using a neural network model that encodes and decodes the audio signal.

In addition, the present invention provides a method and device for improving the quality of a restored audio signal by learning a neural network model to minimize noise generated in the encoding process of the audio signal during the learning process of the neural network model that performs encoding and decoding of the audio signal. to provide.

An audio signal processing method according to an embodiment of the present invention includes obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; calculating the difference between the initial audio signal and the final audio signal in the time domain; converting the initial audio signal and the final audio signal into a mel spectrum; calculating the difference between the mel spectrum of the initial audio signal and the final audio signal in the frequency domain; training the plurality of neural network models based on results calculated in the time domain and frequency domain; and generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.

In the step of training the neural network models, parameters included in the neural network model may be updated so that the sum of the results calculated in the time domain and the results calculated in the frequency domain is minimized.

The plurality of neural networks are in a continuous relationship, and the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal. You can.

The final audio signal may be an audio signal resulting from adding the output audio signals of each of the plurality of neural networks.

An audio signal processing method according to an embodiment of the present invention includes obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; Obtaining a power spectral density and masking threshold for the initial audio signal through a psychoacoustic model; determining a weight according to the relationship between the masking threshold and the power spectral density for each frequency; calculating a difference between the power spectral density of the initial audio signal and the power spectral density of the final audio signal for each frequency based on the determined weight; training the neural network models according to the calculated results; and generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.

In the step of training the neural network models, parameters included in the neural network model may be updated so that the calculated result is minimized.

The masking threshold may be a standard for masking noise generated during encoding and decoding of the neural network models in consideration of the sound pressure of the initial audio signal determined by the psychoacoustic model.

The step of determining the weight includes determining the weight to be higher at a specific frequency as the power spectral density of the initial audio signal with respect to the masking threshold increases, and determining the weight to be higher at a specific frequency as the masking threshold with respect to the power spectral density of the initial audio signal increases. The weight may be determined to be low in frequency.

An audio signal processing method according to an embodiment of the present invention includes obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; Obtaining a masking threshold for the initial audio signal through a psychoacoustic model; identifying noise generated during encoding and decoding of the initial audio signal in the final audio signal; calculating a difference between the masking threshold and noise included in the final audio signal for each frequency; training the neural network models according to the calculated results; and generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.

In the step of training the neural network models, parameters included in the neural network model may be updated so that the calculated result is minimized.

The masking threshold may be a standard for masking noise generated during encoding and decoding of the neural network models in consideration of the sound pressure of the initial audio signal determined by the psychoacoustic model.

An audio signal processing method according to an embodiment of the present invention includes obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; A first loss function for calculating the difference between the initial audio signal and the final audio signal in the time domain, and a second loss function for calculating the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain. calculating the difference between the initial audio signal and the final audio signal using; determining a power spectral density and masking threshold of the initial audio signal using a psychoacoustic model; of the initial audio signal and the final audio signal through a third loss function that calculates the difference between the initial audio signal and the final audio signal in the frequency domain based on the relationship between the power spectral density of the initial audio signal and the masking threshold. calculating the difference; updating parameters included in the plurality of neural network models based on results calculated through the first to third loss functions; And a new final audio signal that is distinguished from the final audio signal can be generated from the initial audio signal using neural network models with updated parameters.

The masking threshold may mask noise generated during encoding and decoding of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model.

Calculating the difference between the initial audio signal and the final audio signal through the third loss function includes determining a weight according to the relationship between the masking threshold and the power spectral density for each frequency; and calculating a difference between the power spectral density of the initial audio signal and the power spectral density of the final audio signal for each frequency through the third loss function based on the determined weight.

The step of determining the weight includes determining the weight to be higher at a specific frequency as the power spectral density of the initial audio signal with respect to the masking threshold increases, and determining the weight to be higher at a specific frequency as the masking threshold with respect to the power spectral density of the initial audio signal increases. The weight may be determined to be low in frequency.

An audio signal processing method according to an embodiment of the present invention includes the steps of: a) obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; b) calculating the difference between the initial audio signal and the final audio signal in the time domain; c) calculating the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain; d) determining a masking threshold using a psychoacoustic model; e) calculating the difference between the noise of the final audio signal and the masking threshold of the initial audio signal determined through the psychoacoustic model in the frequency domain; updating parameters included in the plurality of neural network models based on the results calculated in steps b), c), and d); and generating a new final audio signal from the initial audio signal using neural network models with updated parameters.

The masking threshold may be a standard for masking noise generated during encoding and decoding of the neural network models in consideration of the sound pressure of the initial audio signal determined by the psychoacoustic model.

An audio signal processing method according to an embodiment of the present invention includes obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal; i) a first loss function that calculates the difference between the initial audio signal and the final audio signal in the time domain, ii) a first loss function that calculates the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain a second loss function, iii) a third loss that calculates the difference between the initial audio signal and the final audio signal in a frequency band based on the relationship between the power spectral density of the initial audio signal and a masking threshold determined through a psychoacoustic model; Function and iv) training the plurality of neural network models using a fourth loss function that calculates the difference between the noise included in the final audio signal and the masking threshold of the initial audio signal determined through the psychoacoustic model in the frequency band. steps; and generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.

An audio signal processing method according to an embodiment of the present invention includes an audio signal processing device, wherein the processing device includes a processor, and the processor encodes and decodes an input audio signal to generate an output audio signal. Obtaining a final audio signal for the initial audio signal using a neural network model, i) a first loss function that calculates the difference between the initial audio signal and the final audio signal in the time domain, ii) the initial audio signal and the final audio signal a second loss function that calculates the difference in mel spectrum between audio signals in the frequency domain; iii) the initial audio based on the relationship between the power spectral density of the initial audio signal and a masking threshold determined through a psychoacoustic model; a third loss function that calculates the difference between the signal and the final audio signal in a frequency band; and iv) a frequency band that calculates the difference between the masking threshold of the initial audio signal and the noise included in the final audio signal determined through the psychoacoustic model. The plurality of neural network models can be trained using the fourth loss function calculated in , and a new final audio signal that is distinguished from the final audio signal from the initial audio signal can be generated using the trained neural network models.

An audio signal processing method according to an embodiment of the present invention includes an audio signal processing device, wherein the processing device includes a processor, and the processor encodes and decodes an input audio signal to generate an output audio signal. Obtaining a final audio signal for the initial audio signal using a neural network model, i) a first loss function that calculates the difference between the initial audio signal and the final audio signal in the time domain, ii) the initial audio signal and the final audio signal a second loss function that calculates the difference in mel spectrum between audio signals in the frequency domain; iii) the initial audio based on the relationship between the power spectral density of the initial audio signal and a masking threshold determined through a psychoacoustic model; a third loss function that calculates the difference between the signal and the final audio signal in a frequency band; and iv) a frequency band that calculates the difference between the masking threshold of the initial audio signal and the noise included in the final audio signal determined through the psychoacoustic model. Calculate the difference between the initial audio signal and the final audio signal using at least one of the fourth loss functions calculated in , train the plurality of neural network models based on the calculated results, and train the trained neural network models. A new final audio signal that is distinct from the final audio signal can be generated from the initial audio signal using neural network models.

According to an embodiment of the present invention, when processing audio signals using a neural network model that encodes and decodes audio signals, the loss of the audio signal can be minimized by using a psychoacoustic model in the learning process of the neural network model. .

In addition, according to one embodiment of the present invention, in the learning process of a neural network model that performs encoding and decoding of audio signals, the quality of the restored audio signal is improved by learning the neural network model to minimize noise generated in the encoding process of the audio signal. It can be raised.

1 is a diagram showing the structure of an audio signal processing device according to an embodiment of the present invention.
Figure 2 is a diagram showing the relationship between neural network models and the structure of the neural network model according to an embodiment of the present invention.
Figure 3 is a diagram showing the structure of a loss function that calculates the difference between a final audio signal and an initial audio signal generated by neural network models according to an embodiment of the present invention.
Figure 4 is a diagram showing the results of noise generation depending on whether or not a loss function is used according to an embodiment of the present invention.
Figure 5 is a flow chart showing a method of processing an audio signal according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, various changes can be made to the embodiments, so the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

1 is a diagram showing the structure of an audio signal processing device according to an embodiment of the present invention.

The present invention provides a psychoacoustic model in training a neural network model that encodes and decodes audio signals in order to reduce the loss of audio signals that occurs in the process of encoding and decoding audio signals. Audio signals are processed by training a neural network model using a loss function using PAM.

The audio signal processing device of the present invention may include a processor, and the processor included in the processing device may perform an audio signal processing method. In the present invention, encoding may refer to a process of converting an audio signal into a code vector, and decoding may refer to a process of restoring an audio signal from a code vector.

Here, an audio signal is, in a broad sense, a concept that is distinct from a video signal and refers to a signal that can be identified by hearing when played. In a narrow sense, it is a concept that is distinct from a speech signal, and is a concept that is distinct from a speech signal. It refers to a signal that has no or few characteristics. The audio signal in the present invention should be interpreted in a broad sense and can be understood as an audio signal in the narrow sense when used separately from a voice signal.

Referring to Figure 1, the neural network model 102-104 used in the present invention is implemented in a processing unit, and the neural network model 102-104 encodes the input audio signal to generate a code vector and quantizes the code vector ( quantization). Then, the neural network models 102-104 generate an output audio signal by reconstructing the input audio signal by decoding the quantized code vector.

Referring to FIG. 1, a processing device obtains a final audio signal by using a plurality of successive neural network models 102-104 for the initial audio signal. Specifically, the plurality of neural network models 102-104 are in a continuous relationship, and the ith neural network model converts the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model into an input audio signal. to generate an output audio signal.

For example, the first neural network model 102 generates an output audio signal using the initial audio signal input to the processing device as an input audio signal, and the second neural network model 103 generates the initial audio signal and the output of the first neural network model. An output audio signal is generated using the difference between audio signals as an input audio signal.

And, when there are N neural network models, the N-th neural network model 104 uses the difference between the input audio signal and the output audio signal of the N-1-th neural network model as an input audio signal to generate an output audio signal. Accordingly, the final audio signal relative to the initial audio signal input to the processing device corresponds to an audio signal obtained by adding the output audio signals of each of the plurality of neural network models 102 to 104.

Neural network models 102-104 may be composed of multiple layers including parameters. In the present invention, the neural network model can correspond to an autoencoder implemented as a convolutional neural network (CNN). However, the neural network model of the present invention may be implemented in various forms and is not limited to the examples described above. The specific structure of the neural network model is described later in Figure 2.

A neural network model is trained to reduce the difference between the final and initial audio signals. Specifically, the processing device updates parameters included in a plurality of neural network models so that the result of a loss function that calculates the difference between the final audio signal and the initial audio signal is minimized. In other words, the loss function can be a standard for learning a neural network model.

The processing device may determine the difference between the final audio signal and the initial audio signal by inputting the difference between the input audio signal and the output audio signal of each of the plurality of neural network models into the loss function.

The processing device calculates the difference between the input audio signal and the output audio signal of each of the plurality of neural network models through a loss function, according to a first loss function for the time domain, a second loss function for the frequency domain, and a psychoacoustic model. At least one loss function may be used among a third loss function based on the relationship between the power spectral density of the initial audio signal and the masking threshold and a fourth loss function based on the relationship between the masking threshold according to the psychoacoustic model and the noise generated during the quantization process. there is.

In a first example, the processing device calculates the difference between the input audio signal and the output audio signal of each of the plurality of neural network models in the time domain through a first loss function, and sums the calculated results for each neural network model to create the final audio signal and the initial audio signal. Differences between signals can be determined.

As a second example, the processing device converts the input audio signal and the output audio signal of each of the plurality of neural network models into a mel spectrum, and calculates the difference between the input audio signal and the output audio signal converted through the second loss function into the frequency domain. By calculating and summing the calculated results for each neural network model, the difference between the final audio signal and the initial audio signal can be determined.

As a third example, the processing device may obtain a masking threshold of the initial audio signal through a psychoacoustic model. Additionally, the processing device may obtain the power spectral density (PSD) for the initial audio signal through the psychoacoustic model.

At this time, the masking threshold is based on psychoacoustic theory. It uses the characteristic that small audio signals adjacent to large audio signals are not well perceived in the human auditory structure to mask noise generated during the quantization process of each neural network model. It is a standard for

That is, when generating the final audio signal, the processing device considers the sound pressure of the initial audio signal determined through a psychoacoustic model and can mask noises by removing noises with a sound pressure level lower than the masking threshold for each frequency. .

The processing unit calculates the difference between the initial audio signal and the final audio signal in the frequency band based on the relationship between the masking threshold and the power spectral density of the initial audio signal for each frequency determined by the psychoacoustic model through a third loss function and a neural network. By combining the results calculated for each model, the difference between the final and initial audio signals can be determined.

Here, the psychoacoustic model (PAM) is used to calculate the masking effect by generating frequency-specific power spectral density for the initial audio signal, based on psychoacoustic theory, and determining the masking threshold according to the generated power spectral density. It's a model. Power spectral density refers to the density distribution of energy or power of an audio signal in the frequency domain of the audio signal.

As a fourth example, the processing unit calculates the difference between the noise of the final audio signal and the masking threshold of the initial audio signal determined through the psychoacoustic model in the frequency band through the fourth loss function, and sums the results calculated for each neural network model to obtain The difference between the final audio signal and the initial audio signal can be determined.

The specific calculation method of the third loss function and the fourth loss function will be described later in FIG. 3.

The processing device may determine the difference between the final audio signal and the initial audio signal using at least one loss function among the first to fourth loss functions. The processing device may update parameters included in the plurality of neural network models to minimize the difference between the final audio signal and the initial audio signal calculated through at least one of the first to fourth loss functions.

The processing device may obtain a final audio signal by processing the initial audio signal using a plurality of updated neural network models.

Figure 2 is a diagram showing the relationship between neural network models and the structure of the neural network model according to an embodiment of the present invention.

Figure 2(a) is a diagram showing the relationship between a plurality of neural network models used in the present invention. Figure 2(b) is a diagram showing the structure of one neural network model.

In Figures 2 (a) and (b), s means the initial audio signal, and s ⁽ⁱ⁾ means the input audio signal of the ith neural network model. and, ⁽ⁱ⁾ refers to the output audio signal of the ith neural network model. As can be seen in (a) of Figure 2, the ith neural network model is the difference between the input audio signal and the output audio signal of the i-1th neural network model (s ^(i-1) - ^(i-1) ) as an input audio signal and output audio signal ( ⁽ⁱ⁾ ) is generated.

Each neural network model (s ⁽ⁱ⁾ ) may include an encoder that performs encoding, a code (h ⁽ⁱ⁾ ) that quantizes a code vector generated by encoding an input audio signal, and a decoder that performs decoding. Encoders and decoders may correspond to layers included in the neural network model.

Referring to (b) of FIG. 2, the encoder of the neural network model encodes the input audio signal on a frame-by-frame basis to generate a code vector. For example, referring to (b) of FIG. 2, Bottleneck ResNet Blocks to which ResNet, a classification model using CNN, is applied, can be used as an encoder of a neural network model.

The neural network model can generate a quantized code (h ⁽ⁱ⁾ ) by quantizing and entropy coding the code vector (z ⁽ⁱ⁾ ) generated through the encoder. And, the decoder of the neural network model restores the input audio signal (s ^{(i )) using the quantized code (h (i} )). ⁽ⁱ⁾ ) can be generated. Like the encoder, the decoder can also use Bottleneck ResNet Blocks to which ResNet is applied. However, the model used in the neural network model is not limited to ResNet.

For example, a neural network model to which ResNet is applied can perform encoding and decoding of an input audio signal according to the values listed in Table 1 below.

In Table 1, the input shape and output shape for each layer of the neural network model mean (frame length, channel), and the kernel shape means (kernel size, in-channel, out-channel). means.

Figure 3 is a diagram showing the structure of a loss function that calculates the difference between a final audio signal and an initial audio signal generated by neural network models according to an embodiment of the present invention.

Referring to FIG. 3, the processing device processes the initial audio signal (s) through a plurality of neural network models to produce a final audio signal (s) obtained by summing the output audio signals of each neural network model. ⁽¹⁾ + ⁽²⁾ +...+ In obtaining ^(N) ), the difference between the input audio signal and the output audio signal of each neural network model can be input into the loss function 302.

The processing device may update parameters included in each neural network model using the sum 301 of the difference between the input audio signal and the output audio signal of each neural network model. Through process 307, the processing unit generates a final audio signal for the initial audio signal while updating the parameters so that the result of the loss function 302 is minimized, and adjusts the parameters so that the result of the loss function 302 is minimized. The final audio signal for the initial audio signal can be obtained with the neural network model included.

That is, the processing device trains a plurality of neural network models by updating parameters so that the result of the loss function 302 is minimized, and the neural network models including parameters that minimize the result of the loss function 302 are trained neural networks. Corresponds to the model.

And, the loss function 302 is based on the relationship between the first loss function 303 for the time domain, the second loss function 304 for the frequency domain, the power spectral density of the initial audio signal according to the psychoacoustic model, and the masking threshold. It may include a third loss function 305 based on a masking threshold based on a psychoacoustic model and a fourth loss function 306 based on the relationship between noise generated during the quantization process.

That is, the processing device may obtain the result of the loss function 302 for the difference between the initial audio signal and the final audio signal using at least one of the first to fourth loss functions 303 to 306. A loss function 302 using at least one of the first to fourth loss functions 303 to 306 may be defined according to Equation 1 below.

is the 1st-4th loss function ( It means the loss function 302 determined by, One, 2, 3 and 4 is the 1st-4th loss function ( Determine the loss function used in, or the 1-4 loss functions ( Since each unit is different, this is the weight that adjusts it.

for example, One, 2, When 3 is 0, the processing unit uses the fourth loss function ( ) is used to calculate the difference between the initial audio signal and the final audio signal. or, One, 2, 3 and If 4 is all greater than 0, the processing unit calculates the 1-4 loss function for the difference between the initial audio signal and the final audio signal ( Calculate the difference between the initial and final audio signals by adding up each result.

The first loss function 303 is a loss function that calculates the difference between the initial audio signal and the final audio signal in the time domain. That is, the processing device may calculate the difference between the initial audio signal and the output audio signal in the time domain by inputting the difference between the input audio signal and the output audio signal of each neural network model into the first loss function 303. The first loss function 303 calculates the difference between the input audio signal and the output audio signal according to Equation 2 below.

In Equation 2, T corresponds to the time length of the frame that is the encoding and decoding unit of the neural network model, and t corresponds to the specific time of the initial audio signal. That is, s _t ⁽ⁱ⁾ and _t ⁽ⁱ⁾ means the input audio signal and output audio signal corresponding to a specific time t.

And, i means that it is the ith neural network model among N consecutive neural network models. The first loss function 303 is a function that outputs the result of summing the squares of the differences in the time domain between the input audio signal and the output audio signal at each time (t) for the neural network model (i) of each of the N neural network models.

Ultimately, the smaller the result output by the first loss function 303 means that the final audio signal accurately restores the initial audio signal, so the processing device constructs neural network models to minimize the result of the first loss function 303. train.

The second loss function 304 is a loss function that calculates the difference between the mel spectrum of the initial audio signal and the final audio signal in the frequency domain. Specifically, the processing unit converts the initial audio signal and the final audio signal into a mel spectrum. Mel spectrum means converting the frequency unit of the initial audio signal into mel-unit.

That is, the processing device may calculate the difference between the initial audio signal and the output audio signal in the frequency domain by inputting the difference between the Mel spectra of the input audio signal and the output audio signal of each neural network model to the second loss function 304. The second loss function 304 calculates the difference between the input audio signal and the output audio signal according to Equation 3 below.

In Equation 3, F corresponds to the frequency range of the frame that is the encoding and decoding unit of the neural network model, and f corresponds to the specific resolution included in F. y _f ⁽ⁱ⁾ and _f ⁽ⁱ⁾ means the Mel spectrum of the input audio signal and the Mel spectrum of the output audio signal for a specific frequency f.

i means that it is the ith neural network model among N consecutive neural network models. The second loss function 304 is a function that outputs the result of adding the square of the difference between the Mel spectra of the input audio signal and the output audio signal for each frequency (f) for each of the N neural network models (i).

Ultimately, the smaller the result output by the second loss function 304 means that the final audio signal accurately restores the initial audio signal, so the processing device constructs neural network models to minimize the result of the first loss function 303. train.

And, the third loss function 305 is a loss function that calculates the difference between the initial audio signal and the final audio signal in the frequency domain based on the relationship between the power spectral density of the initial audio signal and the masking threshold determined through the psychoacoustic model. .

The processing device may obtain the power spectral density and masking threshold for the initial audio signal through the psychoacoustic model to use the third loss function 305. The processing device determines a weight according to the relationship between the masking threshold and the power spectral density for each frequency through the third loss function 305, and based on the determined weight, the power spectral density of the initial audio signal and the power spectrum of the final audio signal for each frequency. The difference between densities can be calculated.

Specifically, the processing device may determine a weight representing the relationship between the power spectral density and the masking threshold for the initial audio signal according to Equation 4 below.

In Equation 4 above, at a specific frequency means a weight representing the relationship between the power spectral density and the masking threshold. m represents the masking threshold, and p represents the power spectral density for the initial audio signal.

According to Equation 4, the higher the power spectral density of the initial audio signal at a certain frequency is, the more difficult the processing device is to restore, and thus the higher the weight is. Decide on a low weight.

Then, the processing device calculates the difference between the power spectral density of the initial audio signal and the power spectral density of the final audio signal using the weight determined for each frequency through the third loss function 305. Specifically, the third loss function 305 is determined according to Equation 5.

In Equation 5 above, f means a specific frequency, and x _f ⁽ⁱ⁾ and _f ⁽ⁱ⁾ represents the power spectral density of the input audio signal and the power spectral density of the output audio signal of the neural network model, respectively. And, w _f means the weight determined for a specific frequency.

i means that it is the ith neural network model among N consecutive neural network models. The third loss function 305 outputs the result of multiplying the square of the difference between the power spectral densities of the input audio signal and the output audio signal for each frequency (f) by the weight for each neural network model (i) of the N neural network models. This is the loss function.

Accordingly, the processing device can increase the restoration rate of the initial audio signal by processing audio signals that are difficult to restore in a higher proportion than other audio signals through the third loss function 305 to which weights according to the psychoacoustic model are applied.

At this time, the output audio signal of the neural network model learned through the third loss function 305 may not be able to mask large noise. The processing device can solve this problem using the fourth loss function 306.

The fourth loss function 306 is a loss function that calculates the difference between the masking threshold of the initial audio signal and the noise included in the final audio signal determined through the psychoacoustic model in the frequency band. Here, noise refers to the logarithmic power density function (logarithmic PSD) of the difference between the initial audio signal and the final audio signal.

The processing device may calculate the difference between the initial audio signal and the final audio signal based on the relationship between the previously obtained masking threshold and noise generated during the encoding and decoding process of the initial audio signal.

Specifically, the processing device identifies noise generated during the encoding and decoding process of the initial audio signal in the final audio signal and includes it in the final audio signal and a masking threshold for each frequency as shown in Equation 6 below through the fourth loss function 306. Calculate the difference between the noise.

In Equation 6 above, n _f ⁽ⁱ⁾ and m _f ⁽ⁱ⁾ mean noise and masking thresholds corresponding to a specific frequency (f), respectively. The processing device can determine the minimum difference between the masking threshold and the noise for each neural network model by determining the frequency with the smallest difference between the masking threshold and the noise. The fourth loss function 306 calculates the result for each neural network model by subtracting the determined minimum difference from the result of adding the difference between the noise and the masking threshold for each frequency and outputs the sum result.

That is, the processing device can reduce noise generated during the encoding and decoding process of the initial audio signal by updating the parameters of the neural network model to minimize the result of the fourth loss function 306.

The processing device may train the neural network model to minimize the result of at least one of the first to fourth loss functions 303 to 306, thereby generating a final audio signal for the initial audio signal through the trained neural network models.

Figure 4 is a diagram showing the results of noise generation depending on whether or not a loss function is used according to an embodiment of the present invention.

Figure 4(a) is a graph showing the relationship between the noise of the final audio signal and the masking threshold of the initial audio signal obtained with neural network models trained through the first to third loss functions.

Figure 4(b) is a graph showing the relationship between the noise of the final audio signal and the masking threshold of the initial audio signal obtained with neural network models trained through the 1-4 loss functions.

Specifically, (a) in Figure 4 is in Equation 1 above. One, 2, 3 is greater than 0 4 is 0, and (b) in Figure 4 is One, 2, 3 and As 4 is greater than 0, 1=60, 2=5; 3=1; This is the case where 4=5.

Referring to (a) of FIG. 4, the result is that noise higher than the masking threshold is not masked in section 401. In this case, the quality of the final audio signal is degraded due to noise not included in the initial audio signal.

Referring to (b) of FIG. 4, since neural network models are trained based on the relationship between noise and masking threshold in the fourth loss function, noise higher than the masking threshold does not occur as shown in (a) of FIG. 4.

Figure 5 is a flow chart showing a method of processing an audio signal according to an embodiment of the present invention.

In step 501, the processing device obtains a final audio signal for the initial audio signal using a plurality of neural network models that encode and decode the input audio signal to generate the output audio signal.

At this time, the plurality of neural network models 102-104 are in a continuous relationship, and the ith neural network model converts the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model into an input audio signal. to generate an output audio signal.

In step 502, the processing device inputs the difference between the input audio signal and the output audio signal of each neural network model into the first loss function to calculate the difference between the initial audio signal and the output audio signal in the time domain. .

In step 503, the processing device inputs the difference between the Mel spectra of the input audio signal and the output audio signal of each neural network model to the second loss function to calculate the difference between the initial audio signal and the output audio signal in the frequency domain. It can be calculated.

At step 504, the processing device obtains the power spectral density and masking threshold for the initial audio signal through the psychoacoustic model to use the third loss function.

Then, the processing device determines a weight according to the relationship between the masking threshold and the power spectral density for each frequency through a third loss function, and based on the determined weight, the power spectral density of the initial audio signal and the power spectral density of the final audio signal for each frequency. The difference between them can be calculated.

In step 505, the audio signal processing device identifies noise generated during the encoding and decoding process of the initial audio signal in the final audio signal, and determines the masking threshold and final audio for each frequency as shown in Equation 6 below through the fourth loss function. Calculate the difference between noise included in the signal.

In step 506, the processing device may train a neural network model to minimize the result of at least one loss function among the first to fourth loss functions. Specifically, the processing device may update parameters included in the plurality of neural network models to minimize the difference between the final audio signal and the initial audio signal calculated through at least one of the first to fourth loss functions.

For example, the processing device may determine the difference between the final audio signal and the initial audio signal using only the first-second loss function, and may determine the difference between the final audio signal and the initial audio signal using only the third loss function, The difference between the final audio signal and the initial audio signal can be determined using only the fourth loss function, and the difference between the final audio signal and the initial audio signal can be determined using all of the first to fourth loss functions.

In step 507, the processing device may process the initial audio signal using the updated plurality of neural network models to generate the final audio signal.

Meanwhile, the method according to the present invention is written as a program that can be executed on a computer and can be implemented in various recording media such as magnetic storage media, optical read media, and digital storage media.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or combinations thereof. Implementations may include a computer program product, i.e., an information carrier, e.g., machine-readable storage, for processing by or controlling the operation of a data processing device, e.g., a programmable processor, a computer, or multiple computers. It may be implemented as a computer program tangibly embodied in a device (computer-readable medium) or a radio signal. Computer programs, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be written as a stand-alone program or as a module, component, subroutine, or part of a computing environment. It can be deployed in any form, including as other units suitable for use. The computer program may be deployed for processing on one computer or multiple computers at one site or distributed across multiple sites and interconnected by a communications network.

Processors suitable for processing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. Elements of a computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. Generally, a computer may include one or more mass storage devices that store data, such as magnetic, magneto-optical disks, or optical disks, receive data from, transmit data to, or both. It can also be combined to make . Information carriers suitable for embodying computer program instructions and data include, for example, semiconductor memory devices, magnetic media such as hard disks, floppy disks, and magnetic tapes, and Compact Disk Read Only Memory (CD-ROM). ), optical media such as DVD (Digital Video Disk), magneto-optical media such as Floptical Disk, ROM (Read Only Memory), RAM , Random Access Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically Erasable Programmable ROM). The processor and memory may be supplemented by or included in special purpose logic circuitry.

Additionally, computer-readable media can be any available media that can be accessed by a computer, and can include both computer storage media and transmission media.

Although this specification contains details of numerous specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of particular inventions. It must be understood. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, although features may be described as operating in a particular combination and initially claimed as such, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination may be a sub-combination. It can be changed to a variant of a sub-combination.

Likewise, although operations are depicted in the drawings in a particular order, this should not be construed as requiring that those operations be performed in the specific order or sequential order shown or that all of the depicted operations must be performed to obtain desirable results. In certain cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various device components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and the described program components and devices may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it is possible.

Meanwhile, the embodiments of the present invention disclosed in the specification and drawings are merely provided as specific examples to aid understanding, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that in addition to the embodiments disclosed herein, other modifications based on the technical idea of the present invention can be implemented.

101: audio processing device
102-104: Neural network model

Claims (20)

delete delete delete delete Obtaining a final audio signal for the initial audio signal using a plurality of neural network models that encode and decode the input audio signal to generate an output audio signal;
Obtaining a power spectral density and masking threshold for the initial audio signal through a psychoacoustic model;
determining a weight according to the relationship between the masking threshold and the power spectral density for each frequency;
calculating a difference between the power spectral density of the initial audio signal and the power spectral density of the final audio signal for each frequency based on the determined weight;
training the neural network models according to the calculated results; and
Generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.
Including,
The plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
How to handle it.
According to clause 5,
The step of training the neural network models is,
A processing method for updating parameters included in the neural network model so that the calculated result is minimized.
delete According to clause 5,
The step of determining the weight is,
The greater the power spectral density of the initial audio signal with respect to the masking threshold, the higher the weight is determined at a specific frequency, and the greater the masking threshold is with respect to the power spectral density of the initial audio signal, the lower the weight is determined at the specific frequency. , processing method.
Obtaining a final audio signal for the initial audio signal using a plurality of neural network models that encode and decode the input audio signal to generate an output audio signal;
Obtaining a masking threshold for the initial audio signal through a psychoacoustic model;
identifying noise generated during encoding and decoding of the initial audio signal in the final audio signal;
calculating a difference between the masking threshold and noise included in the final audio signal for each frequency;
training the neural network models according to the calculated results; and
Generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.
Including,
The plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
How to handle it.
According to clause 9,
The step of training the neural network models is,
A processing method for updating parameters included in the neural network model so that the calculated result is minimized.
delete Obtaining a final audio signal for the initial audio signal using a plurality of neural network models that encode and decode the input audio signal to generate an output audio signal;
determining a power spectral density and masking threshold of the initial audio signal using a psychoacoustic model;
i) a first loss function that calculates the difference between the initial audio signal and the final audio signal in the time domain; and ii) a first loss function that calculates the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain. the initial audio through a second loss function and iii) a third loss function that calculates the difference between the initial audio signal and the final audio signal in the frequency domain based on the relationship between the power spectral density of the initial audio signal and the masking threshold. calculating the difference between the signal and the final audio signal;
updating parameters included in the plurality of neural network models based on results calculated through the first to third loss functions; and
Generating a new final audio signal distinct from the final audio signal from the initial audio signal using neural network models with updated parameters.
Including,
The plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
How to handle it.
delete According to clause 12,
Calculating the difference between the initial audio signal and the final audio signal through the third loss function,
determining a weight according to the relationship between the masking threshold and the power spectral density for each frequency; and
Calculating the difference between the power spectral density of the initial audio signal and the power spectral density of the final audio signal for each frequency through the third loss function based on the determined weight.
Including, processing method.
According to clause 14,
The step of determining the weight is,
The greater the power spectral density of the initial audio signal with respect to the masking threshold, the higher the weight is determined at a specific frequency, and the greater the masking threshold is with respect to the power spectral density of the initial audio signal, the lower the weight is determined at the specific frequency. , processing method.
a) obtaining a final audio signal for the initial audio signal using a plurality of neural network models that encode and decode the input audio signal to generate an output audio signal;
b) calculating the difference between the initial audio signal and the final audio signal in the time domain;
c) calculating the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain;
d) determining a masking threshold using a psychoacoustic model;
e) calculating the difference between the noise of the final audio signal and the masking threshold of the initial audio signal determined through the psychoacoustic model in the frequency domain;
updating parameters included in the plurality of neural network models based on the results calculated in steps b), c), and d); and
Generating a new final audio signal from the initial audio signal using neural network models with updated parameters.
Including,
The plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
How to handle it.
delete Obtaining a final audio signal for the initial audio signal using a plurality of neural network models that encode and decode the input audio signal to generate an output audio signal;
i) a first loss function that calculates the difference between the initial audio signal and the final audio signal in the time domain, ii) a first loss function that calculates the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain a second loss function, iii) a third loss that calculates the difference between the initial audio signal and the final audio signal in a frequency band based on the relationship between the power spectral density of the initial audio signal and a masking threshold determined through a psychoacoustic model; Function and iv) training the plurality of neural network models using a fourth loss function that calculates the difference between the noise included in the final audio signal and the masking threshold of the initial audio signal determined through the psychoacoustic model in the frequency band. steps; and
Generating a new final audio signal that is distinct from the final audio signal from the initial audio signal using the trained neural network models.
Including,
The plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
How to handle it.
In the audio signal processing device,
The processing device includes a processor,
The processor,
Obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal, i) calculating the difference between the initial audio signal and the final audio signal in the time domain. a first loss function that calculates, ii) a second loss function that calculates the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain, iii) the initial audio determined through a psychoacoustic model a third loss function that calculates the difference between the initial audio signal and the final audio signal in a frequency band based on the relationship between the power spectral density of the signal and the masking threshold; and iv) the final audio signal determined through the psychoacoustic model. The plurality of neural network models are trained using a fourth loss function that calculates the difference between the included noise and the masking threshold of the initial audio signal in the frequency band, and the final loss is calculated from the initial audio signal using the trained neural network models. generate a new final audio signal that is distinct from the audio signal,
The plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
processing unit.
In the audio signal processing device,
The processing device includes a processor,
The processor,
Obtaining a final audio signal for an initial audio signal using a plurality of neural network models that encode and decode an input audio signal to generate an output audio signal, i) calculating the difference between the initial audio signal and the final audio signal in the time domain. a first loss function that calculates, ii) a second loss function that calculates the difference in mel spectrum between the initial audio signal and the final audio signal in the frequency domain, iii) the initial audio determined through a psychoacoustic model a third loss function that calculates the difference between the initial audio signal and the final audio signal in a frequency band based on the relationship between the power spectral density of the signal and the masking threshold; and iv) the final audio signal determined through the psychoacoustic model. Calculate the difference between the initial audio signal and the final audio signal using at least one loss function among a fourth loss function that calculates the difference between the included noise and the masking threshold of the initial audio signal in the frequency band, and the calculated Train the plurality of neural network models based on the results, and generate a new final audio signal that is differentiated from the final audio signal from the initial audio signal using the trained neural network models, and the plurality of neural networks are,
As a continuous relationship, the i-th neural network model generates an output audio signal by using the difference between the output audio signal of the i-1th neural network model and the input audio signal of the i-1th neural network model as an input audio signal,
The masking threshold is,
It is a standard for masking noise generated during the encoding and decoding process of the neural network models by considering the sound pressure of the initial audio signal determined by the psychoacoustic model,
The final audio signal is,
Corresponding to an audio signal that is the sum of the output audio signals of each of the plurality of neural network models,
processing unit.