KR20190060628A - Method and apparatus of audio signal encoding using weighted error function based on psychoacoustics, and audio signal decoding using weighted error function based on psychoacoustics - Google Patents

Abstract

An objective of the present invention is to use acoustic characteristics of a person to use a perceptually weighted error function to provide increased audio signal quality while having the same model complexity or provide audio signal quality of the same level while having low model complexity. According to an embodiment of the present invention, a neural network applied to a method of audio signal encoding using an audio signal encoding apparatus comprises: a step of generating a masking threshold for a first audio signal before learning; a step of calculating a weight matrix to be applied to frequency components of the first audio signal based on the masking threshold; a step of using the weight matrix to generate a weighted error function by correcting a preset error function; and a step of applying parameters learned by using the weighted error function to the first audio signal to generate a second audio signal.

Description

FIELD OF THE INVENTION [0001] The present invention relates to a method and apparatus for encoding an audio signal using a psychoacoustic-based weighted error function, and a method and apparatus for decoding an audio signal. }

The following embodiments are directed to a method and apparatus for encoding an audio signal using a psychoacoustic-based weighted error function and a method and apparatus for decoding an audio signal. More specifically, the present invention relates to a method and apparatus for decoding an audio signal using a weighted error function considering human auditory characteristics And an audio signal decoding method and apparatus.

Recently, voice and audio codecs for various purposes and applications have been developed in standardization organizations such as ITU-T, MPEG, and 3GPP. Most audio codecs are based on a psychoacoustic model that uses various auditory characteristics of a person. In addition, although the voice codec is mainly based on the voice generation model, at the same time, the person's cognitive characteristics are also utilized for the subjective quality improvement.

As described above, the conventional audio and audio codecs use a method based on human auditory characteristics to effectively control the quantization noise generated in the encoding step.

According to one embodiment, a perceptually weighted error function can be utilized using the human auditory characteristics to provide improved audio signal quality with the same model complexity.

According to an exemplary embodiment, a perceptually weighted error function may be used, using the human auditory characteristic, to provide the same level of audio signal quality with a lower model complexity.

According to an aspect of the present invention, there is provided a neural network applied to an audio signal encoding method using an audio signal encoding apparatus, comprising: generating a masking threshold for a first audio signal before being learned; Calculating a weight matrix to be applied to a frequency component of each of the first audio signals based on the masking threshold; Generating a weighted error function by modifying a preset error function using the weight matrix; And applying the learned parameter to the first audio signal using the weighted error function to generate a second audio signal.

Wherein the weight matrix comprises a weight to be applied to a frequency component of the first audio signal and the weight is inversely proportional to a masking threshold for the first audio signal and proportional to a magnitude of a frequency component of each of the first audio signals Lt; / RTI >

The neural network may be a neural network that further includes comparing the generated second audio signal with the first audio signal to perform a perceptual quality assessment.

The perceptual quality evaluation may be an objective evaluation or MOS (Mean Opinion Score), a Multiple Stimuli with Hidden (MUSHRA), a Perceptual Evaluation of Speech Quality (PESQ), a Perceptual Objective Listening Quality Assessment (POLQA) Reference and Anchor).

Based on the perceptual quality evaluation, a neural network that determines whether or not the topology included in the model can be adjusted.

If the result of the perceptual quality evaluation does not satisfy the predetermined quality requirement, the parameter is re-learned using the model with increased complexity, and the result of the perceptual quality evaluation May be a neural network that re-learns the parameters using a reduced complexity model within the quality requirements if the pre-established quality requirements are met.

According to an aspect of the present invention, there is provided an audio signal encoding method performed by an audio signal encoding apparatus to which a neural network is applied, the method comprising: receiving an input audio signal; The neural network comprising at least one hidden layer and generating a dimensionally reduced potential vector of the input audio signal based on the learned hidden layer parameter using the neural network; And outputting a bitstream by encoding the generated potential vector.

The step of generating a reduced-size potential vector of the input audio signal based on the learned hidden layer parameters using the neural network may include adjusting the topology of the model including the number of hidden layers, Generating the potential vector based on the re-learned parameter by generating the potential vector based on the re-learned parameter by applying the adjusted topology when adjustment of the topology of the model is possible, Signal coding method.

The coding of the latent vector may be a coding method of an audio signal to be binarized to transmit the bitstream through a channel.

According to an aspect of the present invention, there is provided a neural network applied to an audio signal decoding method using an audio signal decoding apparatus, the method comprising: generating a masking threshold for a first audio signal before being learned; Calculating a weight matrix to be applied to a frequency component of each of the first audio signals based on the masking threshold; Generating a weighted error function by modifying a preset error function using the weight matrix; And applying the learned parameter to the first audio signal using the weighted error function to generate a second audio signal.

Wherein the weight matrix comprises a weight to be applied to a frequency component of the first audio signal and the weight is inversely proportional to a masking threshold for the first audio signal and proportional to a magnitude of a frequency component of each of the first audio signals Lt; / RTI >

The neural network may be a neural network that further includes comparing the generated second audio signal with the first audio signal to perform a perceptual quality assessment.

The perceptual quality evaluation may be an objective evaluation or MOS (Mean Opinion Score), a Multiple Stimuli with Hidden (MUSHRA), a Perceptual Evaluation of Speech Quality (PESQ), a Perceptual Objective Listening Quality Assessment (POLQA) Reference and Anchor).

Based on the perceptual quality evaluation, a neural network that determines whether or not the topology included in the model can be adjusted.

If it is determined that the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter is re-learned using the increased complexity model, and the result of the perceptual quality evaluation May be a neural network that re-learns the parameters using a reduced complexity model within the requirements of the quality, if the pre-established quality requirements are met.

According to an aspect of the present invention, there is provided a method of decoding an audio signal performed by an audio signal decoding apparatus to which a neural network is applied, the method comprising: receiving a latent vector generated by applying a parameter learned through the neural network to an input audio signal, Receiving a stream; Recovering the latent vector from the received bitstream; The neural network may include one or more hidden layers and decoding the output audio signal from the restored latent vector using the hidden layer to which the learned parameter is applied.

Wherein the generated potential vector is generated based on the learned parameter if the topology including the number of the hidden layers and the number of nodes belonging to the hidden layer is impossible or unnecessary, The potential vector may be generated based on the re-learned parameter by applying the adjusted topology when the adjustment of the potential vector is possible.

The coding of the latent vector may be a decoding method of an audio signal binarized to transmit the bitstream through a channel.

According to one aspect, in an audio signal encoding apparatus employing a neural network, the audio signal encoding apparatus includes a processor and a memory including one or more instructions executable by the processor, wherein the one or more instructions are executable Wherein the neural network comprises one or more hidden layers and generates a dimensionally reduced potential vector of the input audio signal based on the learned hidden layer parameters using the neural network And outputting a bitstream by encoding the generated potential vector.

According to an aspect, there is provided an audio signal decoding apparatus to which a neural network is applied, the audio signal decoding apparatus including a processor and a memory including one or more instructions executable by the processor, Receiving a quantized bitstream generated by applying a parameter learned through the neural network to an input audio signal and reconstructing the latent vector from the received bitstream, May be an audio signal decoding apparatus that includes one or more hidden layers and decodes an output audio signal from the reconstructed potential vector using the hidden layer to which the learned parameter is applied.

According to one embodiment, a perceptually weighted error function can be utilized using the human auditory characteristics to provide improved audio signal quality with the same model complexity.

According to an exemplary embodiment, a perceptually weighted error function may be used, using the human auditory characteristic, to provide the same level of audio signal quality with a lower model complexity.

FIG. 1 illustrates a structure of an autoencoder including three hidden layers according to an exemplary embodiment of the present invention. Referring to FIG.
Figure 2 may represent a simultaneous masking effect, according to one embodiment.
3 is a diagram illustrating audible and non-audible areas in consideration of the masking effect, according to one embodiment.
4 is a diagram illustrating a learning process of a neural network using a weighted error function according to an exemplary embodiment of the present invention.
5 is a diagram illustrating an audio signal encoding apparatus, a channel, and an audio signal decoding apparatus according to an embodiment.
6 is a diagram illustrating a learning process of a neural network applied to an audio signal encoding method using an audio signal encoding apparatus according to an embodiment.
7 is a diagram illustrating a method of encoding an audio signal performed by an audio signal encoding apparatus to which a neural network is applied, according to an embodiment.
8 is a diagram illustrating a learning process of a neural network applied to an audio signal decoding method using an audio signal decoding apparatus according to an embodiment.
9 is a diagram illustrating an audio signal decoding method performed by an audio signal decoding apparatus to which a neural network is applied, according to an embodiment.
FIG. 10 is a diagram illustrating an audio signal encoding apparatus to which a neural network is applied, according to an embodiment.
11 is a block diagram illustrating an audio signal decoding apparatus to which a neural network is applied, according to an embodiment of the present invention.

Specific structural or functional descriptions of embodiments are set forth for illustration purposes only and may be embodied with various changes and modifications. Accordingly, the embodiments are not intended to be limited to the particular forms disclosed, and the scope of the present disclosure includes changes, equivalents, or alternatives included in the technical idea.

The terms first or second, etc. may be used to describe various elements, but such terms should be interpreted solely for the purpose of distinguishing one element from another. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected or connected to the other element, although other elements may be present in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms " comprises ", or " having ", and the like, are used to specify one or more of the described features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 illustrates a structure of an autoencoder including three hidden layers according to an exemplary embodiment of the present invention. Referring to FIG.

According to an embodiment, audio signal encoding and decoding can be performed by applying a neural network. At this time, the neural network may include various models of deep learning / machine learning. Specifically, a neural network may include an autoencoding method as a model of deep running.

Neural networks can solve optimization problems to minimize the error function or cost function. Here, the optimization is performed by an iterative learning algorithm, and a parameter that minimizes the error through optimization can be found.

, R 차원의 N개의 출력 데이터

를 의미할 수 있고, 파라미터

는 입력 데이터에 적용된 뉴럴 네트워크의 모델을 나타낼 수 있다. For example, the input data to which the neural network is applied can predict the output data as shown in Equation 1 below. Here, the N input data of the D dimension

, N pieces of output data of R dimension

, And the parameter

May represent a model of the neural network applied to the input data.

와 목표 데이터

의 차이는 오류를 나타낼 수 있고, 오류 함수는 아래의 수학식 2와 같이 정의될 수 있다. 여기서, 목표 데이터

는 R 차원의 N개의 목표 데이터일 수 있다.At this time, the predicted output data

And target data

And the error function can be defined as Equation (2) below. ≪ EMI ID = 2.0 > Here,

May be N target data of the R dimension.

과

은 목표 데이터와 출력 데이터의 n-번째 column 벡터를 나타낼 수 있다. 따라서, 아래의 수학식 3과 같이, N개의 데이터에 대해서 총 오류는 계산될 수 있다.here,

and

May represent the n-th column vector of the target data and the output data. Therefore, the total error can be calculated for N data, as shown in Equation (3) below.

Thus, the parameters can be adjusted to minimize the total error, and parameters that minimize the total error by the iterative learning algorithm can be found. There may be models of various neural networks that perform repetitive learning algorithms. Among them, Autoencoding method can be effective for learning latent patterns from input data.

Hereinafter, a neural network to which the Autoencoding method is applied will be described according to an embodiment of the present invention. However, the present invention is not limited to the autoencoding method, and may include a model to which another neural network is applied.

According to one embodiment, the Autoencoder can have the same size of the target data Y and the input data X, and can be effective in reducing the dimension. When the autoencoder includes L hidden layers, a fully-connected deep autoencoder can be recursively defined as shown in Equation 4 below.

과

는 각각

번째 레이어에 대한 가중치(weighting)와 바이어스(bias)를 나타낼 수 있다. 이때, 파라미터

는

를 나타낼 수 있다.here,

and

Respectively

And the weighting and bias for the i < th > layer. At this time,

The

Lt; / RTI >

1 is a diagram illustrating an autoencoder including three hidden layers according to an exemplary embodiment of the present invention. Here, a bottom layer 110 represents an input layer to which data is input, a top layer 120 represents an output layer from which data is output, a middle layer represents a hidden layer 130, Lt; / RTI > In addition, the node is included in the input layer / output layer / hidden layer, among which the node 140 may represent the biased node.

According to an exemplary embodiment, the Autoencoder may be a compression system including a coding unit including an input layer and a hidden layer 1, 2, and a decoding unit including a hidden layer 3 and an output layer. At this time, the hidden layer 2 may be a code layer for compressing input data.

인 경우 code 레이어에서 차원 축소된

를 생성할 수 있다. 또한, 목표 데이터 Y와 입력 데이터 X에 대해 아래의 수학식 5와 같이 설정된 오류 함수를 이용하여, Autoencoder의 복호화부는

로부터 입력 데이터를 복원할 수 있다. The encoding unit of the compression system Autoencoder

If the dimension is collapsed in the code layer

Lt; / RTI > Using the error function set for the target data Y and the input data X as shown in Equation (5) below, the decryption unit of the autoencoder

The input data can be restored.

For an effective code layer, the number of hidden layers and the number of nodes in hidden layers can be increased. At this time, the complexity of the autoencoder and the quality of the output data can be trade-off. Therefore, when the complexity of the autoencoder is large, battery consumption and memory capacity problems may occur.

According to another embodiment, the autoencoder may be a denoising autoencoder that generates a clean signal from which a noise is removed using a noisy signal. Noise may include additive noise, reverberation, and band-pass filtering.

에 의해서

로 표현될 수 있다. 수학식 6에서,

는 변형 함수의 역함수를 근사화한 것으로,

를 나타낼 수 있다. A denoising autoencoder that generates a clean signal from a noisy signal can be expressed as Equation (6) below. In this case, Y may be the magnitude spectrum of the original signal, and X may be the magnitude spectrum of the distorted signal. Where X is the deformation function

By

. ≪ / RTI > In Equation (6)

Is an approximation of the inverse function of the deformation function,

Lt; / RTI >

를 추정하도록 학습되는 것이 효과적일 수 있다. In this case, for example, when the original signal is deformed due to additional noise, the denoising autoencoder may be an ideal mask for removing additional noise rather than directly estimating the original signal, as shown in Equation (7)

It may be effective to learn to estimate.

Here, the ideal mask can be used to remove the added noise of the modified signal using the Hadamard product as shown in Equation (8) below.

는 추정된 이상적인 비율 마스크(ideal ratio mask)일 수 있으며, 아래의 수학식 9와 같이 표현될 수 있다. 여기서, Q는 부가 잡음에 대한 크기 스펙트럼일 수 있으며, 원 신호가 부가 잡음에 의해 변형될 경우 변형 함수는 수학식 10과 같이 정의될 수 있다.At this time,

May be an estimated ideal ratio mask and may be expressed as Equation (9) below. Here, Q may be a magnitude spectrum for additional noise, and a distortion function may be defined as Equation (10) when the original signal is modified by additional noise.

를 학습할 수 있다. 다만, 변형 함수

가 매우 복잡하기 때문에, 많은 히든 레이어와 노드의 개수는 필요할 수 있다. 따라서, 모델의 복잡도와 성능간의 trade-off가 발생할 수 있다. The denoising autoencoder uses a structure similar to Equation (4)

Can be learned. However,

Is very complex, many hidden layers and the number of nodes may be needed. Thus, a trade-off between model complexity and performance can occur.

Therefore, in order to solve the trade-off, the autoencoding method based on the error function using the human auditory characteristic will be described in detail below. By applying the error function using the auditory characteristic, the model complexity can be lowered and the performance of the autoencoder can be improved.

Figure 2 may represent a simultaneous masking effect, according to one embodiment.

The graph 210 shows the audible sound pressure level in decibels (dB) according to the frequency of the audio signal in a quiet environment. For example, it shows the lowest dB in the 4 kHz frequency band, and the threshold of the audible music level may increase with low frequency or high frequency bands. More specifically, most people can not recognize a large tonal signal of about 30 db at 30 Hz, but a human can perceive a relatively small tonal signal of about 10 db at 1 kHz.

Graph 230 may represent a graph modified by a tonal signal present at 1 kHz. That is, the tonal signal can rise the graph 210 at 1 kHz. Thus, a signal smaller in magnitude than the graph 230 can not be recognized by a person.

A masker 220 represents a tonal signal at 1 kHz, and a masker may represent a signal masked by a masker. For example, MASKI may include a signal smaller than the graph 230.

3 is a diagram illustrating audible and non-audible areas in consideration of the masking effect, according to one embodiment.

Fig. 3 shows the spectrum of the input audio signal superimposed on the graph of Fig. Here, the audible region 340 may indicate that the spectrum 360 of the input audio signal has a spectrum greater than the masking threshold curve at that frequency. In addition, the non-visible region 350 may indicate that the spectrum 360 of the input audio signal has a spectrum that is less than the masking threshold curve at that frequency.

For example, at 30 Hz, the graph 310 may be non-audible region 350 because it is larger than graph 360, and at 10 kHz, graph 310 may be larger than graph 360, And may be the audible region 340 because the graph 360 at 4 kHz is smaller than the graph 310.

According to one embodiment, the error function can be modified such that, in the training of the Autoencoder, more of the audible area is considered than the non-audible area, depending on the characteristics of the input audio signal. That is, if the magnitude of a particular frequency component of the input audio signal is less than the corresponding masking threshold, then the person may not be sensitive to the error. Or if the magnitude of a particular frequency component of the input audio signal is greater than the corresponding masking threshold, the person may be more sensitive to the error.

Therefore, for the training of the autoencoder considering human auditory characteristics, the modified error function can be expressed as Equation (11) below. At this time, the modified error function may represent a weighted error function that has modified the predetermined error function.

번째 샘플의 번째 계수가 큰 마스킹 임계치를 가질 경우 대응하는 가중치

은 상대적으로 작을 수 있다. 또 다른 예를 들면,

번째 샘플의

번째 계수가 작은 마스킹 임계치를 가질 경우 대응하는 가중치

은 상대적으로 클 수 있다.Where H denotes a weighting matrix, and the weighting matrix may include a weight applied to a frequency component of each of the input audio signals. For example,

Th sample If the ith coefficient has a large masking threshold, the corresponding weighting value

Can be relatively small. As another example,

Th sample

If the ith coefficient has a small masking threshold, the corresponding weighting value

Can be relatively large.

4 is a diagram illustrating a learning process of a neural network using a weighted error function according to an exemplary embodiment of the present invention.

According to one embodiment, the frequency spectrum of the first audio signal may be obtained for a predetermined length of analysis frame through the time-frequency analysis 401. [ At this time, the frequency spectrum may be obtained by applying a filter bank or MDCT (Modified Discrete Cosine Transform) to the first audio signal, and may be obtained by applying another method.

Here, the first audio signal may represent a training audio signal for parameter learning. In addition, the predetermined length of the analysis frame may include an analysis frame of 10 ms to 50 ms in length of the first audio signal, and may also include analysis frames of different lengths.

는 생성될 수 있다.Thus, through the time-frequency analysis 401, the frequency spectrum Y of the first audio signal or the frequency spectrum of the first audio signal,

Can be generated.

According to one embodiment, a psychoacoustic analysis 402 may be used to calculate a masking threshold for the first audio signal. That is, the masking threshold can be calculated by analyzing the auditory characteristics of the first audio signal.

The psychoacoustic analysis method may include MPEG PAM-I or MPEG PAM-II and may include psychoacoustic analysis by other methods. For example, temporal masking as well as simultaneous masking effects can be used.

According to one embodiment, using the masking threshold computed through psychoacoustic analysis 402, a weight matrix to be applied to the frequency components of each of the first audio signals may be calculated (403).

At this time, the weighting matrix may include a weight to be applied to the frequency component of the first audio signal, and the weight may be set to be inversely proportional to the masking threshold for the first audio signal and proportional to the magnitude of the frequency component of each of the first audio signal. That is, the weight can be determined by the Signal-to-Mask Ratio (SMR), and the SMR can represent the ratio between the size and the size of the masking threshold in the frequency component of each of the first audio signals.

에 대한 가중 행렬의 가중치

는 대응하는 마스킹 임계치에 반비례하고 주파수 성분의 크기에 비례하도록 설정될 수 있다. For example, each frequency component of the first audio signal

≪ / RTI > weight < RTI ID =

May be set to be inversely proportional to the corresponding masking threshold and proportional to the magnitude of the frequency component.

According to one embodiment, when the masking threshold is expressed in decibels, each weight of the weighting matrix may be expressed by Equation (12) below. However, according to one embodiment, it may include a linear scale transformation or a log scale transformation between the masking threshold and the weight, and may also include other scale transformations.

는 오류 가중 함수(404)를 통해 가중된 오류 함수를 생성할 수 있다. 여기서, 수학식 11과 같이 표현되는 가중된 오류 함수는 가중치를 적용하여 모델의 파라미터의 학습에 이용될 수 있다. According to one embodiment,

May generate a weighted error function via the error weight function (404). Here, the weighted error function expressed by Equation (11) can be used to learn the parameters of the model by applying weights.

At this time, the model may include a neural network, for example, an Autoencoder. The model may also include a topology, and the topology of the model may include an input layer, one or more hidden layers, an output layer, and nodes contained in each layer.

학습(405)는 가중된 오류 함수를 이용하여 학습될 수 있다. 예를 들면, 초기 모델의 토폴로지에 대해서 학습이 수행될 수 있으며, 학습이 완료된 모델의 토폴로지를 이용하여 예측된 오디오 스펙트럼 Z는 출력될 수 있다. According to one embodiment, the parameters of the model

Learning 405 can be learned using a weighted error function. For example, learning may be performed on the topology of the initial model, and the predicted audio spectrum Z may be output using the topology of the completed model.

에 대해 학습된 모델의 파라미터를 적용하여 생성될 수 있다. At this time, the predicted audio spectrum Z is the frequency spectrum Y of the first audio signal, or the frequency spectrum of the first audio signal

Lt; RTI ID = 0.0 > model < / RTI >

또는 제1 오디오 신호의 주파수 스펙트럼 Y와 비교하여 품질 평가를 수행할 수 있다. According to one embodiment, the perceptual quality estimate 406 may include a predicted audio spectrum as a first audio signal

Or the frequency spectrum Y of the first audio signal.

The perceptual quality evaluation is based on objective evaluation or MOS (Mean Opinion Score), Multiple Stimuli with Hidden (MUSHRA), Perceptual Evaluation of Speech Quality (PESQ), Perceptual Objective Listening Quality Assessment (POLQA) Reference and Anchor) can be used. Perceptual quality assessment is not limited to this, and may include other quality assessments.

The perceptual quality assessment 406 may indicate a quality assessment for the currently learned model. Based on the quality evaluation, it can be determined whether or not the model requirements such as the preset quality and the complexity of the model are satisfied. Further, based on the quality evaluation, it is possible to determine whether or not the model topology such as the complexity of the model can be adjusted. Here, the complexity of the model can have a positive correlation with the number of hidden layers and the number of nodes.

According to one embodiment, if the topology of the model is not adjustable or is determined not to be adjusted (407), the parameters of the currently learned model may be stored and the learning may be terminated. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, if the topology of the model is determined to be adjustable 407, the topology of the model may be updated 408 and the process for learning described above may be repeated. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning described above can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning described above can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

According to one embodiment, signal processing may be performed on the input audio signal based on the parameters of the learned model. At this time, the signal processing may include, but is not limited to, compression / noise cancellation / encoding / decoding.

5 is a diagram illustrating an audio signal encoding apparatus, a channel, and an audio signal decoding apparatus according to an embodiment.

According to one embodiment, the audio signal encoding apparatus 510 to which the neural network is applied may receive an input audio signal. At this time, the audio signal encoding apparatus 510 can apply the learned model through the neural network. Here, the neural network may include an autoencoder.

번째 히든 레이어를 나타낼 수 있다. The audio signal encoding apparatus 510 may include an autoencoder encoding unit 511 and a quantization unit 512. Here, the autoencoder encoding unit 511 may include a layer from an input layer to a code layer,

The second hidden layer.

The audio signal encoding apparatus 510 can generate a reduced-size latent vector of the received input audio signal based on the learned hidden layer parameters using the neural network. The generated potential vector may be quantized or encoded by the quantization unit 512 and output as a bitstream. Here, the quantizer 512 may include a process of binarizing the bitstream to transmit the bitstream through the transport channel.

The output bit stream can be transmitted to the audio signal decoding apparatus 520 by the transmission channel 530.

번째 히든 레이어부터 출력 레이어까지의 레이어를 포함할 수 있다. The audio signal decoding apparatus 520 may include an autoencoder decoding unit 521 and an inverse quantization unit 522. Here, the autoencoder decoding unit 521

The second hidden layer to the output layer.

The audio signal decoding apparatus 520 can restore the latent vector by inverse-quantizing or inverse-coding the bitstream transmitted through the transmission channel 530 by the dequantizer 522. [ Using the restored potential vector, the autoencoder decoding unit 521 can decode the output audio signal, or the autoencoder decoding unit 521 can calculate the output audio signal. Here, the inverse quantization unit 522 may include a process of binarizing the bit stream.

The parameters of the autoencoder encoding unit 511 and the autoencoder decoding unit 521 may be the parameters learned by FIG. See Figure 4 for details of the learned parameters.

According to one embodiment, a perceptually weighted error function can be utilized using the human auditory characteristics to provide improved audio signal quality with the same model complexity. Or can provide the same level of quality of audio signal with lower model complexity. Therefore, it can be utilized as an audio codec for audio signal compression and decompression.

6 is a diagram illustrating a learning process of a neural network applied to an audio signal encoding method using an audio signal encoding apparatus according to an embodiment.

In step 601, the audio signal encoding apparatus may generate a masking threshold for the first audio signal before being learned. Here, the first audio signal may represent a training audio signal for learning a neural network.

The masking threshold may be calculated by analyzing the auditory characteristics of the first audio signal. The psychoacoustic analysis method may include MPEG PAM-I or MPEG PAM-II and may include psychoacoustic analysis by other methods. For example, temporal masking as well as simultaneous masking effects can be used.

In step 602, the audio signal encoding apparatus may calculate a weighting matrix to be applied to the frequency components of each of the first audio signals, based on the generated masking threshold. At this time, the weighting matrix may include a weight to be applied to the frequency component of the first audio signal, and the weight may be set to be inversely proportional to the masking threshold for the first audio signal and proportional to the magnitude of the frequency component of each of the first audio signal. That is, the weight can be determined by the Signal-to-Mask Ratio (SMR), and the SMR can represent the ratio between the size and the size of the masking threshold in the frequency component of each of the first audio signals.

에 대한 가중 행렬의 가중치

는 대응하는 마스킹 임계치에 반비례하고 주파수 성분의 크기에 비례하도록 설정될 수 있다.For example, each frequency component of the first audio signal

≪ / RTI > weight < RTI ID =

May be set to be inversely proportional to the corresponding masking threshold and proportional to the magnitude of the frequency component.

In step 603, the audio signal encoding apparatus may generate a weighted error function by modifying a preset error function using a weighting matrix. Here, the weighted error function can be used to learn the parameters of the model by applying weights.

At this time, the model may include a neural network, for example, a neural network may include an autoencoder. The model may also include a topology, and the topology of the model may include an input layer, one or more hidden layers, an output layer, and nodes contained in each layer.

In step 604, the audio signal encoding apparatus may apply the learned parameter to the first audio signal using the weighted error function to generate the second audio signal. Here, the parameters of the model can be learned using a weighted error function. For example, learning can be performed on the topology of the initial model, and the predicted audio signal can be output using the topology of the model that has been learned by the iterative learning algorithm. Here, the predicted audio signal may represent the second audio signal.

At this time, the frequency spectrum of the second audio signal may be generated by applying the parameters of the model learned for the frequency spectrum of the first audio signal or the frequency spectrum of the first audio signal modified by the noise.

Comparing the second audio signal with the first audio signal, a perceptual quality assessment can be performed. The perceptual quality evaluation is based on objective evaluation or MOS (Mean Opinion Score), Multiple Stimuli with Hidden (MUSHRA), Perceptual Evaluation of Speech Quality (PESQ), Perceptual Objective Listening Quality Assessment (POLQA) Reference and Anchor) can be used. Perceptual quality assessment is not limited to this, and may include other quality assessments.

A perceptual quality assessment may indicate a quality assessment of the currently learned model. Based on the quality evaluation, it can be determined whether or not the model requirements such as the preset quality and the complexity of the model are satisfied. Further, based on the quality evaluation, it is possible to determine whether or not the model topology such as the complexity of the model can be adjusted. Here, the complexity of the model can have a positive correlation with the number of hidden layers and the number of nodes.

According to one embodiment, if the topology of the model is not adjustable or if it is determined that it is not necessary to adjust, the parameters of the currently learned model may be stored and the learning may be terminated. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, if it is determined that the topology of the model can be adjusted, the topology of the model may be updated and the process for learning described above may be repeated. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning described above can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning described above can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

According to one embodiment, based on the parameters of the learned model, signal processing may be performed on the first audio signal. At this time, the signal processing may include, but is not limited to, compression / noise cancellation / encoding / decoding.

7 is a diagram illustrating a method of encoding an audio signal performed by an audio signal encoding apparatus to which a neural network is applied, according to an embodiment.

In step 701, the audio signal encoding apparatus can receive the input audio signal. At this time, a model learned through the neural network can be applied to the audio signal encoding apparatus. Here, the neural network may include an autoencoder.

In step 702, the audio signal encoding apparatus can generate a reduced-size potential vector of the input audio signal based on the learned hidden layer parameters using the neural network. Here, the process of learning the parameters of the hidden layer is described in detail above.

According to one embodiment, the potential vector may be generated based on the learned parameters if adjustment of the topology of the model, including the number of hidden layers, the number of nodes, is impossible or not needed. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, the latent vector may be generated based on the re-learned parameter by applying the adjusted topology where adjustment of the topology of the model including the number of hidden layers, the number of nodes, is possible. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning the parameters can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning the parameters can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

In step 703, the audio signal encoding apparatus can output the bit stream by encoding the generated potential vector. Here, the potential vector may be quantized or coded to be transmitted through a transmission channel and output as a bitstream.

8 is a diagram illustrating a learning process of a neural network applied to an audio signal decoding method using an audio signal decoding apparatus according to an embodiment.

In step 801, the audio signal decoding device may generate a masking threshold for the first audio signal before being learned. Here, the first audio signal may represent a training audio signal for learning a neural network.

The masking threshold may be calculated by analyzing the auditory characteristics of the first audio signal. The psychoacoustic analysis method may include MPEG PAM-I or MPEG PAM-II and may include psychoacoustic analysis by other methods. For example, temporal masking as well as simultaneous masking effects can be used.

In step 802, the audio signal decoding apparatus may calculate a weighting matrix to be applied to the frequency components of each of the first audio signals, based on the generated masking threshold. At this time, the weighting matrix may include a weight to be applied to the frequency component of the first audio signal, and the weight may be set to be inversely proportional to the masking threshold for the first audio signal and proportional to the magnitude of the frequency component of each of the first audio signal. That is, the weight can be determined by the Signal-to-Mask Ratio (SMR), and the SMR can represent the ratio between the size and the size of the masking threshold in the frequency component of each of the first audio signals.

에 대한 가중 행렬의 가중치

는 대응하는 마스킹 임계치에 반비례하고 주파수 성분의 크기에 비례하도록 설정될 수 있다.For example, each frequency component of the first audio signal

≪ / RTI > weight < RTI ID =

May be set to be inversely proportional to the corresponding masking threshold and proportional to the magnitude of the frequency component.

In step 803, the audio signal decoding apparatus may generate a weighted error function by modifying a preset error function using a weighting matrix. Here, the weighted error function can be used to learn the parameters of the model by applying weights.

At this time, the model may include a neural network, for example, a neural network may include an autoencoder. The model may also include a topology, and the topology of the model may include an input layer, one or more hidden layers, an output layer, and nodes contained in each layer.

In step 804, the audio signal decoding apparatus may apply the learned parameter to the first audio signal using the weighted error function to generate the second audio signal. Here, the parameters of the model can be learned using a weighted error function. For example, learning can be performed on the topology of the initial model, and the predicted audio signal can be output using the topology of the model that has been learned by the iterative learning algorithm. Here, the predicted audio signal may represent the second audio signal.

At this time, the frequency spectrum of the second audio signal may be generated by applying the parameters of the model learned for the frequency spectrum of the first audio signal or the frequency spectrum of the first audio signal modified by the noise.

Comparing the second audio signal with the first audio signal, a perceptual quality assessment can be performed. The perceptual quality evaluation is based on objective evaluation or MOS (Mean Opinion Score), Multiple Stimuli with Hidden (MUSHRA), Perceptual Evaluation of Speech Quality (PESQ), Perceptual Objective Listening Quality Assessment (POLQA) Reference and Anchor) can be used. Perceptual quality assessment is not limited to this, and may include other quality assessments.

A perceptual quality assessment may indicate a quality assessment of the currently learned model. Based on the quality evaluation, it can be determined whether or not the model requirements such as the preset quality and the complexity of the model are satisfied. Further, based on the quality evaluation, it is possible to determine whether or not the model topology such as the complexity of the model can be adjusted. Here, the complexity of the model can have a positive correlation with the number of hidden layers and the number of nodes.

According to one embodiment, if the topology of the model is not adjustable or if it is determined that it is not necessary to adjust, the parameters of the currently learned model may be stored and the learning may be terminated. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, if it is determined that the topology of the model can be adjusted, the topology of the model may be updated and the process for learning described above may be repeated. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning described above can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning described above can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

According to one embodiment, signal processing such as compression / noise cancellation / encoding / decoding may be performed on the input audio signal based on the parameters of the learned model. At this time, the signal processing is not limited thereto.

According to one embodiment, based on the parameters of the learned model, signal processing may be performed on the first audio signal. At this time, the signal processing may include, but is not limited to, compression / noise cancellation / encoding / decoding.

9 is a diagram illustrating an audio signal decoding method performed by an audio signal decoding apparatus to which a neural network is applied, according to an embodiment.

In step 901, the audio signal decoding apparatus may receive the bitstream encoded with the latent vector generated by applying the learned parameters through the neural network to the input audio signal. At this time, the audio signal decoding apparatus can be applied with a learned model through a neural network. Here, the neural network may include an autoencoder.

In step 902, the audio signal decoding apparatus can recover the potential vector from the received bitstream. Here, a bitstream may be generated by quantizing or encoding a potential vector for transmission over a transmission channel.

In step 903, the audio signal decoding apparatus can decode the output audio signal from the reconstructed potential vector using the hidden layer to which the learned parameter is applied. Here, the process of learning the parameters of the hidden layer is described in detail above.

According to one embodiment, the potential vector may be generated based on the learned parameters if adjustment of the topology of the model, including the number of hidden layers, the number of nodes, is impossible or not needed. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, the latent vector may be generated based on the re-learned parameter by applying the adjusted topology where adjustment of the topology of the model including the number of hidden layers, the number of nodes, is possible. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning the parameters can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning the parameters can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

FIG. 10 is a diagram illustrating an audio signal encoding apparatus to which a neural network is applied, according to an embodiment.

The audio signal encoding apparatus 1000 may include a processor 1010 and a memory 1020. Memory 1020 may include one or more instructions executable by a processor.

The processor 1010 may receive an input audio signal. At this time, the processor 1010 of the audio signal encoding apparatus 1000 can apply the learned model through the neural network. Here, the neural network may include an autoencoder.

The processor 1010 of the audio signal encoding apparatus 1000 can generate a dimensionally reduced potential vector of the input audio signal based on the learned hidden layer parameter using the neural network. Here, the process of learning the parameters of the hidden layer is described in detail above.

According to one embodiment, the potential vector may be generated based on the learned parameters if adjustment of the topology of the model, including the number of hidden layers, the number of nodes, is impossible or not needed. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, the latent vector may be generated based on the re-learned parameter by applying the adjusted topology where adjustment of the topology of the model including the number of hidden layers, the number of nodes, is possible. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning the parameters can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning the parameters can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

The processor 1010 of the audio signal encoding apparatus 1000 can output the bitstream by encoding the generated potential vector. Here, the potential vector may be quantized or coded to be transmitted through a transmission channel and output as a bitstream.

11 is a block diagram illustrating an audio signal decoding apparatus to which a neural network is applied, according to an embodiment of the present invention.

The audio signal decoding apparatus 1100 may include a processor 1110 and a memory 1120. Memory 1120 may include one or more instructions executable by a processor.

The processor 1110 may apply the learned parameters to the input audio signal through the neural network to receive the bitstream encoded with the generated potential vector. At this time, the processor 1110 of the audio signal decoding apparatus 1100 can apply the learned model through the neural network. Here, the neural network may include an autoencoder.

The processor 1110 of the audio signal decoding apparatus 1100 can recover the potential vector from the received bit stream. Here, a bitstream may be generated by quantizing or encoding a potential vector for transmission over a transmission channel.

The processor 1110 of the audio signal decoding apparatus 1100 can decode the output audio signal from the reconstructed potential vector using the hidden layer to which the learned parameter is applied. Here, the process of learning the parameters of the hidden layer is described in detail above.

According to one embodiment, the potential vector may be generated based on the learned parameters if adjustment of the topology of the model, including the number of hidden layers, the number of nodes, is impossible or not needed. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

According to one embodiment, the latent vector may be generated based on the re-learned parameter by applying the adjusted topology where adjustment of the topology of the model including the number of hidden layers, the number of nodes, is possible. At this time, the determination as to whether or not the topology of the model can be adjusted depends on the field to which the model is applied and the requirements for the model.

For example, if the quality requirements are not met, the topology of the model can be adjusted so that the complexity of the model increases, and the process for learning the parameters can be repeated. Therefore, when judging whether or not the topology can be adjusted, if the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter can be re-learned using the model with increased complexity. Alternatively, if the quality requirements are not satisfied but the complexity of the model no longer increases, the parameters of the currently learned model may be stored and the learning terminated.

As another example, if the quality requirements are met, the topology of the model can be adjusted so that the complexity of the model is reduced within the quality requirements, and the process for learning the parameters can be repeated. Therefore, when determining whether the topology is adjustable, the parameters can be re-learned using a model with reduced complexity within the quality requirements if the outcome of the perceptual quality evaluation satisfies a predetermined quality requirement.

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; RTI ID = 0.0 > and / or < / RTI > equivalents,

110: input layer
120: Output layer
130: Hidden layer

Claims (20)

In a neural network applied to an audio signal encoding method using an audio signal encoding apparatus,
Generating a masking threshold for the first audio signal before being learned;
Calculating a weight matrix to be applied to a frequency component of each of the first audio signals based on the masking threshold;
Generating a weighted error function by modifying a preset error function using the weight matrix;
Applying the learned parameter to the first audio signal using the weighted error function to generate a second audio signal
≪ / RTI >
The method according to claim 1,
Wherein the weighting matrix includes a weight to be applied to a frequency component of the first audio signal,
The weighting value,
Wherein the first audio signal is set to be in proportion to a masking threshold for the first audio signal and to be proportional to a magnitude of a frequency component of each of the first audio signals.
The method according to claim 1,
The neural network includes:
And comparing the generated second audio signal with the first audio signal to perform a perceptual quality evaluation.
The method of claim 3,
The perceptual quality assessment may include:
Perceptual Evaluation of Speech Quality (PESQ), Perceptual Objective Listening Quality Assessment (POLQA), Perceptual Evaluation of Audio Quality (PEAQ)
A neural network that includes a subjective assessment of MOS (Mean Opinion Score), MUSHRA (Multiple Stimuli with Hidden Reference and Anchor).
The method of claim 3,
And determining whether or not the topology included in the model can be adjusted based on the perceptual quality evaluation.
6. The method of claim 5,
When determining whether the topology is adjustable,
If the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter is re-learned using the model with increased complexity,
Wherein if the result of the perceptual quality assessment satisfies a predetermined quality requirement, the neural network re-learns the parameter using a model with reduced complexity within the quality requirement.
1. An audio signal encoding method performed by an audio signal encoding apparatus to which a neural network is applied,
Receiving an input audio signal;
The neural network comprising at least one hidden layer and generating a dimensionally reduced potential vector of the input audio signal based on the learned hidden layer parameter using the neural network;
Encoding the generated potential vector and outputting a bit stream
And an audio signal encoding method.
8. The method of claim 7,
Wherein generating the dimensionally-reduced latent vector of the input audio signal based on the learned hidden layer parameters using the neural network comprises:
If the adjustment of the topology of the model including the number of hidden layers and the number of nodes is impossible or unnecessary, the potential vector is generated based on the learned parameter,
And if the topology of the model is adjustable, applying the adjusted topology to generate the potential vector based on the re-learned parameter.
8. The method of claim 7,
The coding of the latent vector may be performed by:
A method of encoding an audio signal that is binarized to transmit the bitstream over a channel.
In a neural network applied to an audio signal decoding method using an audio signal decoding apparatus,
Generating a masking threshold for the first audio signal before being learned;
Calculating a weight matrix to be applied to a frequency component of each of the first audio signals based on the masking threshold;
Generating a weighted error function by modifying a preset error function using the weight matrix;
Applying the learned parameter to the first audio signal using the weighted error function to generate a second audio signal
≪ / RTI >
11. The method of claim 10,
Wherein the weighting matrix includes a weight to be applied to a frequency component of the first audio signal,
The weighting value,
Wherein the first audio signal is set to be in proportion to a masking threshold for the first audio signal and to be proportional to a magnitude of a frequency component of each of the first audio signals.
11. The method of claim 10,
The neural network includes:
And comparing the generated second audio signal with the first audio signal to perform a perceptual quality evaluation.
13. The method of claim 12,
The perceptual quality assessment may include:
Perceptual Evaluation of Speech Quality (PESQ), Perceptual Objective Listening Quality Assessment (POLQA), Perceptual Evaluation of Audio Quality (PEAQ)
A neural network that includes a subjective assessment of MOS (Mean Opinion Score), MUSHRA (Multiple Stimuli with Hidden Reference and Anchor).
13. The method of claim 12,
And determining whether or not the topology included in the model can be adjusted based on the perceptual quality evaluation.
15. The method of claim 14,
When determining whether the topology is adjustable,
If the result of the perceptual quality evaluation does not satisfy the preset quality requirement, the parameter is re-learned using the model with increased complexity,
Wherein if the result of the perceptual quality assessment meets a predetermined quality requirement then the parameter is re-learned using a model with reduced complexity within the requirements of the quality.
A method of decoding an audio signal performed by an audio signal decoding apparatus to which a neural network is applied,
Receiving a latent vector-encoded bitstream generated by applying learned parameters through the neural network to an input audio signal;
Recovering the latent vector from the received bitstream;
Wherein the neural network includes one or more hidden layers and decodes the output audio signal from the restored latent vector using the hidden layer to which the learned parameter is applied
And decoding the audio signal.
17. The method of claim 16,
The generated potential vector may be,
If the topology including the number of hidden layers and the number of nodes belonging to the hidden layer is impossible or unnecessary, the potential vector is generated based on the learned parameter,
Wherein said potential vector is generated based on a re-learned parameter by applying an adjusted topology when said topology is adjustable.
17. The method of claim 16,
The coding of the latent vector may be performed by:
And binarizes the bit stream to transmit the bit stream over a channel.
An audio signal encoding apparatus to which a neural network is applied,
The audio signal encoding apparatus comprising a processor and a memory including one or more instructions executable by the processor,
If the one or more instructions are executed on the processor,
Receiving an input audio signal,
Wherein the neural network includes one or more hidden layers and generates a dimensionally reduced potential vector of the input audio signal based on the learned hidden layer parameter using the neural network,
And generating a bitstream by encoding the generated potential vector.
An audio signal decoding apparatus to which a neural network is applied,
The audio signal decoding apparatus comprising a processor and a memory including one or more instructions executable by the processor,
If the one or more instructions are executed on the processor,
A quantized bitstream generated by applying a parameter learned through the neural network to an input audio signal,
Restoring the potential vector from the received bitstream,
Wherein the neural network includes one or more hidden layers and decodes an output audio signal from the restored latent vector using the hidden layer to which the learned parameter is applied.

Images