KR102539165B1 - Residual coding method of linear prediction coding coefficient based on collaborative quantization, and computing device for performing the method - Google Patents

Abstract

A method for coding a residual signal of LPC coefficients based on cooperative quantization and a computing device performing the method are disclosed. The residual signal coding method includes performing LPC (Linear Prediction Coding) analysis and quantization on input speech to generate coded LPC coefficients and LPC residual signals; determining a predicted LPC residual signal by applying the LPC residual signal to cross-module residual learning; performing LPC synthesis using the encoded LPC coefficient and the predicted LPC residual signal; A step of determining an output voice that is a synthesized output according to a result of performing the LPC synthesis may be included.

Description

A method for coding a residual signal of LPC coefficients based on cooperative quantization and a computing device performing the method

The present invention relates to a method for coding a residual signal of LPC coefficients based on cooperative quantization and a computing device performing the method.

Speech coding refers to a method of quantizing a speech signal into a low bit stream for efficient transmission and storage in a communication system. The design of speech codecs addresses the disadvantages of low bit rate, high perceptual quality, low complexity and delay.

Most speech codecs can be classified into vocoders and waveform coders. Vocoders do not use parameters to model the human voice production process, such as vocal, pitch frequency, etc. However, a waveform coder can compress and reconstruct the waveform to make the decoded speech “perceptually” similar to the input speech.

Conventional vocoders are computationally efficient and can encode speech at very low bit rates, whereas waveform coders support a much wider bit rate range with scalable performance and are effective against noise.

Linear Predictive Coding (LPC), which is an all pole linear filter in both conventional vocoders and waveform coders, can efficiently model the power spectrum with only a few coefficients. For a vocoder, the LPC residual is modeled as a synthetic excitation signal using a pitch pulse train or white noise component. In the case of a waveform coder, on the other hand, the residual signal can be directly compressed to the desired bit rate before being synthesized into a decoded signal.

LPC is also useful in modern neural speech codecs. Autoregressive models can greatly improve the quality of synthesized speech, but introduce model complexity during the decoding process.

The present invention relates to a method for coding a speech signal using LPC coefficients and a stepwise autoencoder, and in particular, provides a method and apparatus for a structure and training method for simultaneously optimizing quantization of LPC coefficients and quantization of LPC residual signals. .

The present invention proposes a structure and training method capable of optimizing both LPC coefficients and autoencoders connected in stages.

In a method for coding residual signals of LPC coefficients performed by a computing device according to an embodiment of the present invention, the computing device performs LPC (Linear Prediction Coding) analysis and quantization on an input speech to generate coded LPC coefficients and LPC residual signals. generating; determining a predicted LPC residual signal by applying the LPC residual signal to cross-module residual learning; performing LPC synthesis using the encoded LPC coefficient and the predicted LPC residual signal; A step of determining an output voice that is a synthesized output according to a result of performing the LPC synthesis may be included.

The cross-module residual learning may include applying a high-pass filter to an input speech; applying a pre-emphasis filter to the result of applying the high-pass filter; determining an LPC coefficient from a result of applying the pre-emphasis filter; quantizing the LPC coefficients to generate a soft allocation matrix of the coded LPC coefficients and softmax; and determining an LPC residual signal based on a result of applying the pre-emphasis filter and a result of quantizing the LPC coefficient.

The determining of the LPC coefficients may include performing cross-frame windowing by applying a window to all frames of the input speech to which the pre-emphasis filter is applied; performing sub-frame windowing by applying a window to a plurality of sub-frames corresponding to a middle region among all frames of the input voice in a result of performing the cross-frame windowing; and performing composite windowing by overlapping a result of performing the subframe windowing.

The LPC coefficients can be quantized by applying a trainable softmax to the LPC coefficients in the LSP domain.

The LPC residual signal may be encoded by 1D-CNN autoencoders.

The autoencoders of the 1D-CNN can be sequentially trained by using a residual signal output from a previous autoencoder as an input of a next autoencoder.

In the auto-encoders of the 1D-CNN, differential coding may be applied to an output of the autoencoder, and differential coding may be applied to an output of the autoencoder based on a code length for each frame of the autoencoder.

A computing device performing a method for coding a residual signal of LPC coefficients according to an embodiment of the present invention includes a processor, and the processor performs LPC (Linear Prediction Coding) analysis and quantization on an input speech, A coded LPC coefficient and an LPC residual signal are generated, a predicted LPC residual signal is determined by applying the LPC residual signal to cross-module residual learning, and an LPC is synthesized using the coded LPC coefficient and the predicted LPC residual signal. , and an output voice, which is a synthesized output, may be determined according to a result of performing the LPC synthesis.

The processor applies a high-pass filter to an input voice, applies a pre-emphasis filter to a result of applying the high-pass filter, determines LPC coefficients from a result of applying the pre-emphasis filter, and quantizes the LPC coefficients. Cross-module residual learning may be performed in which a soft allocation matrix of encoded LPC coefficients and softmax is generated, and an LPC residual signal is determined based on a result of applying the pre-emphasis filter and a result of quantizing the LPC coefficient.

The processor performs cross-frame windowing by applying a window to all frames of the input speech to which the pre-emphasis filter is applied to determine LPC coefficients, and as a result of performing the cross-frame windowing, the entirety of the input speech Sub-frame windowing may be performed by applying a window to a plurality of sub-frames corresponding to a middle region among frames, and composition windowing may be performed by overlapping a result of the sub-frame windowing.

The LPC coefficients can be quantized by applying a trainable softmax to the LPC coefficients in the LSP domain.

The LPC residual signal may be encoded by 1D-CNN autoencoders.

The autoencoders of the 1D-CNN can be sequentially trained by using a residual signal output from a previous autoencoder as an input of a next autoencoder.

In the auto-encoders of the 1D-CNN, differential coding may be applied to an output of the autoencoder, and differential coding may be applied to an output of the autoencoder based on a code length for each frame of the autoencoder.

According to an embodiment of the present invention, it is possible to provide a structure and training method for simultaneously optimizing quantization of LPC coefficients and quantization of LPC residual signals by coding a speech signal using LPC coefficients and a stepwise autoencoder.

According to one embodiment of the present invention, it is possible to provide a structure and training method capable of optimizing both LPC coefficients and an autoencoder connected in stages.

1 is a diagram illustrating a method of coding a residual signal of LPC coefficients based on cooperative quantization according to an embodiment of the present invention.
2 is a diagram showing a method of analyzing trainable LPC coefficients according to an embodiment of the present invention.
3 is a diagram illustrating a softmax quantization process according to an embodiment of the present invention.
4 is a diagram for explaining a process of LPC windowing according to an embodiment of the present invention.
5 is a diagram for explaining a process of cross-module residual training (CMRL) according to an embodiment of the present invention.
6 is a diagram illustrating a residual signal coding process of cross-module residual training (CMRL) according to an embodiment of the present invention.
7 is a diagram for explaining centralized distribution of differential coding through coding of a residual signal according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these examples. Like reference numerals in each figure indicate like elements.

Various changes may be made to the embodiments described below. The embodiments described below are not intended to be limiting on the embodiments, and should be understood to include all modifications, equivalents or substitutes thereto.

Terms such as first or second may be used to describe various components, but these terms should only be understood for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

Terms used in the examples are used only to describe specific examples, and are not intended to limit the examples. Expressions in the singular number include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a method of coding a residual signal of LPC coefficients based on cooperative quantization according to an embodiment of the present invention.

According to one embodiment of the present invention, in order to better integrate neural networks and LPCs for scalable waveform coding with low model complexity, collaborative quantization capable of training LPC quantization is proposed. By incorporating different autoencoding modules for the coding of the LPC residual signal, cooperative quantization can learn the optimal bit assignment between the LPC coefficients and the code layer of another neural network. According to the cooperative quantization learning scheme proposed in the present invention, cooperative quantization has improved performance compared to previous schemes and can be extended to match the performance of the latest codec of 24 kbps while having low complexity.

LPC is useful for modern neural speech codecs and can unload computational overhead from neural networks. In addition, cross module residual learning, which is a neural waveform coder, uses LPC as a pre-processor and can model LPC residual signals to match the latest voice quality.

Scalability and efficiency are required in a neural speech codec, which supports a wide range of bit rates for applications in various devices. To this end, according to an embodiment of the present invention, cooperative quantization for jointly learning a codebook of LPC coefficients and residual signals is applied.

Cooperative quantization according to an embodiment of the present invention suggests a digital signal processing method specialized for a domain. According to cooperative quantization, it can be seen that model complexity is much lower while achieving much higher quality at 9 kbps than conventional quantization methods. We also show that cooperative quantization can extend up to 24 kbps, which is superior to AMR-WB and Opus. Cooperative quantization is a neural waveform codec and has much smaller parameters than existing models.

Referring to FIG. 1 , a method of coding a residual signal of LPC coefficients may be performed by a computing device. The computing device may perform the residual signal coding method through the following process.

In step (1), the computing device may receive input speech and perform LPC analysis and quantization. Then, the computing device may output the LPC residual signal and the LPC coefficients through step (1).

At step (2), the computing device may learn the LPC residual signal. In one example, a computing device may learn an LPC residual signal based on cross-module residual learning (CMRL). As a result of learning the LPC residual signal, predicted LPC residual signals may be output. The operation of cross-module residual learning will be described in detail with reference to FIGS. 5 and 6 below.

In step (3), the computing device may perform LPC dequantization and LPC synthesis using the LPC coefficients and the LPC residual signal.

In step (4), the computing device may determine an output speech, which is a synthesized output, by applying de-emphasis filtering to the output result of the LPC synthesis.

2 is a diagram showing a method of analyzing trainable LPC coefficients according to an embodiment of the present invention.

FIG. 2 describes a specific process for LPC analysis and quantization described in step (1) of FIG. 1. According to one embodiment of the present invention, quantization of LPC coefficients can be applied to a neural network training algorithm by integrating LPC analysis into a cross-module residual learning pipeline. The overall process of LPC is based on AMR-WB.

FIG. 2 is performed by the computing device described in FIG. 1 . In step (1) of FIG. 2, the computing device may apply a high-pass filter to the input speech. And, in step (2), the computing device may additionally apply a pre-emphasis filter to the result of applying the high pass to the input voice.

로 설정될 수 있으며, 고주파수에서 아티팩트를 제거하기 위해 사용된다.As an example, the high-pass filter may include a high-pass filter having a cut-off frequency of 50 Hz. And, the pre-emphasis filter is

Can be set to , and is used to remove artifacts at high frequencies.

In step (3) of FIG. 2, the computing device may determine LPC coefficients. In step (2), the input speech to which the pre-emphasis filter is applied may be divided into a plurality of frames. For example, an input voice may be divided into 1024 frames.

Before the LPC coefficient is determined, each of a plurality of divided frames from the input speech may be windowed. A process of processing a window will be described in detail with reference to FIG. 4 .

In step 4 of Figure 2, the computing device may quantize the LPC coefficients. In one example, the computing device may apply trainable softmax quantization to the LPC coefficients in the LSP domain so that each LPC coefficient represents its nearest centroid. Softmax quantization will be described in detail with reference to FIG. 3 .

로 표현된다. LPC에 특화된 중심인

은 학습될 필요가 있으며, 소프트 할당 매트릭스를 구성하기 위해 사용될 수 있다.For each frame x in which the window is processed, the LPC coefficients represented in the LSP domain are

is expressed as A center specializing in LPC

needs to be learned and can be used to construct a soft allocation matrix.

For example, in the present invention, the order of LPC coefficients may be set to 16, and the number of centers may be set to 256 (eg, 8 bits). The size of the soft allocation matrix and the hard allocation matrix is 16*256. And, the columns of the soft allocation matrix are probability vectors, and the columns of the hard allocation matrix are one-hot vectors.

Meanwhile, in step (4) of FIG. 2, the computing device may determine the coded LPC coefficients and the soft allocation matrix A _soft .

In step (5) of FIG. 2, the computing device may determine an LPC residual signal using the quantized LPC coefficients and the input speech to which the pre-emphasis filter is applied in step (2).

<residual coding>

The LPC residual signal calculated in step (1) of FIG. 1 may be compressed by 1D-CNN autoencoders. Here, the autoencoders of the 1D-CNN are the autoencoders described in FIGS. 5 and 6.

에 차등 코딩이 적용될 수 있다. 여기서, m은 각각의 오토인코더들의 프레임별 코드의 길이를 의미한다. 소프트맥스 양자화의 입력 스칼라는

이다.The output of the autoencoder is

Differential coding may be applied to Here, m means the code length for each frame of each autoencoder. The input scalar of softmax quantization is

am.

Softmax quantization starts from distribution of codes expressed as more centralized real values as shown in FIG. 7 . As shown in Fig. 1, quantization of LPC coefficients and residual coding of cross-module residual learning are optimized together. The LPC analysis is to find a pivot that not only minimizes the energy of the residual signal as much as possible, but also easily performs residual compression from the modules of the next cross-module residual learning.

3 is a diagram illustrating a softmax quantization process according to an embodiment of the present invention.

To compress speech signals, the core element of an autoencoder is a trainable quantizer. A trainable quantizer learns a discrete representation of an autoencoder's code layer. A quantization scheme suitable for neural networks, such as soft-hard quantization, is called softmax quantization in end-to-end speech coding.

에 대해, 인코더의 출력은

로 결정된다. 인코더의 출력들 각각은 16비트의 부동소수점 값을 나타낸다. 벡터

로 표현되는

중심(centroid)들이 주어질 때, 소프트맥스 양자화는

에서 각각의 샘플들을

중심들 중 어느 하나로 맵핑할 수 있다. 그리고, 각각의 양자화된 샘플은 log₂J 비트들로 표현될 수 있다. 예를 들어, J가 32일 때 5비트가 될 수 있다.Input frame of S samples

, the output of the encoder is

is determined by Each of the encoder's outputs represents a 16-bit floating point value. vector

expressed as

Given centroids, softmax quantization is

each sample in

You can map to any of the centroids. And, each quantized sample can be represented by log ₂ J bits. For example, when J is 32, it can be 5 bits.

를 사용한다. 여기서, I는 중심들의 코드의 차원을 의미하고, J는 중심들의 벡터의 차원을 의미한다. 하드 할당 매트릭스는 유클리디안 거리 매트릭스

에 기초하여 수학식 1에 의해 결정된다.The quantization process of Softmax is a hard assignment matrix.

Use Here, I means the dimension of the code of centroids, and J means the dimension of the vector of centroids. The hard assignment matrix is the Euclidean distance matrix

It is determined by Equation 1 based on

의 요소들

각각에 대해 가장 가까운 중심을 할당할 수 있다. 이 과정은 차별화(differentiable)되지 않으며, 훈련동안 역확산 오류 흐름(backpropagation error flow)를 차단한다.Quantization of softmax

elements of

You can assign the closest centroid to each. This process is non-differentiable and blocks the backpropagation error flow during training.

Instead, soft assignments are used during training as follows.

를 계산할 수 있다.(i) the computing device is a Euclidean distance matrix between the elements of h and b

can be calculated.

를 이용하여 비유사도 매트릭스로부터 소프트 할당 매트릭스를 계산할 수 있다. 여기서, 소프트맥스 함수는 소프트 할당 매트릭스의 각 열(tow)에 적용하여 확률 벡터

로 변경할 수 있다. 확률 벡터는

에 가장 유사한 가장 높은 확률값을 홀딩한다. 훈련동안

는 하드 할당들(hard assignment)을 근사화하고, 근사화된 결과는 입력 코드로서 디코더에 제공된다.(ii) the computing device is a softmax function

The soft assignment matrix can be calculated from the dissimilarity matrix using Here, the softmax function is applied to each column (tow) of the soft allocation matrix to generate a probability vector.

can be changed to The probability vector is

Hold the highest probability value most similar to . during training

approximates the hard assignments, and the approximated result is provided to the decoder as an input code.

는

와 같이 소프트맥스 함수의 소프트함(softness)을 제어한다. 소프트 할당 매트릭스

와 하드 할당 매트릭스

간의 갭(gap)이 최소화가 되도록

는 300으로 설정될 수 있다.additional variables

Is

Controls the softness of the softmax function. soft allocation matrix

with hard allocation matrix

to minimize the gap between

may be set to 300.

는 열에서 가장 큰 확률값을 제로로 변경함으로써

를 교체한다.

는 양자화된 코드

를 생성한다.(iii) at test time;

by changing the largest probability value in the column to zero

replace

is the quantized code

generate

4 is a diagram for explaining a process of LPC windowing according to an embodiment of the present invention.

In FIG. 2 , an input voice to which pre-emphasis filtering is applied may be divided into a plurality of frames. As an example, an input speech may be divided into frames of 1024 sample points. Before LPC coefficients are calculated in the input speech, a window is processed in each frame of the input speech to perform LPC windowing.

As can be seen in FIG. 4, a symmetric window may have an emphasized weight in the middle 50% area. And, the symmetrical window is the left half of the Hann window having 512 sample points in the first 25% area and the right half of the Hann window having 512 sample points in the remaining 25% area.

And, LPC is performed on the window-processed frame in the time domain s. The result of predicting the tth sample is determined by Equation 2 below.

는 t번째 샘플의 예측을 의미하고,

는 i번째 LPC 계수를 의미한다. 프레임들은 50%만큼 오버랩된다. LPC 차수는 16차로 설정될 수 있다. 일례로, LPC 계수는 Levinson Durbin 알고리즘에 기초하여 결정되고, 이 알고리즘은 양자화에 강인한 LSP(line spectral pair)로 표시될 수 있다.

Means the prediction of the tth sample,

denotes the i th LPC coefficient. The frames overlap by 50%. The LPC order may be set to the 16th order. As an example, the LPC coefficients are determined based on the Levinson Durbin algorithm, which can be expressed as a line spectral pair (LSP) that is robust to quantization.

According to one embodiment of the present invention, windowing of sub-frames is applied to calculate the LPC residual signal. As an example, (a) of FIG. 4 shows windowing for cross frames, (b) shows windowing for subframes, and (c) shows composite windowing. The computing device separately calculates a residual signal of each subframe for the divided speech frames and quantized LPC coefficients from the input speech.

At this time, in the middle 50% of the frame of 1024 sample points (Fig. 4(a)) (eg, [256:768] for the first analysis frame [0:1024], and [512:1536] for the second analysis frame) [768:1280]) can be divided into 7 subframes ((b) in FIG. 4). Each of the 7 subframes has a size of 128 sample points and overlaps between frames by 50%. As shown in (b) of FIG. 4, 5 subframes in the middle among 7 subframes may be window processed by the Hann function. Also, among the seven subframes, the first subframe and the last subframe may be asymmetrically windowed.

The LPC residual signal can be calculated for 7 subframes corresponding to 512 sample points in the middle region, 50% of the total frame having 1024 sample points. If an overlap of analysis frames by 50% occurs between subframes, there is no overlap between residual segments.

5 is a diagram for explaining a process of cross-module residual training (CMRL) according to an embodiment of the present invention.

<End-to-end speech coding autoencoders>

The 1D-CNN structure in time domain samples provides the desired autoencoder for end-to-end speech coding. As illustrated in Table 1, in the autoencoder, the encoder part consists of four ResNet stages, the downsampling convolution layer is halved to a feature map in the middle, and the channel compression layer is a 256-dimensional real number. Form real-valued code. Table 1 may correspond to the structure of the autoencoder included in the cross-module residual training of FIG. 5 . In an autoencoder, the decoder part has a mirrored structure of the encoder part. However, the upsampling layer in the decoder part may restore the original frame size (512 sample points) from the reduced code length (256 sample points).

In the structure of 1D-CNN autoencoder, the input tensor and output tensor are represented by width and channel, but the kernel shape is expressed by width, input channel, and out channel. is expressed

In the pipeline of cross-module residual training, the LPC coding module provides a pre-processor with a fixed bit rate of 2.4 kbps. It can effectively model the spectral envelope, but may not help with quantization of the residual signal. For example, if LPC doesn't model effectively for a frame, cooperative quantization can weight the next autoencoder more heavily to use more bits.

According to one embodiment of the present invention, a trainable quantization module capable of restoring an LPC residual signal together with other autoencoder modules in cross-module residual training can be generated by dividing the LPC process.

Referring to FIG. 5 , an LPC residual signal is generated by LPC analysis and quantization of a speech signal. And, the LPC residual signal is applied to cross-module residual training. Cross-module residual training basically has a structure in which an autoencoder structure and softmax quantization are combined. In cross-module residual training, the LPC residual signal is dimensionally reduced in the encoder part of the autoencoder and then softmax quantization is applied. Then, the result of softmax quantization is applied to the decoder part of the autoencoder again, and the original LPC residual signal is restored by extending the dimension.

That is, cross-module residual training can code the residual signal obtained by LPC-filtering the speech signal with an autoencoder having a CMRL structure. In this case, bits allocated to LPC quantization and bits allocated to LPC residual coding may be independent of each other. By allowing LPC quantization to be trained, the performance of the speech codec can be improved by adjusting bits allocated to LPC quantization and quantization of the LPC residual signal according to the characteristics of the speech signal.

6 is a diagram illustrating a residual signal coding process of cross-module residual training (CMRL) according to an embodiment of the present invention.

The cross-module residual training of FIG. 6 is an example of residual signal learning in step (2) of FIG. 1 .

Referring to FIG. 6 , cross module residual learning is serialization of an autoencoder list so that residual training is possible between building block modules of the autoencoder. Cross-module residual training serializes the building block modules of an autoencoder rather than relying on a single autoencoder.

으로부터 출력 신호

를 생성한다. 이 때, i-1번째 오토인코더(601)은 입력 신호와 출력 신호가 서로 유사하도록 훈련될 수 있으며, i-1번째 오토인코더(601)의 입력 신호와 출력 신호의 차이는 잔차 신호로서 i번째 오토 인코더(602)의 입력으로 설정될 수 있다. 즉, i번째 오토 인코더(602)의 입력인

는 i-1번째 오토인코더(601)의 입력 신호와 출력 신호의 차이인

-

로 결정될 수 있다.Referring to FIG. 6 , an i−1 th autoencoder 601, an i th autoencoder 602, and an i+1 th autoencoder 603 may be connected in series. And, the i-1th autoencoder 601 is an input signal

output signal from

generate In this case, the i-1 th autoencoder 601 can be trained so that the input signal and output signal are similar to each other, and the difference between the input signal and the output signal of the i-1 th autoencoder 601 is a residual signal, i th It can be set as an input of the auto-encoder 602. That is, the input of the i-th auto-encoder 602

Is the difference between the input signal and the output signal of the i-1th autoencoder 601

-

can be determined by

를 입력받고,

를 예측하도록 훈련할 수 있다. 가장 처음에 배치된 오토인코더를 제외하고, i번째 오토인코더의 입력인

는 잔차 신호이거나 또는 이전에 배치된 오토인코더들에 의해 재구성되지 않은 잔차 신호의 합계와 입력 음성인

간의 차이일 수 있다.

는 하기 수학식 3에 의해 결정될 수 있다.Referring to FIG. 6, the i-th autoencoder 602 is

input,

can be trained to predict Except for the first placed autoencoder, the input of the ith autoencoder

is the residual signal or the sum of the residual signals not reconstructed by previously placed autoencoders and the input speech

may be the difference between

Can be determined by Equation 3 below.

Cross-module residual training distributes the effort to optimize one neural network. Cross-module residual training can make neural audio coding algorithms more suitable for user terminals with limited energy supply and storage space by lowering the model complexity in terms of learnable parameters.

According to the cross-module residual training pipeline, each autoencoder can be trained sequentially by using the residual signal of the previous module as the input of the current module. When all autoencoders are trained, a fine-tuning process is performed to improve the overall reconstruction quality.

A loss function used in training of each of the autoencoders is composed of a reconstruction error and a regularizer. The loss function is determined by Equation 4

는 MSE(mean squared error)로 측정된다.

는 멜 스케일(mel-scale) 주파수 도메인에서 손실 함수를 측정함으로써 비지각적인

에 의해 캡쳐되지 않도록 보상한다. 4개의 멜 필터(mel-filter) 뱅크들은 128, 32, 16 및 8의 사이즈로 특정되고, 이는 coarse-to-ne differentiation이 가능하도록 한다.When the input of cooperative quantization is given in the time domain, it is required to minimize the loss function in the time domain and frequency domain. time domain error

is measured as the mean squared error (MSE).

is non-perceptual by measuring the loss function in the mel-scale frequency domain.

compensates for not being captured by Four mel-filter banks are specified with sizes of 128, 32, 16 and 8, which allows coarse-to-ne differentiation.

와

는 소프트맥스 양자화를 위한 레귤라이저이다. 소프트 할당 매트릭스

는 도 3에서 이미 설명되었다.

는 소프트 할당 매트릭스가 보다 더 하드 할당 매트릭스에 가깝도록 보장할 수 있는

로 설명된다.In Equation 4,

and

is a regulator for softmax quantization. soft allocation matrix

has already been described in FIG. 3 .

can ensure that the soft allocation matrix is closer to the hard allocation matrix.

is explained by

는 비트율을 제어하기 위해 소프트맥스 양자화된 비트 스트링의 엔트로피를 계산할 수 있다. 먼저, 각 커널의 주파수는 수학식 5에 따라 소프트 할당 매트릭스의 열들을 합산함으로써 계산된다.

can calculate the entropy of the softmax quantized bit string to control the bit rate. First, the frequency of each kernel is calculated by summing the columns of the soft allocation matrix according to Equation 5.

는 얼마나 자주 코드들이 각 커널에 할당되는지를 나타내며, 수학식 6과 같이 결정된다Probability distribution of kernels

indicates how often codes are allocated to each kernel, and is determined as in Equation 6

And, entropy is defined as in Equation 7.

가 조절됨으로써 모델은 원하는 비트율의 범위로 미세조정된다. 그리고, 허프만 코딩을 그룹화된 샘플 쌍들(쌍별로 2개의 인접한 샘플들)에 적용하는 것은 좀더 높은 압축율을 제공한다.

By adjusting , the model is fine-tuned to a desired bit rate range. And, applying Huffman coding to grouped sample pairs (two adjacent samples per pair) provides a higher compression ratio.

7 is a diagram for explaining centralized distribution of differential coding through coding of a residual signal according to an embodiment of the present invention.

The present invention proposes a more simplified and scalable waveform neural codec. In cooperative quantization, LPC coefficient quantization becomes a trainable element to be optimally combined with residual quantization.

Meanwhile, the method according to the present invention is written as a program that can be executed on a computer and can be implemented in various recording media such as magnetic storage media, optical reading media, and digital storage media.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or combinations thereof. Implementations may be a computer program product, i.e., an information carrier, e.g., a machine-readable storage, for processing by, or for controlling, the operation of a data processing apparatus, e.g., a programmable processor, computer, or plurality of computers. It can be implemented as a computer program tangibly embodied in a device (computer readable medium) or a radio signal. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be written as a stand-alone program or in a module, component, subroutine, or computing environment. It can be deployed in any form, including as other units suitable for the use of. A computer program can be deployed to be processed on one computer or multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Processors suitable for processing a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from read only memory or random access memory or both. Elements of a computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. In general, a computer may include, receive data from, send data to, or both, one or more mass storage devices that store data, such as magnetic, magneto-optical disks, or optical disks. It can also be combined to become. Information carriers suitable for embodying computer program instructions and data include, for example, semiconductor memory devices, for example, magnetic media such as hard disks, floppy disks and magnetic tapes, compact disk read only memory (CD-ROM) ), optical media such as DVD (Digital Video Disk), magneto-optical media such as Floptical Disk, ROM (Read Only Memory), RAM (RAM) , Random Access Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and the like. The processor and memory may be supplemented by, or included in, special purpose logic circuitry.

In addition, computer readable media may be any available media that can be accessed by a computer, and may include both computer storage media and transmission media.

Although this specification contains many specific implementation details, they should not be construed as limiting on the scope of any invention or what is claimed, but rather as a description of features that may be unique to a particular embodiment of a particular invention. It should be understood. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Further, while features may operate in particular combinations and are initially depicted as such claimed, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination is a subcombination. or sub-combination variations.

Similarly, while actions are depicted in the drawings in a particular order, it should not be construed as requiring that those actions be performed in the specific order shown or in the sequential order, or that all depicted actions must be performed to obtain desired results. In certain cases, multitasking and parallel processing can be advantageous. Further, the separation of various device components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and devices described may generally be integrated together into a single software product or packaged into multiple software products. You have to understand that you can.

On the other hand, the embodiments of the present invention disclosed in this specification and drawings are only presented as specific examples to aid understanding, and are not intended to limit the scope of the present invention. In addition to the embodiments disclosed herein, it is obvious to those skilled in the art that other modified examples based on the technical idea of the present invention can be implemented.

Claims (14)

A method for coding a residual signal of LPC coefficients performed by a computing device,
Generating, by a computing device, LPC (Linear Prediction Coding) analysis and quantization on input speech to generate coded LPC coefficients and LPC residual signals;
determining a predicted LPC residual signal by applying the LPC residual signal to cross-module residual learning;
performing LPC synthesis using the encoded LPC coefficient and the predicted LPC residual signal;
Determining an output voice that is a synthesized output according to a result of performing the LPC synthesis
including,
The cross-module residual learning,
applying a high-pass filter to the input voice;
applying a pre-emphasis filter to the result of applying the high-pass filter;
determining an LPC coefficient from a result of applying the pre-emphasis filter;
quantizing the LPC coefficients to generate a soft allocation matrix of the coded LPC coefficients and softmax; and
Determining an LPC residual signal based on a result of applying the pre-emphasis filter and a result of quantizing the LPC coefficient
including,
Determining the LPC coefficient,
performing cross-frame windowing by applying a window to all frames of the input speech to which the pre-emphasis filter is applied;
performing sub-frame windowing by applying a window to a plurality of sub-frames corresponding to a middle region among all frames of the input voice in a result of performing the cross-frame windowing;
Performing composite windowing by overlapping a result of performing the subframe windowing
Residual signal coding method comprising a. delete delete According to claim 1,
The LPC coefficients may be quantized by applying a trainable softmax to LPC coefficients in the LSP domain. According to claim 1,
The LPC residual signal is encoded by 1D-CNN autoencoders. According to claim 5,
The autoencoders of the 1D-CNN,
A residual signal coding method in which the residual signal, which is the output of the previous autoencoder, is sequentially trained by being used as the input of the next autoencoder. According to claim 5,
Auto encoders of the 1D-CNN,
Differential coding is applied to the output of the autoencoder,
The residual signal coding method of applying differential coding to the output of the autoencoder based on the length of a code for each frame of the autoencoder. A computing device performing a residual signal coding method of LPC coefficients,
The computing device includes a processor;
The processor performs LPC (Linear Prediction Coding) analysis and quantization on the input speech to generate coded LPC coefficients and LPC residual signals,
Applying the LPC residual signal to cross-module residual learning to determine a predicted LPC residual signal;
Performing LPC synthesis using the encoded LPC coefficient and the predicted LPC residual signal;
Determining an output voice, which is a synthesized output, according to a result of performing the LPC synthesis;
the processor,
apply a high-pass filter to the input voice;
Applying a pre-emphasis filter to the result of applying the high-pass filter;
Determine an LPC coefficient from the result of applying the pre-emphasis filter,
quantizing the LPC coefficients to generate a soft allocation matrix of coded LPC coefficients and softmax;
Performing cross-module residual learning for determining an LPC residual signal based on a result of applying the pre-emphasis filter and a result of quantizing the LPC coefficient;
the processor,
The processor, to determine the LPC coefficient,
Performing cross-frame windowing by applying a window to an entire frame of the input speech to which the pre-emphasis filter is applied;
Performing sub-frame windowing by applying a window to a plurality of sub-frames corresponding to a middle region among all frames of the input voice in the result of performing the cross-frame windowing;
A computing device for performing composite windowing by performing overlap on a result of performing the sub-frame windowing. delete delete According to claim 8,
The LPC coefficients may be quantized by applying a trainable softmax to the LPC coefficients in the LSP domain. According to claim 8,
The LPC residual signal is encoded by autoencoders of 1D-CNN. According to claim 12,
The autoencoders of the 1D-CNN,
A computing device that is sequentially trained by using the residual signal, the output of the previous autoencoder, as the input of the next autoencoder. According to claim 12,
Auto encoders of the 1D-CNN,
Differential coding is applied to the output of the autoencoder,
The output of the autoencoder is a computing device to which differential coding is applied based on the length of a code for each frame of the autoencoder.

Images