JP2023175767A - Device and method for hostile blind bandwidth extension of end-to-end using one or more convolutional networks and/or recurrent network - Google Patents

Abstract

To provide a device and method for processing a narrow band voice input signal and obtaining a wide band voice output signal by extending a band of the narrow band voice input signal.SOLUTION: A device for processing a narrow band voice input signal and obtaining a wide band voice output signal comprises: a signal envelope extrapolator (120) including a first neural network (125) for inputting a plurality of samples of a signal envelope of the narrow band voice input signal and outputting a plurality of extrapolated samples of the signal envelope; an excitation signal extrapolator (130) for inputting a plurality of samples of excitation signals of the narrow band voice input signal and outputting a plurality of extrapolated excitation signal samples; and a combiner (140) for generating the wide band voice output signal so that the wide band voice output signal extends a bandwidth for the narrow band voice input signal by depending on the plurality of extrapolated samples of the signal envelope and the plurality of extrapolated excitation signal samples.SELECTED DRAWING: Figure 1

Description

Application for application of Article 30, Paragraph 2 of the Patent Act Website publication date: December 18, 2020 Website address: https://ieeexplore. ieee. org/document/9287465

Description The present invention relates to an apparatus and method for end-to-end adversarial blind bandwidth expansion using one or more convolutional and/or recurrent networks.

Voice communication is a technology that most people use every day, creating vast amounts of data that need to be transmitted over VoIP (Voice over Internet Protocol), mobile phone, and public switched telephone networks. There is. According to a 2017 OFCOM study, an average of 156.75 minutes of calls are made to mobile phones per subscription each month (https://www.ofcom.org.uk/research-and-data/multi- see sector-research/cmr/cmr-2018/interactive).

Even if the amount of data transferred is small, it is desirable that the quality of the audio is high. To achieve this goal, speech compression techniques have evolved over the past decades from compressing band-limited speech with simple pulse code modulation [1], to following models of speech production and human perception that can code full-band speech. It has evolved into coding schemes [2] and [3]. Despite the existence of such standardized voice codecs, their adoption in mobile phones or public switched telephone networks will take years, if not decades. For these reasons, AMR-NB [4] is most frequently used as a codec for mobile phone voice communications that only encodes frequencies from 200 Hz to 3400 Hz (commonly referred to as narrowband, NB). However, transmitting band-limited audio impairs not only the acoustic quality but also the intelligibility [5], [6], [7]. Blind band extension (BBWE), also known as artificial band extension or audio super-resolution, artificially reproduces missing frequency components without transmitting additional information from the encoder. BBWE can be added to a decoder's toolchain without changing the transmission network, so it can be used to improve perceived audio quality and clarity until better codecs are introduced into the network. act as an intermediate solution [5], [6], [8]. To save transmission bandwidth or improve quality, robust BBWE may be a viable solution for modern voice transmission. Furthermore, in other types of applications such as audio restoration where band-limited audio is stored or archived, BBWE is the only possible solution to extend the audio bandwidth.

Although BBWE has a long tradition in the field of speech audio signal processing [9], [10], it is not in speech signal processing that solutions based on deep neural networks (DNNs) have been considered. In most cases, the work is done by researchers with a background in artificial intelligence (AI) or image processing. Such DNN-based systems are commonly referred to as speech super resolution (SSR). In image processing, the task of estimating a high-resolution image from one or more lower-resolution observations is called super-resolution, and has received considerable attention within the computer vision community. Recently, deep convolutional neural networks (Deep Convolutional Neural Networks) have produced better results than conventional methods [11], and super-resolution generative inverse problem networks are considered to be the most advanced [12].

A good BBWE can not only improve the perceived quality of speech but also improve the word error rate of automatic speech recognition systems [13].

Generative adversarial networks (GANs) can better recover finer structures for more realistic reproduction. However, some of these systems cannot be directly applied to voice communication scenarios. In the design of BBWE, in addition to the different properties of the underlying signals (eg, different dimensions), the following points need to be taken into consideration. First, the delay of the algorithm (the amount of time the decoded audio lags behind the original audio) must not be too large. Furthermore, the computational complexity and memory consumption must be able to meet the requirements of real-time processing in embedded systems such as mobile phones.

Recurrent neural networks are suitable for analyzing and predicting time series such as audio. In fact, speech is considered stationary or quasi-periodic in a broad sense for durations on the order of 20-25 ms, and its temporal correlation can be exploited in RNNs with relatively small models. CNNs, on the other hand, perform well in pattern recognition and upscaling tasks such as image super-resolution. They also have the advantage that processing can be highly parallelized. Therefore, both architectures need to be considered in audio processing, especially in BBWE.

As mentioned above, in the state of the art, the principle of BBWE was first presented by Karl-Otto Schmidt in 1933 [9], who used analog nonlinear devices to extend the bandwidth of transmitted audio. The idea of performing (non-blind) bandwidth extension on the excitation signal of an audio codec dates back to at least 1959 [10]. Subsequently, several so-called parametric BWEs have been published that utilize the separation of the audio signal into an excitation and a spectral envelope, based on a source filter model of human speech production. These systems apply statistical models to extrapolate the spectral envelope while generating the excitation signal by spectral folding [14], spectral transformation [8], or nonlinearity [15]. Statistical models for envelope extrapolation include simple codebook mapping [16], hidden Markov models [14], (shallow) neural networks [17], or more recently DNNs [18].

Before using DNNs, the inputs to statistical models were often manually adjusted functions [14], [17], [19], [20]. With the introduction of DNN, this approach directly uses logarithmic short-time Fourier transform (STFT) energy [18], [21], [22] or time-domain audio signals [23], [24], [25]. It can be simplified as follows. The same goes for the output of statistical models. Instead of modeling subband energies [8] or other envelope representations [21], DNNs can be used to model the entire audio signal in the time domain, or even a combination of time and frequency domains [26]. However, it has sufficient power to model the spectral magnitude per bin [15]. However, if the spectral magnitude is modeled, it is necessary to reconstruct the phase by spectral folding or transformation [18], [21], [15], [27].

Regarding learning objectives, designing an efficient DNN-based solution requires choosing an appropriate architecture, primarily the learning loss function and the network type. Typical loss functions include mean square error [21], categorical cross entropy (CE) loss [28], adversarial loss [29], [30], [25], or a mixture of losses [31]. be. The loss function can also determine the data representation.

Regarding mean square error and cross entropy, psychoacoustically motivated losses can be achieved by combining mean square error (MSE) losses with logarithmic subband or bin energies [8] . The loss function derived from cross-entropy (XE) predicts sample bits (or sample sizes) as classes, so the signal to be modeled must be quantized at a resolution that is not too high to be processed by a DNN. There is. Predicting 2 ¹⁶ classes of audio signals quantified in 16 bits is currently very expensive to process in DNNs. Fortunately, quantifying the content of audio signals above 3.4 kHz with 8 bits does not significantly degrade the quality [32]. Since the distribution of data to be learned using cross-entropy loss is preferably Gaussian distribution rather than the Laplacian distribution of audio signals [34], it is usually shaped by a nonlinear function. Surprisingly, the μ-law function used in [35], [32], [23], [24] to make audio data x more Gaussian is the world's first standardized digital audio It is exactly the same as codec [1]

Regarding adversarial loss, the distribution of speech in the time domain is very complex and difficult to model, even with today's powerful networks. A generative model trained with MSE or CE loss to match this complex distribution will only produce a smoothed approximation thereof. Applied to BBWE, this means that the resulting audio signal lacks clarity and energy [30].

Generative Adversarial Networks [36] can be considered as a type of extended loss function. Here, two networks are competing: a generator and a discriminator. Figure 2 shows a generative adversarial network. The generator attempts to generate realistic data, and the discriminator distinguishes between the generated data and the data from the training database. After successful learning, the discriminator is no longer needed and its purpose is only to improve the generator's loss. The reason why adversarial learning is suitable for learning generative models such as BBWE is that it is possible to model some modes of a distribution without smoothing or averaging all modes.

Regarding the class of networks, it should be noted that another important aspect in the design of a DNN is the selection of the class of network to use. Generally, fully connected layers [18], [21], convolutional neural networks (CNNs) [11], [37], or recurrent neural networks (RNNs) and their subtypes long short-term memory (LSTM) are commonly used. ) units [38], [39], [8] or gated recurrent units (GRU) [40], [39] are well known. Fully connected layers are only used in systems operating on frames [18], [21], whereas RNNs and CNNs allow processing of time-domain data in a streaming manner [23], [24].

WaveNet (registered trademark) is also adopted for BBWE. In [42], it is trained on clean speech conditioned on the bitstream parameters of encoded NB speech. Here, the network acts as a decoder and implicitly provides bandwidth expansion. In response to this, [24] provides WaveNet (registered trademark) with feature amounts calculated using NB signals. If the learning is successful, only the features are given to the network and the NB audio signal is ignored.

WaveNet®-based models claim very high perceptual quality, but are difficult to learn and have very high computational complexity during evaluation. This has led to several optimizations and alternative models (e.g. [43]). One particular alternative is LPCNet, designed for either original speech synthesis [32] or speech coding [44]. In LPCNet, the convolutional layer of WaveNet (registered trademark) is replaced with a recurrent layer.

The purpose of the invention is to provide an improved concept for blind bandwidth expansion.

The object of the invention is to provide a computer system according to claim 25 by means of an apparatus according to claim 1, by a method according to claim 19, by a method according to claim 20, by a method according to claim 23, and by a method according to claim 20. Solved by the program.

According to one embodiment, an apparatus is provided for processing a narrowband audio input signal to obtain a wideband audio output signal by performing bandwidth expansion of the narrowband audio input signal. The apparatus includes a signal envelope extrapolator including a first neural network, the first neural network configured to receive a plurality of samples of a signal envelope of a narrowband audio input signal as an input to the first neural network. and configured to determine a plurality of extrapolated signal envelope samples as output values of the first neural network. Further, the apparatus includes an excitation signal extrapolator 130 configured to receive a plurality of samples of an excitation signal of a narrowband audio input signal and configured to determine a plurality of extrapolated excitation signal samples. Be prepared. Additionally, the apparatus relies on the plurality of extrapolated signal envelope samples and in dependence on the plurality of extrapolated excitation signal samples to convert the wideband audio output signal into the narrowband audio input signal. A combiner 140 is configured to generate a wideband audio output signal to extend the bandwidth.

Further, according to one embodiment, a method is provided for processing a narrowband audio input signal to obtain a wideband audio output signal by performing bandwidth expansion of the narrowband audio input signal. The method includes:

- receiving as an input value of a first neural network a plurality of samples of a signal envelope of said narrowband audio input signal; and as an output value of said first neural network receiving a plurality of samples of a signal envelope of said narrowband audio input signal; a step of determining the sample;

- receiving a plurality of samples of an excitation signal of a narrowband audio input signal and determining a plurality of extrapolated excitation signal samples;

- said wideband audio input signal is dependent on a plurality of extrapolated signal envelope samples and said plurality of extrapolated excitation signal samples to extend the bandwidth with respect to said narrowband audio input signal; A step of generating an output signal.

Additionally, a method for training a neural network is provided according to one embodiment.

- the neural network receives as an input to the neural network a first plurality of line spectral frequencies of the narrowband audio input signal;

- the neural network determines, as an output value of the first neural network, a second plurality of line spectral frequencies of the broadband audio output signal, each of the one or more second plurality of line spectral frequencies being one or more of the first plurality of line spectral frequencies; associated with a frequency that is greater than any frequency associated with any of the multiple line spectral frequencies of .

- a second plurality of line spectral frequencies of the wideband audio output signal are transformed from a line spectral frequency domain to a linear predictive coding domain to obtain a second plurality of linear predictive coding coefficients of the wideband audio output signal;

- the finite impulse response filter transforms the second plurality of linear predictive coding coefficients of the wideband audio output signal from the linear predictive coding domain to the finite impulse response filter domain, and converts the linear predictive coding coefficients transformed by the plurality of finite impulse filters; It is used to obtain the coding coefficients.

- The method comprises the step of training said first neural network in dependence on linear predictive coding coefficients transformed with a plurality of finite impulse filters.

In an embodiment, when the first neural network is trained, linear predictive coding coefficients transformed by a plurality of finite impulse filters, or values derived from linear predictive coding coefficients transformed by a plurality of finite impulse filters. can be fed back to a neural network, for example.

According to embodiments, training the first neural network depends on the linear predictive coding coefficients transformed with the plurality of finite impulse filters and the plurality of extrapolated excitation signal samples, e.g. Samples of wideband audio output signals are generated and the plurality of wideband audio output signals or values derived from the samples of the plurality of wideband audio output signals are fed back to, for example, a neural network.

Furthermore, a method for training the first and/or second neural networks according to embodiments is provided.

- the first neural network receives as input values of the first neural network a plurality of samples of the signal envelope of the narrowband audio input signal and as output values of the first neural network a plurality of extrapolated samples; and/or the second neural network receives the plurality of samples of the excitation signal of the narrowband audio input signal as input values of the second neural network; A plurality of extrapolated excitation signal samples are determined as output values of the network.

- the first and/or second neural network is trained using a discriminator neural network, and once the first and/or second neural network is trained, the first and/or second neural network A discriminator neural network operates as a generative adversarial network.

- during training of the first and/or second neural network, the discriminator neural network receives output values of the first and/or second neural network as input values of the discriminator neural network; Alternatively, a derived value derived from the output value of the first and/or second neural network is received as an input value of the discriminator network.

- upon receiving the input value of the discriminator neural network, the discriminator neural network determines, as an output of the discriminator neural network, a quality representation of the input value of the discriminator neural network; /or the second neural network learns depending on the quality representation.

According to an embodiment, the discriminator neural network is, for example, a first discriminator neural network. The first neural network is trained using, for example, a first discriminator neural network, and the first neural network is trained depending on the quality representation, which is the first quality representation. The second neural network is trained using, for example, a second discriminator neural network, and during training of the second neural network, the second neural network and the second discriminator neural network are , operates as a second generative adversarial network. During training of the second neural network, the second discriminator neural network receives, e.g., an output value of the second neural network as an input value of the second discriminator neural network, or, e.g. , may receive as an input value of the second discriminator network a derived value derived from the output value of the second neural network. Upon receiving the input value of the second discriminator neural network, the second discriminator neural network selects the input value of the second discriminator neural network as an output of the second discriminator neural network. determining a second quality representation, wherein the second neural network is configured to learn in dependence on the second quality representation.

Furthermore, computer programs are provided, each of which is configured to perform one of the methods described above when executed on a computer or signal processing device.

As previously discussed, blind band extension improves the perceptual quality and intelligibility of telephone-quality audio by artificially reproducing missing frequency content that is transmitted unencoded with an audio codec. Embodiments provide a new approach based on deep neural networks to solve this problem. These embodiments are based on convolutional or iterative architectures. Everything operates in the time domain. Motivated by the source-filter model of human speech production, two of the presented systems decompose the speech signal into a spectral envelope and an excitation signal; each of them is individually augmented with a dedicated DNN. Bandwidth. All systems learn to mix adversarial and perceptual losses. To avoid mode collapse and more stable adversarial learning, spectral normalization may be employed, for example, in the discriminator.

In embodiments, we provide two BBWEs based on deep neural networks with adversarial learning targeted at speech coding scenarios.

According to embodiments, two novel deep network structures for blind bandwidth expansion are provided, one based on a convolutional kernel and one based on a recurrent kernel.

Both networks can be trained with a mixture of adversarial and spectral losses, for example.

The two systems are adversarially trained BBWEs that are "end-to-end," meaning that the input is time-domain audio and the output is also time-domain audio.

In embodiments, for example, hinge loss or spectral normalization can be applied to improve the performance of the GAN.

Embodiments provide a new approach for BBWE based on a generative model used for band extension of audio signals.

In the two systems presented, a well-established paradigm in the world of speech coding is adopted, i.e. the decomposition into speech signals known as envelope and source filter models can be applied, for example, to GAN models. . As a result, the computational complexity may be reduced, eg, by about a third. This approach has been tested and evaluated within a BBWE application, but is not limited thereto. The system according to embodiments significantly improves the speech recognition error rate for NB speech.

Some embodiments provide a generative model for generating enhanced speech from encoded, band-limited, or corrupted speech.

According to embodiments, the target speech for learning may be decomposed into an envelope and an excitation, for example. The envelope may be, for example, an LPC coefficient. The excitation can be, for example, an LPC residual.

In some embodiments, the envelope and excitation may be learned separately, for example. Each of the envelopes and excitations can be learned, for example, with a mixture of adversarial losses (known from generative adversarial networks (GANs)) and L1 losses. A characteristic loss is also added to the learning of the excitation signal.

According to embodiments, the envelope may be learned, for example, with an encoded and/or band-limited and/or corrupted envelope representation as input and the original envelope as target. A possible envelope representation may be, for example, an LPC coefficient.

In embodiments, the input for learning the excitation signal may be, for example, encoded and/or band-limited and/or corrupted time-domain audio and/or compressed feature representations. For example, the target is the original clean voice.

According to embodiments, to learn the excitation signal, the loss may be propagated through the envelope, for example. This can be done, for example, by considering the envelope as a DNN layer that propagates losses. If the envelope is represented by an LPC filter, this filter may be, for example, a pure IIR filter. In this case, the loss may, for example, propagate slowly or not at all (also known as the vanishing gradient problem). In embodiments, the IIR filter can be approximated by a FIR filter, for example by truncating the impulse response. As a result, the envelope can be implemented, for example, as a convolutional layer (CNN layer) within the network.

Some embodiments are based on the decomposition of the audio signal into an excitation signal and an envelope, similar to audio codecs [2], [4]. This is achieved using linear predictive coding (LPC). Recurrent layers are easy to predict because they simply model the excitation signal. In some embodiments, LPCNet is also employed in BBWE [33].

Embodiments of the present invention will be described in more detail below with reference to the drawings.

FIG. 1 illustrates an apparatus for processing a narrowband audio input signal to obtain a wideband audio output signal by performing band extension of the narrowband audio input signal according to an embodiment. FIG. 2 shows a generative adversarial network. FIG. 3 shows a single layer of softmax gate activation CNN-GAN. FIG. 4 shows the proposed system based on the decomposition of the audio signal into an excitation signal and an LPC envelope. FIG. 5 shows the transfer function of an FIR filter resulting from an order 12 IIR LPC filter and a truncated impulse response. FIG. 6 shows the structure of a DNN that extrapolates the excitation signal. Figure 7 shows one of the matrices from GRU after sparsification. FIG. 8 shows a GAN discriminator network composed of six convolutional layers. The kernel of each layer has 32 samples and the stride is 2. Figure 9 shows the perceived objective listening quality analysis of various BBWEs with 95% confidence intervals. FIG. 10 shows the Frecce deep speech distance (FDSD) of various BBWEs. FIG. 11 shows word and character error rates for various BBWEs. FIG. 12 shows the short-term objective intelligibility measurements (STOI) of the proposed system. FIG. 13 shows the results of listening tests that evaluated various BBWEs in box plots with 95% confidence intervals set for each item. Figure 14 shows the results of a listening test evaluating various BBWEs in a bar graph with 95% confidence intervals averaged across all items. FIG. 15 shows the results of a listening test in which various BBWEs were evaluated, and the evaluations from each user are shown in a warm plot. FIG. 16 shows normalized objective and subjective measures.

FIG. 1 illustrates an apparatus for processing a narrowband audio input signal to obtain a wideband audio output signal by performing band extension of the narrowband audio input signal according to an embodiment.

The apparatus comprises a signal envelope extrapolator 120 including a first neural network 125, the first neural network 125 is configured to calculate the signal envelope of a narrowband audio input signal as an input value of the first neural network 125. and is configured to receive a plurality of samples of the extrapolated signal envelope as an output value of the first neural network 125.

Further, the apparatus includes an excitation signal extrapolator 130 configured to receive a plurality of samples of an excitation signal of a narrowband audio input signal and configured to determine a plurality of extrapolated excitation signal samples. Be prepared.

Additionally, the apparatus relies on the plurality of extrapolated signal envelope samples and in dependence on the plurality of extrapolated excitation signal samples to convert the wideband audio output signal into the narrowband audio input signal. A combiner 140 is configured to generate a wideband audio output signal to extend the bandwidth.

According to embodiments, the input values of the first neural network 125 are a first plurality of line spectral frequencies of the narrowband audio input signal, and the first neural network 125 is configured to e.g. The network 125 may be configured to determine, as an output value, a second plurality of line spectral frequencies of the broadband audio output signal, each of the one or more second plurality of line spectral frequencies being one or more of the first plurality. associated with a frequency that is greater than any frequency associated with any of the line spectral frequencies.

In embodiments, once the first neural network 125 has trained, the signal envelope extrapolator 120 can generate a plurality of broadband The linear predictive coding coefficients may be configured to convert into finite impulse response filter coefficients.

For example, a broadband LPC filter coefficient, even if it is an IIR filter coefficient, is converted into a finite impulse response filter coefficient by calculating and truncating the impulse response. This is done during training so that the wideband LPC filter coefficients used for conversion to finite impulse response filter coefficients can be derived from the original wideband speech, for example.

According to embodiments, upon training the first neural network 125, the signal envelope extrapolator 120 is configured to feed back the error or slope of the error between the wideband audio output signal and the original wideband audio signal. configured, for example.

As outlined above, embodiments backpropagate the error gradient. Error is here the difference between the generated wideband speech and the true wideband speech.

Generally, an excitation is generated from narrowband speech and an envelope is generated therefrom. Finally, wideband speech is derived.

During application, the output of excitation signal extrapolator 130 may be fed to signal envelope extrapolator 120, for example.

In embodiments, during learning with backpropagation, the error gradient is first passed backwards to the signal envelope extrapolator 120 and then to the excitation signal extrapolator 130.

If the signal envelope is an IIR structure or filter, the gradient cannot be passed. For this purpose, the signal envelope is transformed into a finite impulse response filer.

According to embodiments, the first neural network 125 is trained using, e.g., a first discriminator neural network, and when the first neural network 125 is trained, e.g. 125 and the first discriminator neural network are configured to operate as a generative adversarial network. During training of the first neural network 125, the first discriminator neural network is configured, for example, to receive an output value of the first neural network 125 as an input value of the first discriminator neural network. or is configured to receive a derived value derived from the output value of the first neural network 125 as an input value of the first discriminator network. Upon receiving the input value of the first discriminator neural network, the first discriminator neural network may e.g. The circuit is configured to determine a first quality indication of an input value of the network. The first neural network 125 is then configured, for example, to learn in dependence on the first quality indication.

In embodiments, upon receiving the input value of the first discriminator neural network, the first discriminator neural network determines that the input value of the first discriminator neural network is artificially generated. The quality indicator indicates the probability that the output value of the first discriminator neural network is related to the recorded audio signal rather than the recorded audio signal, or that the output value of the first discriminator neural network is related to the recorded signal or to an artificially generated signal. For example, the quality indicator is configured such that the quality indicator indicates a value for estimating whether the quality indicator is correct or not.

According to embodiments, the first neural network 125 or the second neural network 135 is trained using, for example, a loss function that depends on the quality indication determined by the first discriminator neural network.

In embodiments, the loss function relies on, for example, a hinge loss, or a Wasserstein distance, or an entropy-based loss.

In an embodiment, the loss function depends on the (additional) Lp loss.

In embodiments, the first discriminator neural network may be trained using recorded audio, for example.

According to embodiments, the excitation signal extrapolator 130 includes a second neural network 135 that receives, for example, a narrowband audio input as an input value of the second neural network 135. The signal is configured to receive a plurality of samples of an excitation signal and/or is a narrowband audio input signal and/or is a shaped version of said narrowband audio input signal. The second neural network 135 is configured, for example, to determine a plurality of extrapolated excitation signal samples as an output value of the second neural network 135.

In embodiments, the input values of the second neural network 135 are, for example, a first plurality of time-domain signal samples of an excitation signal of a narrowband audio input signal and/or, for example, a narrowband audio input signal. and/or may be, for example, a shaped version of a narrowband audio input signal. Here, the second neural network 135 determines that the plurality of extrapolated excitation signal samples is a second one of the extended time-domain excitation signal whose bandwidth has been expanded relative to the excitation signal of the narrowband audio input signal. For example, the output value of the second neural network 135 is configured to be determined to be a sample of a plurality of time-domain signals.

According to embodiments, the second neural network 135 is trained using, for example, a second discriminator neural network, and during training of the second neural network 135, the second neural network 135 and the second discriminator neural network are configured to operate as a second generative adversarial network. During training of the second neural network 135, the second discriminator neural network is configured such that, for example, the output value of the second neural network 135 is received as the input value of the second discriminator neural network. , or configured to receive, as an input value of the second discriminator network, a derived value derived from the output value of the second neural network 135. and/or the second discriminator neural network is configured, for example, to receive the output value of the combiner 140 as an input value of the second discriminator neural network.

Upon receiving the input value of the second discriminator neural network, the second discriminator neural network selects the input value of the second discriminator neural network as an output of the second discriminator neural network. wherein the second neural network 135 is configured, for example, to learn in dependence on the second quality indication.

In embodiments, the apparatus includes a signal analyzer configured to generate from the narrowband audio input signal a plurality of samples of the signal envelope of the narrowband audio input signal and a plurality of samples of the excitation signal of the narrowband audio input signal. 110, for example.

According to embodiments, first neural network 125 includes, for example, one or more convolutional neural networks.

In embodiments, first neural network 125 includes, for example, one or more deep neural networks.

Next, a specific embodiment will be described.

In the following, three DNN-based BBWEs are described according to embodiments: two are based on convolutional architectures and one is based on a mix of convolutional and iterative architectures. All can be learned adversarially using, for example, the same discriminator, the same perceptual loss, and the same optimization algorithm. The architecture of the first BBWE is based on WaveNet (registered trademark), and the other architectures are based on LPCNet. First, all generation networks are presented, and all systems share the same discriminator, which is explained below.

First, the convolutional BBWE described in the embodiments will be explained.

The first architectural proposal for this task is a stack of convolutional neural networks (CNNs), which is currently a standard component of GANs. The use of CNN enables high-speed processing, especially on GPUs.

A WaveNet (registered trademark)-like structure was adopted as the convolutional generation model. Specifically, in a stack of 20 layers, in every layer, each layer uses causal convolution with a kernel size of 33, and softmax gate activation [45]. Bias is omitted. One of these layers is shown in FIG.

FIG. 3 shows a single layer of CNN-GAN with softmax-gated activations. The CNN layer has a one-dimensional kernel with 32 input channels and 64 output channels. Half of the output channels are fed to tanh activation and the other half are fed to softmax activation. Residual connections avoid gradient vanishing and maintain stable and effective learning.

As outlined, each CNN layer has 32 input channels and 64 output channels. Half of the output channels are fed to tanh activation and the other half are fed to softmax activation. Both activations are multiplied by the channel dimension to form the 32 channel outputs of each layer. This type of activation is more robust to reconstruction artifacts than both ReLU and sigmoid gate activation.

An additional input layer maps a 1-dimensional input signal to a 32-dimensional signal, and an additional output layer maps a 32-dimensional signal to a 1-dimensional output signal.

The weights of the convolution kernel are normalized using weight normalization [46] to enable stable learning behavior. We also apply batch normalization to the output features from the CNN layer to speed up the learning process.

Therefore, a complete convolutional layer consists of causal convolution followed by batch normalization and finally softmax gate activation to obtain the final output. Connections or shortcuts from inputs to outputs exist to avoid vanishing gradients and maintain stable and effective learning [47].

In this convolutional BBWE, the model is run on the raw audio waveform in the time domain. The input signal is first resampled from NB to WB using simple Sinc interpolation and then fed to the generator model. The generator reliably extends the original bandwidth of this upsampled signal to obtain a complete WB structure with clearly higher perceptual quality.

This system is called CNN-GAN.

Here, LPC-GAN according to an embodiment will be explained.

In particular, two systems according to embodiments are provided that differ from convolutional ones in two ways: first, the architecture of the DNN differs. Second, the audio signal is decomposed into an excitation signal and an envelope. It is inspired by BBWE based on LPCNet [33], but some embodiments apply it. The motivation for decomposing the signal into excitation and envelope is the same as in LPCnet-based BBWE [33], which is to reduce the amount of computation of the entire system.

FIG. 4 shows a block diagram of the system, and FIG. 6 shows in detail one of the DNN bandwidths extending the excitation signal. In particular, FIG. 4 shows the proposed system based on decomposing the audio signal into an excitation signal and an LPC envelope. All solid paths operate on samples, and all dashed paths operate on 15ms frames.

In FIG. 4, the input NB audio signal is separated into an LPC representing a spectral envelope and an excitation signal (also known as residual). The excitation signal and input signal are fed to the first DNN for extrapolation to the WB excitation signal. This path works on the sample shown here in solid line. The LPC is extrapolated to the WB envelope using the second DNN in the top pass. This path operates in 15ms frames and is shown here as a dashed line. LPC coefficients are IIR filter coefficients and are extrapolated in the LSF domain since operations such as extrapolation can destabilize the filter [48]. LSF is a bijective transformation of LPC that has several advantages: first, it is less sensitive to noise perturbations, and the ordered set of LSF with minimum distance between coefficients makes it always stable. LPC filter is guaranteed. Second, the spectral envelope at a particular frequency depends only on one of the LSFs, so incorrect extrapolation of a single LSF coefficient mainly affects the spectral envelope over a limited frequency range. do. These characteristics are suitable for extrapolation to a set representing the WB envelope. The extrapolated LSF coefficients are transformed to the LPC domain to form an extrapolated excitation signal and an output signal. This is accomplished in a variety of ways for learning and assessment.

The extrapolated excitation signal formed by the LPC envelope forms the output WB signal. When learning a DNN to extrapolate the excitation signal, it is necessary to propagate the gradient through an LPC filter, which can be achieved by performing LPC filtering with an additional DNN layer. Since the LPC filter is a pure IIR filter, this DNN layer must be a layer with recurrent units. Unfortunately, backpropagating the gradient through recurrent layers causes the gradient to vanish (also known as the vanishing gradient problem [38]), resulting in poor learning. As a solution to this problem, IIR filter coefficients are converted to FIR filter coefficients by calculating the truncated impulse response from the IIR filter. It is known from signal processing that any IIR filter can be approximated by a FIR filter by truncating the infinite impulse response [34]. Then, LPC shaping can be realized with a convolutional layer. Figure 5 shows the effect of truncating to 64 samples.

FIG. 5 shows the transfer function of the FIR filter resulting from a 12th order IIR LPC filter and a truncated impulse response.

The IIR LPC envelope is smooth, but the truncated FIR envelope has a lot of ripple and does not track the IIR envelope well at high frequencies. For this, before calculating the truncated impulse response, the LPC coefficients are multiplied by an exponential function:

In FIG. 5, the impulse response has been truncated to 64 samples. The green filter is one in which the IIR LPC coefficient is processed using equation (4), and the red filter is one in which no processing is performed.

Early experiments have shown that the FIR-shaped signal contains artifacts, which are easily identified by the discriminator. As a result, the adversarial losses were not balanced and the generator was poorly trained. This can be solved by calculating the adversarial loss of the actual generated unshaped excitation signal.

LPC shaping by the FIR filter is performed only during the learning time. During the evaluation time, the LPC coefficients are applied as an IIR filter since there is no need to backpropagate the gradients.

Two different DNNs are used for extrapolation of the excitation signal. The first is based on a mixture of convolutional and recurrent layers, and the second is based solely on convolutional architectures. Regarding the first, details are shown in FIG.

FIG. 6 shows a structure in which the DNN extrapolates the excitation signal. The shape of the signal in parentheses is given with the batch dimension omitted. T is the length of the input signal.

The purpose of the first CNN layer is to add a characteristic dimension to the one-dimensional time-domain signal. This characteristic dimension is required in the GRU layer, otherwise the GRU matrix would collapse into a simple vector. CNNs add a distinctive dimension by running kernels, commonly called channels, in parallel. As a result, 256 channels are required to maintain compatibility between the CNN layer and the GRU layer.

This makes the calculations very complex. This can be prevented by dividing the channels into 16 groups of 16 channels each. This is equivalent to placing 16 layers in parallel for every 16 channels. The structure of the CNN layer (kernel size, gate activation, etc.) may be, for example, the same as described for the convolutional BBWE above. Since the output of the second GRU layer still has a characteristic dimension, it is compressed into a one-dimensional signal with a single convolution kernel with kernel size 1.

The main source of computational complexity lies in the first GRU matrix. To further reduce complexity, these matrices can be made sparse during training [49].

After repeating the initial training with a dense matrix, small blocks are identified and forced to zero. A Boolean matrix stores the indices of these blocks. As training progresses, more blocks can be zeroed out until the desired sparsity is achieved. Similar to [32], 16x1 blocks are used, but all diagonal terms are also included. The final proportion of elements stored in the matrix is:

Ignoring the computational overhead for indexing, this sparsification scheme reduces GRU's computational complexity by 90%. FIG. 7 shows one of the sparse matrices after learning. This system is called LPC-RNN-GAN.

In particular, FIG. 7 shows one of the matrices from GRU after sparsification.

DNNs based on convolutional architectures only have the same structure as described in Sc. III-A has three structural differences. Firstly, the size of the CNN kernel is only 17, and secondly, to compensate for the resulting smaller receptive field, the system uses a dilated convolution with a dilation factor of 2 for each layer. It is what we are using. Third, to reduce complexity, the system takes advantage of the grouping described above by dividing the channel dimension into four groups. Furthermore, we show below that this can reduce the computational complexity by about a third. This system is called LPC-CNN-GAN.

The DNN that extrapolates the LPC envelope is also a combination of a CNN layer, followed by a GRU layer and a final CNN layer. The CNN layer has a two-dimensional kernel with kernel size 3, which operates on current, past, and future frames, and is the main source of algorithmic delay for the entire system.

A discriminator according to an embodiment will be described below.

The discriminator acts as a convolutional encoder that extracts the latent representation of the input signal and evaluates the adversarial loss. CNN-GAN, LPC-CNN-GAN, and LPC-RNN-GAN use the same discriminator architecture composed of convolutional layers for adversarial learning. Stable adversarial learning is achieved by applying spectral normalization to the convolution kernels of discriminator layers [50]. This kind of normalization forces the Lipschitz condition on the function learned by the discriminator. This was found to be important for an effective and stable adversarial learning method. The discriminator operates with conditional settings [51], so the input signal includes a real/false WB audio waveform concatenated with an upsampled NB audio waveform along the channel dimension. FIG. 8 shows the discriminator. It consists of 6 convolutional layers, the kernel size is 32, and the stride is 2 steps. Bias omitted. Leaky ReLU with a negative slope of 0:2 is used for activation.

In particular, FIG. 8 shows a GAN discriminator network consisting of six convolutional layers with a kernel of 32 samples operating with a stride of 2 in each layer. The numbers within the layers represent the dimensions of the input and output channels for each layer.

Since the conditioning input is time-domain NB audio, the discriminator rejects audio generated with a different waveform than the original. LPCNet based on BBWE has fewer restrictions on generated waveforms, as will be described later. To achieve a GAN with fewer constraints on the generated waveforms, a second discriminator is evaluated that takes a low-dimensional feature representation as input. The feature is the Mel Frequency Cepstral Coefficient (MFCC) [52] calculated on NB speech. This discriminator, combined with the absence of Lp loss, tends to penalize speech that is not generated with a different waveform than the original.

Here, a consideration regarding the purpose of learning is presented.

ここで、Ｄ（）はディスクリミネーターの生の出力である。Ｌｉｍ ｅｔ． ａｌ．［５３］は、最初のＧＡＮ論文［３６］で用いられた損失やワッサースタイン距離（Ｗａｓｓｅｒｓｔｅｉｎ ｄｉｓｔａｎｃｅ）［５４］と比較して、ヒンジ損失はモード崩壊が少なく、より安定した学習挙動を示すことを示した。 The adversarial metric used in this work is the hinge loss [53]:

Here, D() is the raw output of the discriminator. Lim et. al. [53] found that compared to the loss and Wasserstein distance [54] used in the first GAN paper [36], the hinge loss has less mode collapse and exhibits more stable learning behavior. Indicated.

Initial experiments using the proposed system show that hinge loss performs similarly to feature matching. As already observed in [30], [25], the adversarial loss can be corrected by the Lp norm computed on the samples and features. Here, the L1 norm calculated using samples in the time domain and the L2 norm calculated using logarithmic Mel energy are used as the feature loss L _mel . The total loss training of the generator is:

In the following, the experimental setup is described.

As learning materials, we used several publicly available audio databases [55], [56], [57] and audio items in other languages. A total of 13 hours of learning material was used, all of which was resampled to a sampling frequency of 16kHz. Silent parts of the training data were removed using voice-activation-detection [58]. The input signal of the NB was encoded with AMR-NB at 10.2 kbps. The target clean audio signal was previously enhanced with a first order filter E as shown below.

が、生成された音声に適用された。この理由は、生成された音声の中で高周波数があまり強調されなくなる可能性のある音声のスペクトルの傾きを補正するためである。１２次のＬＰＣ包絡線は、ハニング窓でウィンドウ化された１２８サンプルのフレームに対して、時間領域の自己相関を計算した後、レビンソン再帰を用いて抽出される。その後、上記のＬＰＣ－ＧＡＮに関して説明したように、例えばＦＩＲフィルタに変換される。ＤＮＮは、８項目のバッチで学習され、各項目には１秒の音声が含まれる。 is the inverse (de-emphasis) filter D.

was applied to the generated audio. The reason for this is to correct for the spectral tilt of the voice, which may cause high frequencies to become less emphasized in the generated voice. The 12th order LPC envelope is extracted using Levinson recursion after calculating time domain autocorrelation for a frame of 128 samples windowed with a Hanning window. It is then converted into, for example, an FIR filter, as described for the LPC-GAN above. The DNN is trained in batches of 8 items, each item containing 1 second of audio.

The optimization algorithm for both the generator and discriminator is Adam [59], where the learning rate of the generator is 0.0001 and the learning rate of the discriminator is 0.0004. For a more stable adversarial loss, the coefficients (beta parameters) used to calculate the moving average of the gradient and its square are set to 0.5 and 0.99, respectively. The learning rate of the generator and discriminator is set to 0.0001, since the RNN of LPC-RNN-GAN (see the discussion on LPC-GAN above) usually learns slower than CNN. The beta parameters for training the generator are set to 0.7 and 0.99. The coefficient λ that controls the amount of adversarial loss in equation (10) is set to 0.0015. Sparsification of the GRU layer starts from the 160th batch, and final sparsification is achieved at the 10000th batch. All CNN layers are trained with batch normalization to speed up training and prevent the network from falling into mode collapse.

The additional frame rate network that extrapolates the LPC coefficients in the LSF domain has 10 CNN layers, followed by a single GRU and a final CNN layer. The initial CNN layer is a two-dimensional convolution with kernel size 3x3, 16 channels, tanh activation function, and residual connections. The matrix size of GRU is 16x16, the final convolution layer is 5 channels, and the number of missing LSF coefficients is concatenated with the NB LSF coefficients to form the WB LSF coefficients.

In the following, we compare the presented system with the BBWE-based LCPNet of [33]. In contrast to the published systems, the DNN used for extrapolation of the LPC envelope is here adversarially trained. For this, the same discriminator architecture is used, only adapting the input dimensions.

All DNNs were implemented and trained using PyTorch [60].

Below, we will discuss this from an evaluation perspective. The systems provided in accordance with the embodiments are compared to previously published systems by objective measures and subjective measures by listening tests. Computational complexity estimates are given and compared to state-of-the-art speech coding techniques. Objective and subjective tests show that the proposed system provides substantially better quality than previous techniques. Systems according to embodiments are shown to reduce the Word Error Rate of speech recognition systems.

The perceptual quality of the presented BBWE is evaluated by objective measures that have been used to date to access the quality of speech, as well as subjective measures by listening tests. Furthermore, the algorithm delay and computational complexity are given for each BBWE. Examine whether correlations between objective and subjective outcomes have sufficient power to predict subjective ratings.

Regarding computational complexity, the computational complexity of the proposed BBWE is an estimate of Weighted Million Operations per Second (WMOPS) per audio sample. WMOPS is an ITU unit for calculating the computational complexity of standardized speech processing tools [61]. Addition (ADD), multiplication (MUL), and multiply-accumulate (MAC) operations each count as one operation, while complex operations such as tanh, sigmoid, or softmax operations count as 25 operations each. be done. Below, a numerical value is calculated for each audio sample. This value is multiplied by the sampling frequency to calculate the estimated value of WMOPS. This should be considered a rough approximation that does not take into account the benefits of today's parallel processing architectures. The results are summarized in Table 1 along with the computational complexity of EVS [2], [62], a state-of-the-art standardized audio codec.

In particular, Table 1 shows the computational complexity and algorithmic delay of the systems provided by several embodiments, LPCNet-BBWE [33] and EVS [2], [62] (EVS is the state-of-the-art ) is a standardized audio codec. WMOPS is an ITU standard for calculating computational complexity [61] and is calculated at a sampling frequency of 16 kHz.

Regarding the computational complexity of LPC-RNN-GAN and LPC-CNN-GAN: As mentioned above for LPC-GAN, this system has an initial CNN layer that divides the one-dimensional signal into 256 channels. These layers are the same CNN layers as above, except that the channels are grouped into blocks as described for LPC-GAN above. Here, a total of 256 channels are grouped into 16 blocks of 16 channels each. This is equivalent to having 16 CNN layers with 16 channels in parallel.

The operation of one RNN layer for a single audio sample is shown in equation (5). Here, M _i is the input dimension and M _h is the output (or hidden) dimension. Then, calculation of the reset gate and update gate (first two lines of the equation) requires M _i *M _h *2 MAC operations and M _h sigmoid operations, respectively. The new gate (line 3 of the equation) requires M _i *M _h *2+M _h MAC operations and M _h tangents hyperbolic operations. Finally, the output (last row) requires M _h *2 MAC operations. The first large GRU layer uses a sparsified matrix (see discussion on LPC-GAN above), so operations are computed with a reduced matrix size. Overhead due to additional addressing-operations is ignored. In the first GRU, all matrices are square with M _i =M _h =256, and in the second GRU M _i =256 and M _h =32.

The final CNN layer requires 32 ADD operations just to sum the output dimensions. The computational complexity of the LPC-CNN-GAN is calculated as having four such networks with channel dimensions of only 8 in parallel, as described above.

During evaluation, the LPC filter is applied as a 12-tap IIR filter, requiring 12 MAC operations per sample. Conversion from LPC to LSF coefficients and vice versa is ignored here. This is because these transformations are performed on a frame-by-frame basis and their contribution to the overall complexity is expected to be small. Table 1 above summarizes the number of operations with the parameterizations used.

Regarding algorithm delay, algorithm delay is the theoretical delay in milliseconds between input audio and processed output audio caused by block processing of audio samples. CPU or GPU time is not considered. The numbers are summarized in Table 1 above.

Regarding LPC-RNN-GAN and LPC-CNN-GAN, the source of algorithm delay in these systems is the initial convolutional layer and LPC processing. The GRU layer does not introduce any algorithmic delays. The kernel size of the four convolutional layers is 16 tabs, and 16 tabs are computed in future samples, resulting in a 4 ms delay. Therefore, the algorithmic delay in LPC processing due to the windowed autocorrelation function is 15 ms. Since this block processing is independent of the convolutional layer, the total algorithmic delay for the entire system is 15 ms. LPC-CNN-GAN uses a kernel that is half the size of CNN-GAN and has an extension of 2, so it has the same algorithm delay as CNN-GAN.

Listening tests with human listeners are the ultimate basis for assessing (e.g., objective) perceptual quality, but require considerable effort to perform. Objective indicators are an easy-to-use alternative. Here, Perceptual Objective Listening Quality Analysis, Fr´echet Deep Speech Distance, Word Error Rate, and Short Time Objective intelligibility measurement (Short Four different measurements are used: - Time Objective Intelligibility measure). All measurements except word error rate are computed on an approximately 1 hour multilingual, multi-speaker database that is not part of the training set.

Perceptual Objective Listening Quality Analysis (POLQA) is a standardized method aimed at predicting the perceived quality of audio signals encoded with the same Mean Opinion Scale (MOS) used in listening tests. [63]. The estimation results are summarized in Fig. 9, showing that LPC-RNN-GAN achieves the highest evaluation, followed by CNN-GAN.

In particular, FIG. 9 shows the Perceived Objective Listening Quality Analysis (POLQA) of various BBWEs with 95% confidence intervals. A larger value means better quality.

Assessing the quality of audio and images generated by GANs is a difficult challenge. In typical use cases, GANs generate items from noise, so metrics based on the Lp norm cannot be used because there is no standard to compare them to.

FIG. 10 shows the Fréchet deep speech distance (FDSD) of various BBWEs. The smaller the value, the higher the quality.

ここで、Ｓは置換数、Ｄは削除数、Ｉは挿入数、Ｃは転写の正しい単語の数である。 Regarding word error rate, BBWE can not only improve perceptual quality but also speech intelligibility [5], [6] and even the performance of automatic speech recognition (ASR) systems. State-of-the-art ASR systems are based on DNNs trained on audio with a fixed sampling frequency (mainly 16 kHz). As a result, the performance of such systems is significantly degraded if the audio is coded with the NB codec. We evaluate the impact of speech coding using AMR-NB on the word error rate (WER) of state-of-the-art ASR systems and how BBWE can alleviate this impact. The ASR system used here is a Mozilla RNN based deep speech system [68] with a connectionist time classification (CTC) loss [69] trained on a general spoken multilingual speech corpus [70]. It is an open implementation. The evaluation is performed against the evaluation set of this database. The WER metric is evaluated at the word level of the transcribed audio and is calculated as follows:

Here, S is the number of substitutions, D is the number of deletions, I is the number of insertions, and C is the number of correct words in the transcription.

FIG. 11 shows the word error rate (WER) and character error rate (CER) for various BBWEs. A smaller value means higher performance. In particular, FIG. 11 shows the ASR performance of AMR-NB and various BBWEs, along with Character Error Rate (CER), which is calculated similarly to WER, but at the character level instead of the word level.

Table 2 shows an example of one of the worst performing items. Interestingly, uncoded items perform better on average, but when using AMR-NB coded items from the database, outliers perform worse than 0:6. There isn't. Items processed with BBWE improve average WER, but also produce outliers with WER greater than 8:0.

Regarding the short-term objective intelligibility measurement (STOI), the short-term objective intelligibility measurement (STOI) is calculated as an estimate of the linear correlation coefficient between the clean temporal energy envelope and the BBWE-processed speech subband. defined. These subbands are derived from dividing the audio signal into Hanning-windowed frames of 256 samples length with 50% overlap, zero-padded each frame to 512 samples, and based on a Fourier-transformed time-frequency representation. calculated. The 15 1/3 octave bands are calculated by averaging the DFT bins.

Originally, this measurement was calculated on audio sampled at a sampling frequency of 10 kHz. Since we are evaluating the quality of WB audio, this measurement is extended to 16kHz.

Figure 12 shows the results of the presented system.

In particular, FIG. 12 shows a short-term objective intelligibility measurement (STOI) of the proposed system. The smaller the value, the lower the quality.

According to this measurement, LPC-RNN-GAN exhibits the best performance, followed by LPC-CNN-GAN.

Below, we will consider subjective perceived quality.

To finally judge the perceptual quality of the proposed system, the MUSHRA listening test [71] was conducted. According to the MUSHRA methodology, test items include references marked as such, hidden references, and AMR-NB coded signals that serve as anchors. Twelve experienced listeners participated in the test. The speech items used in the test are approximately 10 seconds long and are not part of the training or test set. The items include audio from native speakers of Chinese, English, French, German, and Spanish. The results are shown in FIG. 13 for each item, and in FIG. 14 for the average of all items, as a boxplot showing the average value and 95% confidence interval. The results are shown in a bar graph in FIG.

In particular, Figure 13 shows the results of a listening test assessing various BBWEs as boxplots with 95% confidence intervals for each item.

Figure 14 shows the results of a listening test assessing various BBWEs in a bar graph with 95% confidence intervals averaged across all items.

FIG. 15 shows the results of a listening test in which various BBWEs were evaluated, and the evaluations from each user are shown in a warm plot.

The system marked as CNN-feat-cond is a CNN-GAN trained with a discriminator with conditional inputs based on the features described above for the discriminator. L1 loss is also removed from the learning objectives.

The results show that all the presented systems significantly improve the quality of AMR-NB audio for all items. With the exception of CNN-feat-cond, none of the presented systems significantly outperforms the others. The best system tends to be LPC-CNN-GAN, which also significantly outperforms the CNN-feat-cond system.

Examining the results for single items reveals that quality is highly item dependent. LPC-CNN-GAN is not always the best performing system. For the two items Spanish women, German women, and men, the LPCNet-based system performs best. For Chinese male items, LPC-RNN-GAN performs best, and for Spanish male items, CNN-GAN performs best. CNN-GANs often have the least noisy artifacts, but often fail to reconstruct fricatives well.

Quality variations are particularly large in LPCNet-based systems. While this system provides very high quality, it can introduce severe artifacts such as clicks and pitch instability. GAN-based systems, on the other hand, are not susceptible to such severe artifacts, but suffer from broadband crackling noise. LPCNet-based systems, and in some cases feature adjustment-based systems, change the characteristics of the audio because both systems place fewer constraints on the generated waveforms. In the MUSHRA test, this can lead to lower scores, as in various test methods such as the Absolute Category Rating (ACR) test where no references are given.

In order to confirm the extent to which objective measures reflect subjective evaluations, we examine the correlation with MOS values of listening tests. For fair comparison, all measurements are normalized to zero mean and standard deviation. FDSD, WER, and CER give lower values for better quality estimates, so these values are initially negated.

FIG. 16 shows the normalized values, and Table 3 shows the correlation values.

In particular, FIG. 16 shows normalized objective and subjective measurements.

STOI showed the highest correlation with MOS value, followed by POLQA, WER, CER. WER follows and turns out to be the only measurement with the same order as the listening test results. The difference between WER and FDSD values is strange. This is because both measurements are based on the output of similar networks (DeepSpeech and DeepSpeech 2).

Two fundamentally different approaches for performing BBWE were compared: the GAN model and the autoregressive model. Both approaches rely on generative models that can model complex data distributions, such as the distribution of speech in the time domain, and neither approach suffers from smoothing problems.

Both approaches have moderate computational complexity compared to state-of-the-art models such as WaveNet® [35].

BBWE based on LPCNet is the model with the lowest computational complexity. The main reason for the reduced complexity is that this model imposes fewer constraints on the generated waveforms. The waveform generated by LPCNet can be significantly different from the original waveform, but GAN-based BBWE maintains the original waveform due to conditioning and mixing with adversarial and L1 losses. Unfortunately, changing the conditioning to functional conditioning and removing the L1 loss did not improve the quality of the speech produced.

LPC-RNN-GAN and LPC-CNN-GAN differ in the DNN used for extrapolation of the excitation signal. The former is based on a mixture of CNN and RNN, while the latter uses only CNN.

The computational complexity of both DNNs is approximately the same. Although there is no big difference in performance, the performance of LPC-CNN-GAN tends to be superior. Furthermore, the training time of CNN is short and is not very sensitive to hyperparameter tuning. LPC-RNN-GAN successfully applies sparsification for the first time in the context of GAN learning.

Correlating listening test results with objective measurements yields ambiguous results. The authors of [66] showed that FDSD measurements work well for estimating the quality of adversarially generated speech, but here small differences between the systems presented are not accessible. The measures that best correlate with subjective outcomes are the STOI and WER measures.

Although some aspects have been described so far in the context of devices, these aspects are also descriptions of corresponding methods, and it is clear that the blocks or devices correspond to method steps or features of method steps. be. Similarly, aspects described in the context of method steps are also descriptions of corresponding blocks or items or functions of the corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

Depending on particular implementation requirements, embodiments of the invention may be implemented in hardware or software, or at least partially in hardware or at least partially in software. The implementation is, for example, a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, FLASH memory, having electronically readable control signals stored thereon. , can be performed using a digital storage medium that cooperates (or is cooperable) with a programmable computer system on which the respective method is executed. Thus, a digital storage medium may be computer readable.

Some embodiments according to the invention are a data carrier having an electronically readable control signal, the data carrier being a programmable computer system such that one of the methods described herein is carried out. including data carriers that can cooperate with

Generally, embodiments of the invention may be implemented as a computer program product comprising program code, the program code being configured to perform one of the methods when the computer program product is executed on a computer. is operable to do so. The program code may be stored on a machine-readable carrier, for example.

Other embodiments have a computer program for performing one of the methods described herein stored on a machine-readable carrier.

In other words, an embodiment of the method of the invention is therefore a computer program having a program code for carrying out one of the methods described herein, the computer program being executed on a computer. This is the case when it is executed.

A further embodiment of the method of the invention therefore provides a data carrier (or digital storage medium, or computer readable medium). A data carrier, digital storage medium or recording medium is typically tangible and/or non-transitory.

A further embodiment of the method of the invention is therefore a sequence of data streams or signals representing a computer program for carrying out one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted over a data communications connection, such as the Internet, for example.

Further embodiments comprise processing means, such as a computer or a programmable logic device, configured or adapted to carry out one of the methods described herein.

A further embodiment comprises a computer having a computer program installed thereon for performing one of the methods described herein.

A further embodiment according to the invention provides an apparatus configured to transmit (e.g. electronically or optically) to a receiver a computer program for performing one of the methods described herein. Or have a system. The receiver is, for example, a computer, mobile device, storage device, etc. The device or system may constitute, for example, a file server for transmitting computer programs to a receiver.

In some embodiments, programmable logic devices (e.g., field programmable gate arrays) can be used to perform the functionality of some or all of the methods described herein. . In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

The apparatus described herein can be implemented using hardware devices, computers, or a combination of hardware devices and computers.

The methods described herein can be performed using a hardware device, a computer, or a combination of a hardware device and a computer.

The embodiments described above are merely illustrative of the principles of the invention. It is understood that modifications and variations in the arrangements and details described herein will be apparent to those skilled in the art. It is the intention, therefore, herein to be limited only by the scope of the appended claims and not by the specific details presented in the description and discussion of the embodiments.

References
[1] International Telecommunication Union, "Pulse code modulation (pcm) of voice frequencies," ITU-T Recommendation G.711, November 1988.

[2] S. Bruhn, H. Pobloth, M. Schnell, B. Grill, J. Gibbs, L. Miao, K. Jaervinen, L. Laaksonen, N. Harada, N. Naka, S. Ragot, S. Proust , T. Sanda, I. Varga, C. Greer, M. Jelinek, M. Xie, and P. Usai, "Standardization of the new 3GPP EVS codec," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, 2015, pp. 5703-5707. [Online]. Available: https://doi.org/10.1109/ICASSP.2015.7179064

[3] S. Disch, A. Niedermeier, CR Helmrich, C. Neukam, K. Schmidt, R. Geiger, J. Lecomte, F. Ghido, F. Nagel, and B. Edler, “Intelligent gap filling in perceptual transform coding of audio," in Audio Engineering Society Convention 141, Los Angeles, Sep 2016. [Online]. Available: http://www.aes.org/e-lib/browse.cfm?elib=18465

[4] 3GPP, "TS 26.090, Mandatory Speech Codec speech processing functions; Adaptive Multi-Rate (AMR) speech codec; Transcoding functions," 1999.

[5] P. Bauer, R. Fischer, M. Bellanova, H. Puder, and T. Fingscheidt, "On improving telephone speech intelligibility for hearing impaired persons," in Proceedings of the 10. ITG Conference on Speech Communication, Braunschweig, Germany, September 26-28, 2012, pp. 1-4. [Online]. Available: http://ieeexplore.ieee.org/document/6309632/

[6] P. Bauer, J. Jones, and T. Fingscheidt, "Impact of hearing impairment on fricative intelligibility for artificially bandwidth-extended telephone speech in noise," in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pp. 7039-7043. [Online]. Available: https://doi.org/10.1109/ICASSP.2013.6639027

[7] J. Abel, M. Kaniewska, C. Guillaume, W. Tirry, H. Pulakka, V. Myllylae, J. Sjoberg, P. Alku, I. Katsir, D. Malah, I. Cohen, MAT Turan, E. Erzin, T. Schlien, P. Vary, AH Nour-Eldin, P. Kabal, and T. Fingscheidt, "A subjective listening test of six different artificial bandwidth extension approaches in english, chinese, german, and korean," in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai, China, March 20-25, 2016, 2016, pp. 5915-5919. [Online]. Available: https://doi.org/10.1109/ ICASSP.2016.7472812

[8] K. Schmidt and B. Edler, "Blind bandwidth extension based on convolutional and recurrent deep neural networks," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 5444-5448.

[9] K. Schmidt, "Neubildung von unterdrueckten Sprachfrequenzen durch ein nichtlinear verzerrendes Glied," Dissertation, Techn. Hochsch. Berlin, 1933.

[10] M. Schroeder, "Recent progress in speech coding at bell telephone laboratories," in Proceedings of the third international congress on acoustics, Stuttgart, 1959.

[11] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov 1998.

[12] C. Ledig, L. Theis, F. Huszar, J. Caballero, AP Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network," CoRR, vol. abs/1609.04802, 2016. [Online]. Available: http://arxiv.org/abs/1609.04802

[13] X. Li, V. Chebiyyam, and K. Kirchhoff, "Speech audio super-resolution for speech recognition," in Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, September 15-19, 2019, 09 2019.

[14] P. Jax and P. Vary, "Wideband extension of telephone speech using a hidden markov model," in 2000 IEEE Workshop on Speech Coding. Proceedings., 2000, pp. 133-135.

[15] K. Schmidt and B. Edler, "Deep neural network based guided speech bandwidth extension," in Audio Engineering Society Convention 147, Oct 2019. [Online]. Available: http://www.aes.org/e- lib/browse.cfm? elib=20627

[16] H. Carl and U. Heute, "Bandwidth enhancement of narrow-band speech signals," in Signal Processing VII: Theories and Applications: Proceedings of EUSIPCO-94 Seventh European Signal Processing Conference, September 1994, pp. 1178-1181 .

[17] H. Pulakka and P. Alku, "Bandwidth extension of telephone speech using a neural network and a filter bank implementation for highband mel spectrum," IEEE Trans. Audio, Speech & Language Processing, vol. 19, no. 7, pp. 2170-2183, 2011. [Online]. Available: https://doi.org/10.1109/TASL.2011.2118206

[18] K. Li and C. Lee, "A deep neural network approach to speech bandwidth expansion," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24 , 2015, 2015, pp. 4395-4399. [Online]. Available: https://doi.org/10.1109/ICASSP.2015.7178801

[19] P. Bauer, J. Abel, and T. Fingscheidt, "Hmm-based artificial bandwidth extension supported by neural networks," in 14th International Workshop on Acoustic Signal Enhancement, IWAENC 2014, Juan-les-Pins, France, September. 8-11, 2014, 2014, pp. 1-5. [Online]. Available: https://doi.org/10.1109/IWAENC.2014.6953304

[20] J. Sautter, F. Faubel, M. Buck, and G. Schmidt, "Artificial bandwidth extension using a conditional generative adversarial network with discriminative training," in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 7005-7009.

[21] J. Abel, M. Strake, and T. Fingscheidt, "A simple cepstral domain dnn approach to artificial speech bandwidth extension," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp .5469-5473.

[22] J. Abel and T. Fingscheidt, "Artificial speech bandwidth extension using deep neural networks for wideband spectral envelope estimation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 1, pp. 71-83, 2018.

[23] Z. Ling, Y. Ai, Y. Gu, and L. Dai, "Waveform modeling and generation using hierarchical recurrent neural networks for speech bandwidth extension," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. . 26, no. 5, pp. 883-894, May 2018.

[24] A. Gupta, B. Shillingford, YM Assael, and TC Walters, "Speech bandwidth extension with wavenet," ArXiv, vol. abs/1907.04927, 2019.

[25] S. Kim and V. Sathe, "Bandwidth extension on raw audio via generative adversarial networks," 2019.

[26] Y. Dong, Y. Li, X. Li, S. Xu, D. Wang, Z. Zhang, and S. Xiong, “A time-frequency network with channel attention and non-local modules for artificial bandwidth extension ," in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6954-6958.

[27] J. Makhoul and M. Berouti, "High-frequency regeneration in speech coding systems," in ICASSP '79. IEEE International Conference on Acoustics, Speech, and Signal Processing, April 1979, pp. 428-431.

[28] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis," CoRR, vol. abs/1802.08435, 2018. [Online]. Available: http://arxiv.org/abs/1802. 08435

[29] S. Li, S. Villette, P. Ramadas, and DJ Sinder, "Speech bandwidth extension using generative adversarial networks," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 5029-5033.

[30] SE Eskimez, K. Koishida, and Z. Duan, "Adversarial training for speech super-resolution," IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 2, pp. 347-358, 2019.

[31] X. Hao, C. Xu, N. Hou, L. Xie, ES Chng, and H. Li, "Time-domain neural network approach for speech bandwidth extension," in ICASSP 2020 - 2020 IEEE International Conference on Acoustics , Speech and Signal Processing (ICASSP), 2020, pp. 866-870.

[32] J. Valin and J. Skoglund, "Lpcnet: Improving neural speech synthesis through linear prediction," in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 5891-5895 .

[33] K. Schmidt and B. Edler, "Blind bandwidth extension of speech based on lpcnet," in 2020 28th European Signal Processing Conference (EUSIPCO).

[34] L. Rabiner and R. Schafer, Digital Processing of Speech Signals. Englewood Cliffs: Prentice Hall, 1978.

[35] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, AW Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio ," in The 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA, 13-15 September 2016, 2016, p. 125.

[36] IJ Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," 2014.

[37] Y. Gu and Z. Ling, "Waveform modeling using stacked dilated convolutional neural networks for speech bandwidth extension," in Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017 , 2017, pp. 1123-1127. [Online]. Available: http: //www.isca-speech.org/archive/Interspeech 2017/abstracts/0336.html

[38] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997. [Online]. Available: https://doi. org/10.1162/neco.1997.9.8.1735

[39] Y. Gu, Z. Ling, and L. Dai, "Speech bandwidth extension using bottleneck features and deep recurrent neural networks," in Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA. , September 8-12, 2016, pp. 297-301. [Online]. Available: https://doi.org/10.21437/ Interspeech.2016-678

[40] J. Chung, C. Guelcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," NIPS Deep Learning workshop, Montreal, Canada, 2014. [Online]. Available: http://arxiv.org/abs/1412.3555

[41] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, "Conditional image generation with pixelcnn decoders," CoRR, vol. abs/1606.05328, 2016. [ Online]. Available: http://arxiv.org/abs/1606.05328

[42] WB Kleijn, FSC Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and TC Walters, "Wavenet based low rate speech coding," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 676-680.

[43] Z. Jin, A. Finkelstein, GJ Mysore, and J. Lu, "Fftnet: A real-time speaker-dependent neural vocoder," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 2251-2255.

[44] J.-M. Valin and J. Skoglund, "A real-time wideband neural vocoder at 1.6 kb/s using lpcnet," ArXiv, vol. abs/1903.12087, 2019.

[45] A. Mustafa, A. Biswas, C. Bergler, J. Schottenhamml, and A. Maier, "Analysis by Adversarial Synthesis - A Novel Approach for Speech Vocoding," in Proc. Interspeech, 2019, pp. 191-195 [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-1195

[46] T. Salimans and DP Kingma, "Weight normalization: A simple reparameterization to accelerate training of deep neural networks," in Advances in NeurIPS, 2016, pp. 901-909.

[47] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.

[48] Yao Tianren, Xiang Juanjuan, and Lu Wei, "The computation of line spectral frequency using the second chebyshev polynomials," in 6th International Conference on Signal Processing, 2002., vol. 1, Aug 2002, pp. 190-192 vol.1.

[49] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis," 2018.

[50] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, "Spectral normalization for generative adversarial networks," 2018.

[51] M. Mirza and S. Osindero, "Conditional generative adversarial nets," ArXiv, vol. abs/1411.1784, 2014.

[52] A. Salman, E. Muhammad, and K. Khurshid, "Speaker verification using boosted cepstral features with gaussian distributions," in 2007 IEEE International Multitopic Conference, 2007, pp. 1-5. [53] JH Lim and JC Ye, "Geometric gan," 2017.

[54] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein gan," 2017.

[55] C. Veaux, J. Yamagishi, and K. Macdonald, "Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit," 2017.

[56] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: An ASR corpus based on public domain audio books," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, pp. 5206-5210. [Online]. Available: https://doi.org/10.1109/ICASSP.2015.7178964

[57] M. Soloducha, A. Raake, F. Kettler, and P. Voigt, "Lombard speech database for german language," in Proc. DAGA 2016 Aachen, 03 2016.

[58] "Webrtc vad v2.0.10," https://webrtc.org.

[59] DP Kingma and J. Ba, "Adam: A method for stochastic optimization," CoRR, vol. abs/1412.6980, 2014.

[60] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf , E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019 , pp. 8024-8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

[61] ITU-T Study Group 12, Software tools for speech and audio coding standardization, Geneva, 2005.

[62] GT 26.445, "EVS codec; detailed algorithmic description; technical specification, release 12," Sep. 2014.

[63] ITU-T Study Group 12, P.863 : Perceptual objective listening quality prediction, Geneva, 2018.

[64] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a nash equilibrium,” CoRR, vol. abs/1706.08500, 2017. [Online]. Available: http://arxiv.org/abs/1706.08500

[65] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford,X. Chen, and X. Chen, "Improved techniques for training guns," in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016, pp. 2234-2242. [Online]. Available: https:// proceedings.neurips.cc/paper/2016/file/8a3363abe792db2d8761d6403605aeb7-Paper.pdf

[66] M. Binkowski, J. Donahue, S. Dieleman, A. Clark, E. Elsen, N. Casagrande, LC Cobo, and K. Simonyan, "High fidelity speech synthesis with adversarial networks," CoRR, vol. abs. /1909.11646, 2019. [Online]. Available: http://arxiv.org/abs/1909.11646

[67] D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, E. Elsen, JH Engel, L. Fan, C. Fougner, T. Han, AY Hannun, B. Jun, P. LeGresley, L. Lin, S. Narang, AY Ng, S. Ozair, R. Prenger, J. Raiman, S. Satheesh, D. Seetapun, S. Sengupta, Y. Wang, Z. Wang, C. Wang, B. Xiao, D. Yogatama, J. Zhan, and Z. Zhu, "Deep speech 2: End-to-end speech recognition in english and mandarin," CoRR, vol. abs/1512.02595, 2015. [Online]. Available: http://arxiv.org/abs/1512.02595

[68] AY Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and AY Ng, “Deep speech: Scaling up end-to-end speech recognition," CoRR, vol. abs/1412.5567, 2014. [Online]. Available: http://arxiv.org/abs/1412.5567

[69] A. Graves, S. Fernandez, and F. Gomez, "Connectionist temporal classification: Labeling unsegmented sequence data with recurrent neural networks," in In Proceedings of the International Conference on Machine Learning, ICML 2006, 2006, pp. 369 -376.

[70] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, FM Tyers, and G. Weber, “Common voice: A massively- multilingual speech corpus," CoRR, vol. abs/1912.06670, 2019. [Online]. Available: http://arxiv.org/abs/1912.06670

[71] ITU-R, Recommendation BS.1534-1 Method for subjective assessment of intermediate sound quality (MUSHRA), Geneva, 2003.

Claims (25)

An apparatus for processing a narrowband audio input signal to obtain a wideband audio output signal by performing bandwidth expansion of the narrowband audio input signal, the apparatus comprising:
a signal envelope extrapolator (120) comprising a first neural network (125), wherein the first neural network (125) uses the narrowband audio as an input value of the first neural network (125); configured to receive a plurality of signal envelope samples of an input signal and configured to determine a plurality of extrapolated signal envelope samples as an output value of said first neural network (125); the signal envelope extrapolator (120);
an excitation signal extrapolator (130) configured to receive a plurality of samples of an excitation signal of the narrowband audio input signal and configured to determine a plurality of extrapolated excitation signal samples;
The wideband audio output signal has a bandwidth relative to the narrowband audio input signal depending on the plurality of extrapolated signal envelope samples and depending on the plurality of extrapolated excitation signal samples. a combiner (140) configured to produce the wideband audio output signal such that the width is expanded;
equipment, including. the input values of the first neural network (125) are a first plurality of line spectral frequencies of the narrowband audio input signal; (125) configured to determine a second plurality of line spectral frequencies of the broadband audio output signal as the output value of (125), wherein each of the one or more second plurality of line spectral frequencies is , is associated with a frequency that is greater than any frequency associated with any of the first plurality of line spectral frequencies. Upon training the first neural network (125), the signal envelope extrapolator (120) calculates an impulse response and truncates the impulse response to 3. The apparatus of claim 2, configured to convert wideband linear predictive coding coefficients of 1 to finite impulse response filter coefficients. Upon training the first neural network (125), the signal envelope extrapolator (120) feeds back the error between the wideband audio output signal and the original wideband audio signal or the slope of the error. 4. The apparatus of claim 3, configured to. The first neural network (125) is trained using a first discriminator neural network, and when the first neural network (125) is trained, the first neural network (125) and the first discriminator neural network is configured to operate as a generative adversarial network;
During training of the first neural network (125), the first discriminator neural network uses the output of the first neural network (125) as an input value of the first discriminator neural network. configured to receive a value, or configured to receive a derived value derived from the output value of the first neural network (125) as the input value of the first discriminator network. is,
Here, upon receiving the input value of the first discriminator neural network, the first discriminator neural network outputs the first discriminator neural network as an output of the first discriminator neural network. configured to determine a first quality representation of said input values of a minator neural network, said first neural network (125) configured to learn in dependence on said first quality representation; The apparatus according to any one of claims 1 to 4. Upon receiving the input value of the first discriminator neural network, the first discriminator neural network determines whether the quality indication is such that the input value of the first discriminator neural network is artificially or artificially such that the quality indication indicates the probability that the output value of the first discriminator neural network is related to the recorded audio signal rather than the generated audio signal. 6. The apparatus of claim 5, configured to determine the quality indication to indicate an estimate of the quality associated with the generated signal. The first neural network (125) or the second neural network (135) is trained using a loss function that is dependent on the quality indication determined by the first discriminator neural network. The apparatus according to claim 5 or claim 6. 8. The apparatus of claim 7, wherein the loss function depends on a hinge loss, or a Wasserstein distance, or an entropy-based loss.

Apparatus according to any of claims 7 to 9, wherein the loss function depends on an additional Lp-loss.

12. The apparatus of any of claims 4 to 11, wherein the first discriminator neural network is trained using recorded audio. The excitation signal extrapolator (130) includes a second neural network (135), wherein the second neural network (135) receives the narrowband audio signal as an input value of the second neural network (135). configured to receive a plurality of samples of said excitation signal of an input signal, and/or is said narrowband audio input signal, and/or is a shaped version of said narrowband audio input signal, and said 13. Apparatus according to any of claims 1 to 12, arranged to determine the plurality of extrapolated excitation signal samples as output values of a second neural network (135). the input values of the second neural network (135) are a first plurality of time-domain signal samples of the excitation signal of the narrowband audio input signal and/or are the narrowband audio input signal; and/or a shaped version of said narrowband audio input signal, wherein said second neural network (135) determines that said plurality of extrapolated excitation signal samples is a shaped version of said narrowband audio input signal. determining the output value of the second neural network (135) to be a sample of a second plurality of time-domain signals of an extended time-domain excitation signal with extended bandwidth relative to the excitation signal; 14. The apparatus of claim 13, configured. The second neural network (135) is trained using a second discriminator neural network, and during training of the second neural network (135), the second neural network (135) and the the second discriminator neural network is configured to operate as a second generative adversarial network;
During the learning of the second neural network (135), the second discriminator neural network is configured to:
the output value of the second neural network (135) configured to receive the output value of the second neural network (135) or as the input value of the second discriminator network; a derived value derived from, and/or
configured to receive the output of the combiner (140);
Upon receiving the input value of the second discriminator neural network, the second discriminator neural network is configured to: , wherein the second neural network (135) is configured to learn in dependence on the second quality representation. The apparatus according to claim 13 or claim 14. The apparatus is configured to generate the plurality of samples of the signal envelope of the narrowband audio input signal and the plurality of samples of the excitation signal of the narrowband audio input signal from the narrowband audio input signal. 16. Apparatus according to any of claims 1 to 15, comprising a signal analyzer (110). 17. Apparatus according to any preceding claim, wherein the first neural network (125) comprises one or more convolutional neural networks. 18. Apparatus according to any of claims 1 to 17, wherein the first neural network (125) comprises one or more deep neural networks. A method for processing a narrowband audio input signal to obtain a wideband audio output signal by performing bandwidth expansion of the narrowband audio input signal, the method comprising:
receiving as an input value of a first neural network a plurality of samples of a signal envelope of said narrowband audio input signal; and as an output value of said first neural network a plurality of extrapolated signal envelope samples; a step of determining
receiving a plurality of samples of an excitation signal of the narrowband audio input signal and determining a plurality of extrapolated excitation signal samples;
the wideband audio output signal being extended in bandwidth with respect to the narrowband audio input signal in dependence on the plurality of extrapolated signal envelope samples and the plurality of extrapolated excitation signal samples; generating the wideband audio output signal;
including methods. A method for training a neural network, the method comprising:
the neural network receives as an input to the neural network a first plurality of line spectral frequencies of a narrowband audio input signal;
The neural network determines a second plurality of line spectral frequencies of the wideband audio output signal as an output value of the first neural network, where one or more of the second plurality of line spectral frequencies each of which is associated with a frequency that is greater than any frequency associated with any of the first plurality of line spectral frequencies;
The second plurality of line spectral frequencies of the wideband audio output signal are converted from the line spectral frequency domain to the linear predictive coding domain to obtain the second plurality of linear predictive coding coefficients of the wideband audio output signal. converted,
Converting the second plurality of linear predictive coding coefficients of the wideband audio output signal from the linear predictive coding domain to a finite impulse response filter domain, and converting the linear predictive coding coefficients transformed by the plurality of finite impulse filters into a finite impulse response filter domain. A finite impulse response filter is used to obtain
The method includes the step of training the first neural network depending on linear predictive coding coefficients transformed by the plurality of finite impulse filters. When the first neural network is trained, the linear predictive coding coefficients converted by the plurality of finite impulse filters, or the values derived from the linear predictive coding coefficients converted by the plurality of finite impulse filters, 21. The method of claim 20, wherein the neural network is fed back. Training the first neural network to generate a plurality of samples of the broadband audio output signal depending on the linear predictive coding coefficients transformed with the plurality of finite impulse filters and the plurality of extrapolated excitation signal samples. 21. The method of claim 20, wherein a plurality of wideband audio output signals or values derived from a plurality of samples of the wideband audio output signal are fed back to the neural network. A method for training a first and/or second neural network, the method comprising:
The first neural network receives a plurality of samples of a signal envelope of the narrowband audio input signal as an input value of the first neural network, and receives a plurality of samples of a signal envelope of the narrowband audio input signal as an output value of the first neural network. determining samples of an extrapolated signal envelope; and/or the second neural network receives a plurality of samples of the excitation signal of the narrowband audio input signal as input values of the second neural network; receiving and determining the plurality of extrapolated excitation signal samples as output values of the second neural network;
The first and/or the second neural network is trained using a discriminator neural network, and when the first and/or the second neural network is trained, the first and/or the second neural network is trained using a discriminator neural network. the second neural network and the discriminator neural network operate as a generative adversarial network;
During the training of the first and/or the second neural network, the discriminator neural network uses the input values of the first and/or the second neural network as input values of the discriminator neural network. receiving an output value, or receiving a derived value derived from the output value of the first and/or the second neural network as the input value of the discriminator network;
Upon receiving the input value of the discriminator neural network, the discriminator neural network determines, as an output of the discriminator neural network, a quality representation of the input value of the discriminator neural network; and The method, wherein the first and/or the second neural network learns depending on the quality indication. The discriminator neural network is a first discriminator neural network,
the first neural network is trained using the first discriminator neural network, the first neural network is trained in dependence on the quality representation, which is a first quality representation;
The second neural network is trained using a second discriminator neural network, and during training of the second neural network, the second neural network and the second discriminator neural network are , operates as a second generative adversarial network,
During training of the second neural network, the second discriminator neural network receives the output value of the second neural network as an input value of the second discriminator neural network. or receiving a derived value derived from the output value of the second neural network as the input value of the second discriminator network;
Upon receiving the input value of the second discriminator neural network, the second discriminator neural network is configured to: determining a second quality representation of the input values of , the second neural network being configured to learn in dependence on the second quality representation;
24. The method according to claim 23. 25. A computer program product for carrying out a method according to any of claims 19 to 24 when executed on a computer or a signal processor.

Images